JP6088344B2 - Imaging device and noise removal program - Google Patents

Description

  The present invention relates to a photographing apparatus and a noise removing program that take an image while removing noise.

An output image of an imaging system such as a digital camera generally includes “noise”. Here, noise refers to fluctuations in pixel values that occur each time a shooting attempt is made even if the same object is shot. This fluctuation is caused by the fact that the signal (input stimulus) input to the imaging system and the response of the image sensor (sensor response) in the imaging system both behave stochastically.
In general, "true images" (images obtained by averaging infinitely taken images) from which noise of each pixel value is removed are more from the viewpoint of image processing and image presentation than images that contain noise. It is often desirable. It should be noted that the problem of obtaining a “true image” or an image close to this from an image obtained by a single shooting is mathematically equivalent to an inverse problem and requires an estimation process assuming some statistical model.
Conventionally, a technique for reducing blur and noise from an image captured by a camera or the like by solving an estimation problem assuming a deteriorated image model in which an original image deteriorates due to a point spread function has been disclosed (see Patent Document 1). ).

JP 2007-183842 A

The technique described in Patent Document 1 estimates a true image from an image captured by an image sensor based on a deteriorated image model in order to reduce noise and the like.
However, image degradation is not inherently caused by only the image sensor, but also by an optical system such as a lens that forms an image on the image sensor.
That is, the conventional method does not perform control for optimizing the optical system, so that there is a problem that an error is included in the estimated image and an optimal image cannot be obtained.

  The present invention has been made in view of such problems, and an object of the present invention is to provide a photographing apparatus and a noise removal program that improve the accuracy of noise removal by adaptively controlling the optical system.

  In order to solve the above-mentioned problem, an imaging apparatus according to claim 1 records an optical system capable of focus control, an imaging unit that captures an image formed by the optical system, and an image captured by the imaging unit. An image capturing apparatus including an image recording unit configured to include a statistic estimation unit, a filter parameter derivation unit, an optical system control unit, and a noise removal unit.

In such a configuration, the imaging apparatus uses the statistical quantity estimation unit to perform a first probability distribution obtained by modeling a true image to be imaged with a statistical model using the statistical quantity as a parameter, and a point spread that is an optical filter that identifies the optical system. From the second probability distribution that models the process of obtaining a captured image recorded in the image recording means with a predetermined noise level from a true image via a function, the first probability distribution is determined by the maximum likelihood method. Estimate statistics. Here, the true image means an ideal image that does not contain noise.
Thus, the photographing apparatus can estimate the statistical properties of the true image to be photographed from the actually captured image.

Then, the image capturing device uses the filter parameter deriving unit to perform point spread that minimizes the expected value of the error between the true image and the image after noise removal on the condition that the statistic and the noise level estimated by the statistic estimating unit are used. A filter parameter (for example, standard deviation) specifying the distribution shape of the function is obtained.
Thereby, the photographing apparatus can specify the distribution shape of the point spread function when the true image and the image after noise removal are most approximated by the filter parameter.

Then, the photographing apparatus refers to the look-up table in which the relationship between the filter parameter and the focus amount of the optical system is previously associated by the optical system control unit, and the focus corresponding to the filter parameter obtained by the filter parameter deriving unit. The amount controls the focus of the optical system.
Thereby, the photographing apparatus can perform focus control so that the true image approximates the image after noise removal.

Then, the image capturing apparatus uses the point spread function specified by the filter parameter obtained by the filter parameter deriving unit, using the point spread function specified by the filter parameter deriving unit, on the condition that the statistical amount and the noise level estimated by the statistical amount estimating unit are used as a condition. An image that maximizes the posterior probability of the probability distribution obtained by integrating the first and second probability distributions with respect to the captured image recorded in the recording unit is generated as an image after noise removal.
As a result, the imaging device generates a captured image that most closely approximates the true image in the entire system through the optical system and the imaging system via the probability distribution obtained by integrating the first and second probability distributions. Can do.

  According to a second aspect of the present invention, in the photographing apparatus according to the first aspect, the filter parameter deriving unit stores in advance a combination of a filter parameter that specifies a point spread function, a statistic, and a noise level. It is characterized in that the minimum filter parameter is selected from the up-table from the statistical amount and noise level estimated by the statistical amount estimating means.

  In such a configuration, the image capturing apparatus uses the filter parameter deriving unit to distribute the point spread function for allowing the true image to be captured to have the statistic estimated by the statistic estimating unit in an environment with a predetermined noise level. Filter parameters that specify the shape can be selected.

  According to a third aspect of the present invention, there is provided the photographing apparatus according to the first or second aspect, wherein the optical system control means adjusts the focus value of the optical system after the optical system is focused. The current focus value and the filter parameter derived by the filter parameter deriving means are the most from the lookup table in which the filter parameter of the point spread function and the defocus amount corresponding to the deviation from the focus value are stored in advance. A defocus amount corresponding to an approximate filter parameter is selected, and the optical system is driven and controlled so that the focus is shifted by the defocus amount.

In such a configuration, the imaging apparatus shifts the focus by the defocus amount selected from the look-up table after the optical system control means focuses on the imaging target. As a result, the photographing apparatus shoots with an appropriate blur added to the photographing target.
In other words, depending on the level of noise, the photographing apparatus can generate an optimal image by adding blur (defocus) rather than in a focused state.

  According to a fourth aspect of the present invention, in the photographing apparatus according to any one of the first to third aspects, the noise removing unit performs drive control of the optical system a predetermined number of times. As a result, an image after noise removal is generated for the captured image recorded in the image recording means.

In such a configuration, the imaging apparatus acts to increase the accuracy of the statistic estimated by the statistic estimation means by performing drive control of the optical system a predetermined number of times.
Thereby, the photographing apparatus can improve the accuracy of noise removal.

  According to a fifth aspect of the present invention, there is provided a noise removal program that includes an optical system capable of focus control, an imaging unit that captures an image formed by the optical system, and an image recording unit that records an image captured by the imaging unit. And the computer of the photographing apparatus having the above functions as statistical quantity estimation means, filter parameter derivation means, optical system control means, and noise removal means.

  In such a configuration, the noise removal program is a first probability distribution obtained by modeling a true image to be imaged with a statistical model using a statistical amount as a parameter, and an optical filter for specifying an optical system. From the second probability distribution that models the process of obtaining a captured image recorded in the image recording means with a predetermined noise level from the true image via the spread function, the first probability distribution is obtained by the maximum likelihood method. Estimate the statistic of.

  Then, the noise removal program minimizes the expected value of the error between the true image and the image after noise removal on the condition that the filter parameter derivation means is the statistic estimated by the statistic estimation means and the noise level. A filter parameter (for example, standard deviation) for specifying the distribution shape of the spread function is obtained.

  The noise removal program corresponds to the filter parameter obtained by the filter parameter deriving means by referring to a look-up table in which the relationship between the filter parameter and the focus amount of the optical system is previously associated by the optical system control means. The focus of the optical system is driven and controlled by the focus amount.

  Then, the noise removal program uses the point spread function specified by the filter parameter obtained by the filter parameter deriving unit, using the noise and the noise level estimated by the statistical amount estimating unit as a condition. For the captured image recorded in the image recording means, an image that maximizes the posterior probability of the probability distribution obtained by integrating the first and second probability distributions is generated as an image after noise removal.

The present invention has the following excellent effects.
According to the first and fifth aspects of the present invention, the noise is reduced most in the entire system of the photographing apparatus that images the photographing object by the optical system and deletes the noise by adaptively controlling the optical system. An optimal captured image can be generated.
Thus, the inventions according to claims 1 and 5 can take an image with the highest accuracy (close to a true image) in a noise environment.

  According to the second aspect of the invention, in addition to the effect of the first aspect, the parameters of the optical filter (point spread function) for specifying the optical system are obtained from the look-up table. The image after noise removal can be generated at high speed.

  According to the third aspect of the invention, in addition to the effect of the first aspect, since the defocus amount of the optical system is acquired from the lookup table, the defocus amount can be specified at a high speed, and noise removal can be performed. Images can be generated at high speed.

  According to the fourth aspect of the invention, in addition to the effect of the first aspect, the statistical amount estimated by the statistical amount estimating means works in a direction approximating the statistical amount of the true image. Can be estimated more accurately, and noise can be more accurately removed from the captured image.

It is a functional block diagram which shows the structure of the imaging device which concerns on embodiment of this invention. It is an example of the graph which shows the relationship between the average square error of a true image and the image after noise removal, and the filter parameter of an optical filter, Comprising: A vertical axis is an average square error and a horizontal axis is a filter parameter. It is a flowchart which shows operation | movement when operating the imaging device which concerns on embodiment of this invention as a moving image camera. It is a flowchart which shows operation | movement when the imaging device which concerns on embodiment of this invention is operated as a still image camera. It is a figure for demonstrating the simulation result of the imaging device which concerns on embodiment of this invention, (a) is a true image, (b) is an image which applied the optimal defocus value, (c) is defocused. It is an image that has not been performed.

Embodiments of the present invention will be described below with reference to the drawings.
[Configuration of Shooting Device]
First, with reference to FIG. 1, the structure of the imaging device 1 which concerns on embodiment of this invention is demonstrated. This photographing apparatus 1 controls an optical system to remove noise from an image obtained by photographing a photographing target.
Here, the photographing apparatus 1 includes an optical system 10, an image sensor 11, an image recording unit 12, a control unit 13, a statistic storage unit 14, and a filter parameter storage unit 15.

The optical system 10 is a lens (group) that forms an image of a subject to be photographed (subject) on the image sensor 11. The optical system 10 includes a focus lens that is driven by the drive unit 100 and moves in the optical axis direction to perform focus control. The driving amount (focus value) for the driving unit 100 is input from the control means 13.
That is, the optical system 10 functions as an optical filter that generates blur or the like on a true image. Here, the true image is an ideal virtual image obtained by shooting the shooting target without blurring or noise.

The image sensor 11 (imaging means) converts an optical image formed by the optical system 10 into an electric signal. The image sensor 11 is, for example, a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like.
The image sensor 11 writes an image (image data) converted into an electric signal into the image recording means 12 as a sensor output.

The image recording means 12 records an image output from the image sensor 11 and is a general recording medium such as a semiconductor memory.
The image recorded in the image recording unit 12 is an image obtained by adding blurring by the optical system 10 (optical filter), noise by the image sensor 11, and the like (image before noise removal) to a true image to be captured. It has become.
The image recorded in the image recording unit 12 is read out by the control unit 13.

  The control means 13 performs reverse processing of the optical filter (reverse optical filter) and noise removal processing on the image (image before noise removal) recorded on the image recording means 12 while controlling the drive of the optical system 10. It is. Here, the control unit 13 includes a statistic estimation unit 130, a filter parameter derivation unit 131, an optical system control unit 132, and a noise removal unit 133.

The statistic estimation means 130 estimates a statistic (statistical parameter) when a true image obtained by photographing a photographing target (subject) is modeled with a predetermined statistical model.
Here, a statistical model (first probability distribution, second probability distribution) in which the statistic estimation means 130 estimates a statistic will be described using mathematical expressions.

(Statistical model of true image: first probability distribution)
First, it is assumed that a statistical model (first probability distribution) of a true image is a conditional probability distribution P (s | β, h) of the following equation (1).

  Here, H (s; β, h) is the energy of the prior distribution defined by the following formula (2). Further, s is a pixel value of a true image arranged in a vector form. Β and h are statistics (statistical parameters), which correspond to the secondary statistics defined by equation (2).

Here, s _i and s _j represent individual pixel values of the image s (1 ≦ i ≦ N, 1 ≦ j ≦ N; N is the number of pixels of the image s). Note that the first term on the right side of the first formula in Formula (2) is obtained by multiplying the sum of squares of the differences for all combinations of two adjacent pixels i and j by the parameter β, and the second term on the right side. Is obtained by multiplying the square sum of all the pixel values of the image s by the parameter h.
The first formula of the formula (2) is represented by the second formula. J in the second expression of the expression (2) is a matrix such that J _ij = 1 when the pixels i and j are adjacent to each other, and J _ij = 0 in other cases, and I is a unit matrix. . T represents transposition.

In the statistical model represented by the equations (1) and (2), the larger the parameter β, the smoother the image, and the larger the parameter h, the lower the contrast.
Here, since the matrix J is an object to be translated with respect to the image (does not depend on the location of the image), the Fourier transform of the point spread function when this is regarded as a convolution process is expressed as J ^~ (J tilde), a true image When placing the Fourier transform and s ^{~ (s} tilde), equation (2), as shown in the following equation (3), expressed as the sum of the calculation of each frequency k using convolution theorem. N represents the number of pixels of the image s.

(Statistical model of imaging system: second probability distribution)
Further, a statistical model (second probability distribution) of a process in which a photographed image (image before noise removal; an image recorded in the image recording unit 12 in FIG. 1) is obtained from a true image is expressed by the following equation (4). ) With a conditional probability distribution P (r | s; R).

  Here, L (s, r; A, R) is the energy of likelihood defined by the following equation (5). Further, s is a pixel value of a true image, and r is a pixel value of a captured image (image before noise removal) arranged in a vector form. A is an optical filter (a point spread function when the optical system 10 forms an image on the image sensor 11), and R is a parameter representing a noise level. Here, it is assumed that the noise is Gaussian noise added for each pixel.

Here, s _j represents a pixel value of the image s (1 ≦ i ≦ N), and r _i represents a pixel value of the image r (1 ≦ j ≦ N). A _ij is the value of the point spread function at the pixel position of the i-th row and j-th column in the two pixels i and j. T represents transposition.
Here, since the optical filter is a translation target, Equation (5) is expressed as the sum of operations for each frequency k as shown in Equation (6) below. N represents the number of pixels of the images s and r.

Here, s ^~ (s tilde) is the Fourier transform of the true image s, r ^~ (r tilde) is the Fourier transform of the captured image r, and A ^~ (A tilde) is the Fourier transform of the optical filter (point spread function) A. Indicates.

The statistic estimation means 130 is a statistic that approximates a captured image to a true image based on a statistical model of the true image (the above formula (1)) and a statistical model of the shooting system (the above formula (4)). The quantities (statistical parameters; here β and h) are estimated.
That is, the statistic estimation means 130 calculates a statistic that maximizes the function value of the likelihood function (log likelihood function) regarding the statistic under the captured image r by the maximum likelihood method.
From the equations (1) and (4), the log likelihood function F (β, h) regarding the statistical parameters β and h under the captured image r is given by the following equation (7).

Therefore, the statistic estimation means 130 obtains statistical parameters β and h that maximize the log-likelihood function F (β, h) given by this equation (7). The noise level R is assumed to be given in advance as R = (N / S) ² corresponding to the sensor sensitivity value (S / N; signal-to-noise ratio [SNR]) of the image sensor 11.

As a method for solving the maximization problem of the log likelihood function F (β, h), a general method may be used, and for example, a gradient method can be used.
That is, the statistic estimation means 130 obtains the statistical parameters β and h by repeating the gradient method update step shown in the following equation (8) until the value converges. Note that w is a predetermined value for determining the update amount.

Thereby, the statistic estimation means 130 can estimate the image feature of the photographing target (true image) by the statistic (statistic parameters β, h) of a predetermined statistical model.
The statistic estimation means 130 outputs the estimated statistic to the filter parameter derivation means 131 and writes it to the statistic storage means 14.

The filter parameter deriving unit 131 derives a parameter (filter parameter) of the optical system (optical filter) on the condition that the statistical amount estimated by the statistical amount estimating unit 130 and a noise level given in advance are used as conditions.
This filter parameter is information for specifying the filter shape (distribution shape) of the optical filter, and is, for example, a standard deviation, a variance, etc., which are statistics when the point spread function of the optical filter is assumed to be a normal distribution.
Here, s is a true image estimated to have the statistic obtained by the statistic estimation means 130, and s ^ (s) is an image obtained as a result of noise removal from the inverse optical filter (denoised image). Hat), when the number of pixels of the image is N, the image s ^ after noise removal has an error from the true image s, for example, a mean square error MSE (Mean Square Error) shown in the following equation (9). Desirably minimal.

  The expected value <MSE> of the MSE has a lower limit as shown in the following formula (10), and becomes a lower limit when the statistic is correctly estimated by the statistic estimation means 130. Β ^ (β hat) and h ^ (h hat) indicate statistical parameters of the image s ^ (s hat) after noise removal.

As can be seen from this equation (10), the MSE expected value <MSE> depends on the filter shape (distribution shape) of the optical filter A.
For example, assuming that the point spread function of the optical filter is a normal distribution with a standard deviation γ, the filter dependency as shown in FIG. 2 is shown, and the MSE takes the minimum value for a specific filter parameter γ ^* .
The value of γ ^* depends on the statistical parameters β and h and the noise level R.

That is, the filter parameter deriving unit 131 solves the minimization problem regarding the MSE _{min based} on the statistical parameters β and h which are the statistical amounts estimated by the statistical amount estimating unit 130 and the noise level R of the image sensor 11. Thus, an optimum filter parameter γ ^* is derived.

Specifically, the filter parameter deriving unit 131 adds the MSE _min value for the combination of the statistic (statistical parameters β, h), the noise level R, and the filter parameter γ to the lookup table storage unit (not shown). Are stored in advance as a four-dimensional lookup table, and a filter parameter γ ^* that minimizes the value of MSE _min corresponding to β, h, R is selected.

Also, for example, the filter parameter deriving unit 131 minimizes the statistics (statistical parameters β, h), the noise level R, and the MSE obtained in advance by β, h, R in the lookup table storage unit (not shown). Γ to be converted may be stored in advance as a three-dimensional lookup table, and γ corresponding to β, h, R may be selected as the filter parameter γ ^* . Note that γ for minimizing MSE may be obtained in advance by a general method such as exhaustive search, gradient method, Newton method, or the like.

Further, for example, the filter parameter deriving unit 131 may calculate the filter parameter γ ^* by calculation without using a lookup table.
A general method such as a gradient method or a Newton method may be used as a method for solving the minimization problem by calculating the filter parameter γ ^* for minimizing the MSE. For example, in order to calculate the filter parameter γ ^* that minimizes the MSE by the gradient method, the update step of the gradient method shown in the following equation (11) is set to the value for the MSE _min shown in the equation (10). By repeating until the convergence, the minimum solution γ, that is, the filter parameter γ ^* is obtained.

Accordingly, the filter parameter deriving unit 131 can derive a parameter (filter parameter) for controlling the optical filter.
The filter parameter deriving unit 131 outputs the derived filter parameter to the optical system control unit 132 and writes it to the filter parameter storage unit 15.

The optical system control unit 132 controls driving of the optical system 10 (optical filter) based on the filter parameter derived by the filter parameter deriving unit 131.
Here, the optical system control means 132 has a point spread due to blur (when the optical filter is a normal distribution, its standard deviation γ) matches the filter parameter γ ^* derived by the filter parameter derivation means 131 (most approximate). Thus, the focus value of the optical system 10 is controlled.

Specifically, the optical system control unit 132 uses an existing autofocus technique (for example, a contrast detection method, a phase difference detection method, etc.) to change the optical system in a defocusing direction from a focused state. Here, the defocus amount is a change in the focus value from the focused state until the point spread (in this case, the standard deviation) of the optical filter coincides with the filter parameter γ ^* (most approximate). Amount.

The defocus amount may be associated with the defocus amount having the standard deviation γ ^* in advance with respect to the focus value.
For example, the look-up table storage means not shown in the figure shows the value of the function γ (f, Δf) when the focus value in a focused state is f and the deviation (defocus amount) from the focus value is Δf. Is stored as a two-dimensional lookup table.

This two-dimensional lookup table depends on the lens in the optical system 10. Therefore, the two-dimensional lookup table measures the point spread function for various combinations of (f, Δf) in the lens used in the optical system 10 in advance, and obtains the standard deviation γ of the normal distribution that approximates this. Remember it.
Then, the optical system control unit 132 acquires a defocus amount Δf that is a value closest to the filter parameter γ ^* with respect to the current focus value f from the two-dimensional lookup table.
Then, the optical system control unit 132 outputs a focus value (f + Δf) to the optical system 10 as an optical system control signal for performing focus control of the optical system 10.

  Here, the optical system control means 132 drives the optical system 10 in the direction of defocusing. However, this is because an image captured by the image sensor 11 can obtain an image closer to the true image in the defocused state than in the focused state depending on the noise level. .

The noise removing unit 133 performs reverse optical filter processing and noise removal processing on the captured image (pre-noise removal image) recorded in the image recording unit 12.
Here, the noise removing unit 133 applies the above-described formula (1) to the captured image recorded in the image recording unit 12 on the condition that the statistical amount estimated by the statistical amount estimating unit 130 and a predetermined noise level are used. An image that maximizes the posterior probability of the probability distribution obtained by integrating the probability distribution of 1) and the equation (4) is generated as an image after noise removal.
Here, the posterior probability distribution P (s | r; β, h, R) of the true image s under the captured image (image before noise removal) r is expressed by the following equation (12) from Bayes' theorem. Given in.

The noise removing unit 133 outputs s = s ^ (s hat) that maximizes the posterior probability as an image after noise removal.
In addition, when the formula (2) and the formula (5) are substituted into the formula (12), the posterior probability distribution P (s | r; β, h, R) is represented by the following formula (13).

  Note that, under the statistical model described above, the posterior probability distribution is a normal distribution, and the mode value is equal to the average. Therefore, the noise removing unit 133 generates a noise-removed image s ^ (s hat) that maximizes the posterior probability according to the following equation (14).

Here, the noise removal unit 133 refers to the statistic estimated by the statistic estimation unit 130 and stored in the statistic storage unit 14 for the statistic (statistical parameters β, h). Note that the matrices J and I and the noise level R are the same as those described in the equations (2) and (5), and thus the description thereof will be omitted. The removal means 133 is preset. The optical filter A represents a point spread function of the optical system 10 as described in the equation (5), and parameters for specifying the function are sequentially stored in the filter parameter storage unit 15.
That is, for the optical filter A, the noise removing unit 133 refers to the filter parameters stored in the filter parameter storage unit 15 (for example, the standard deviation and variance when the point spread function is a normal distribution), and The point spread function specified by the parameter is used as the optical filter A.

The statistic storage unit 14 stores the statistic estimated by the statistic estimation unit 130 and is a general recording medium such as a semiconductor memory.
The filter parameter storage unit 15 stores the filter parameter derived by the filter parameter deriving unit 131, and is a general recording medium such as a semiconductor memory.

  As described above, by configuring the image capturing apparatus 1, the image capturing apparatus 1 controls the optical system 10 to remove noise from an image including noise captured by the image sensor 11, thereby providing an optical system. And noise generated in a series of processing through the photographing system can be accurately removed.

The control means 13 of the photographing apparatus 1 is operated by a program (noise removal program) for causing the computer in the photographing apparatus 1 to function as each means described above.
That is, the photographing apparatus 1 realizes the functions of the respective units of the control unit 13 by developing a noise removal program stored in a storage unit (not shown) in a RAM (Random Access Memory) and operating it by a computer. Can do.

[Operation of the camera]
Next, referring to FIG. 3 (refer to FIG. 1 as appropriate for the configuration), the operation of the imaging apparatus 1 according to the embodiment of the present invention will be described.
Here, the photographing apparatus 1 will be described as a moving image camera (video camera).
First, the photographing apparatus 1 captures an image of a photographing object via the optical system 10 by the image sensor 11 and writes it in the image recording means 12 (step S1).

Then, the photographing apparatus 1 determines in advance from an image (photographed image) written in the image recording means 12 under the condition of the noise level of the image sensor 11 given in advance by the statistic estimation means 130 of the control means 13. A statistic (statistical parameter) that maximizes the likelihood function (log likelihood function; see Equation (7)) related to the statistic of the statistical model is calculated (step S2).
As a result, the statistic estimation means 130 estimates the image characteristics of the object to be photographed by using a statistic (statistic parameter) of a predetermined statistical model.

Then, the photographing apparatus 1 uses the filter parameter deriving unit 131 of the control unit 13 to calculate the true image, the noise from the statistical amount (statistical parameter) calculated (estimated) in step S2 and the noise level given in advance. A parameter (filter parameter) of the optical filter that minimizes the mean square error MSE with the noise-removed image from which noise has been removed by the removing unit 133 is derived (step S3). This filter parameter is, for example, a standard deviation for specifying a filter when the optical filter has a normal distribution.
At this time, the filter parameter deriving unit 131 may acquire the filter parameter from a statistical parameter, a noise level, or the like using a lookup table prepared in advance, or a filter parameter that minimizes the mean square error MSE. You may obtain | require by calculation by solving a minimization problem.

The photographing apparatus 1 then drives the optical system 10 so that the filter parameter derived in step S3 by the optical system control unit 132 of the control unit 13 and the optical filter parameter of the optical system 10 match (most approximate). And the optical system 10 is driven and controlled (step S4).
At this time, the optical system control unit 132 performs a two-dimensional look in which the value of the filter parameter (standard deviation) is associated in advance with the focus value in the focused state and the deviation amount (defocus amount) therefrom. From the up table, the defocus amount that is the closest to the filter parameter with respect to the current focus value is acquired.

Then, the optical system control unit 132 outputs a focus value obtained by adding the defocus amount to the current focus value to the optical system 10.
As a result, the focus of the optical system 10 is controlled, and the image formed on the image sensor 11 becomes a defocused image (captured image [image before noise removal]) according to the noise level, and the image recording unit 12. To be recorded.

The photographing apparatus 1 is recorded in the image recording unit 12 by the noise removing unit 133 of the control unit 13 based on the statistical amount (statistical parameter) estimated in step S2 and the noise level given in advance. A reverse optical filter process and a noise removal process are performed on the captured image (image before noise removal) (step S5).
That is, the noise removing unit 133 generates an image (image after noise removal) that maximizes the posterior probability of the true image under the captured image (image before noise removal) (see the above formula (14)).
Thereafter, the photographing apparatus 1 returns to step S1 and continues to operate.

Accordingly, the imaging apparatus 1 can acquire an image from which noise has been removed while sequentially adjusting the optical system 10 during moving image shooting.
Here, in order to simplify the description, the imaging apparatus 1 drives and controls the optical system in steps S2 to S4 after the imaging apparatus 1 captures and records the imaging target in step S1. However, steps S2 to S4 are not necessarily performed every time an imaging target is imaged and recorded.
For example, after first performing the operations of Steps S1 to S5, the imaging apparatus 1 may be configured to operate Steps S2 to S4 each time imaging and recording in Step S1 are performed a predetermined number of times.

Next, referring to FIG. 4 (refer to FIG. 1 as appropriate for the configuration), the operation when the photographing apparatus 1 according to the embodiment of the present invention is a still image camera (still camera) will be described.
Even when the photographing apparatus 1 is operated as a still image camera, the basic operation is the same as that of the moving image camera described in FIG.

In the same manner as the moving image camera described with reference to FIG. 3, the photographing apparatus 1 performs the operations from step S1 to step S4, and then the control unit 13 accumulates the number of operations in steps S1 to S4. 3 times, etc.) is determined (step S4B).
If the number of operations in steps S1 to S4 has not reached the predetermined number (No in step S4B), the imaging device 1 returns to step S1 and continues the operation.

On the other hand, when the number of operations in steps S1 to S4 reaches a predetermined number (Yes in step S4B), the photographing apparatus 1 uses the noise removal unit 133 of the control unit 13 to estimate the statistics (statistical parameters) in step S2. Based on the noise level given in advance, reverse optical filter processing and noise removal processing are performed on the captured image (image before noise removal) recorded in the image recording means 12 (step S5).
As described above, by repeatedly performing steps S1 to S4, the imaging apparatus 1 can improve the accuracy of estimation of the statistic by the statistic estimation unit 130, and can improve the noise removal accuracy.

  As described above, the imaging apparatus 1 can optimize an image throughout the entire imaging system by adaptively controlling the optical system to remove noise, and in an environment where noise occurs in the image sensor. The image with the least deterioration can be obtained.

[Processing result]
Next, a simulation result of the photographing apparatus 1 will be described with reference to FIG. FIG. 5A is a Lenna standard image used as a true image. 5B and 5C, the image of FIG. 5A is converted by a point spread function corresponding to the optical system 10 and deteriorated by predetermined noise (sensor noise) (image recording means 12). The image after noise removal (noise removal means 133).

Note that FIG. 5B is a diagram in which the image of FIG. 5A is converted by a point spread function corresponding to the filter parameters obtained by the statistic estimation unit 130 and the filter parameter derivation unit 131, and is degraded by predetermined noise. Then, the image after the noise removal processing is performed a plurality of times. That is, the image in FIG. 5B corresponds to an image from which noise is removed by controlling the optical system 10 so that the optical system control unit 132 has an optimum defocus value according to the filter parameter. The image in FIG. 5C corresponds to an image that has not been defocused.
However, here, in order to make the processing result easy to see, noise that degrades the image (that is, the sensor sensitivity value of the image sensor 11) is set to “0 dB”.

  As shown in FIG. 5B, defocusing is performed by optimizing the focus value (defocus value) of the optical system 10 of the imaging apparatus 1 even when the captured image is deteriorated due to sensor noise. An image closer to the true image (FIG. 5A) can be taken than the image of FIG.

The configuration and operation of the imaging device 1 according to the embodiment of the present invention have been described above, but the present invention is not limited to this embodiment.
For example, here, the statistic estimation unit 130 estimates the statistic using a statistical model modeled with a statistical parameter β that specifies the smoothness of the image and a statistical parameter h that specifies the contrast. However, the statistic estimation means 130 may estimate the statistic with a statistical model modeled with another statistic. For example, average luminance or the like may be used as a statistic.

Further, here, the filter parameter deriving unit 131 has been described with an example in which the point spread function of the optical filter is assumed to be a normal distribution and the number of filter parameters is one (standard deviation γ). However, a plurality of filter parameters may be used as in the case where the point spread function is expressed by a general normal distribution.
For example, when the point spread function is a general normal distribution as shown in the following equation (15), the filter parameters are _two , γ ₁ and γ ₂ . Note that Γ is a gamma function.

  Thus, when the number of filter parameters is two, the dimensions of the look-up table used by the filter parameter deriving unit 131 and the optical system control unit 132 may be increased by one.

1 photographing device 10 optical system (optical filter)
11 Image sensor (imaging means)
DESCRIPTION OF SYMBOLS 12 Image recording means 13 Control means 130 Statistics estimation means 131 Filter parameter derivation means 132 Optical system control means 133 Noise removal means 14 Statistics storage means

Claims (5)

An imaging apparatus comprising: an optical system capable of focus control; an imaging unit that captures an image formed by the optical system; and an image recording unit that records an image captured by the imaging unit.
A first probability distribution obtained by modeling a true image to be photographed with a statistical model using a statistic as a parameter, and a noise determined in advance from the true image via a point spread function that is an optical filter for specifying the optical system A statistic estimation means for estimating the statistic by a maximum likelihood method from a second probability distribution modeling a process of obtaining a captured image recorded in the image recording means at a level;
A filter for specifying a distribution shape of the point spread function that minimizes an expected value of an error between the true image and an image after noise removal on the condition that the statistic estimated by the statistic estimation means and the noise level Filter parameter derivation means for obtaining parameters;
With reference to a look-up table in which the relationship between the filter parameter and the focus amount of the optical system is previously associated, the focus of the optical system is adjusted with the focus amount corresponding to the filter parameter obtained by the filter parameter deriving means. Optical system control means for driving control;
The imaging recorded in the image recording unit using a point spread function specified by the filter parameter obtained by the filter parameter deriving unit on condition of the statistical amount estimated by the statistical amount estimating unit and the noise level Noise removing means for generating an image that maximizes the posterior probability of the probability distribution obtained by integrating the first and second probability distributions as an image after noise removal with respect to the image;
An imaging apparatus comprising:   The filter parameter derivation means includes a statistic estimated by the statistic estimation means and a statistic estimated by the statistic estimation means from a look-up table in which a combination of the filter parameter for specifying the point spread function, the statistic and the noise level is stored The photographing apparatus according to claim 1, wherein a minimum filter parameter is selected at a noise level.   The optical system control means, after focusing the optical system, the focus value when the focus is achieved, the filter parameter of the point spread function, and the defocus that is the amount of deviation from the corresponding focus value A defocus amount corresponding to the current focus value and the filter parameter closest to the filter parameter derived by the filter parameter deriving means is selected from a lookup table in which the amount is stored in advance. The photographing apparatus according to claim 1, wherein the optical system is driven and controlled to shift a focus.   The noise removing unit generates the image after the noise removal with respect to a captured image recorded in the image recording unit as a result of performing drive control of the optical system a predetermined number of times. The imaging device according to any one of claims 1 to 3. A computer of an imaging apparatus comprising: an optical system capable of focus control; an imaging unit that captures an image formed by the optical system; and an image recording unit that records an image captured by the imaging unit.
A first probability distribution obtained by modeling a true image to be photographed with a statistical model using a statistic as a parameter, and a noise determined in advance from the true image via a point spread function that is an optical filter for specifying the optical system A statistic estimation means for estimating the statistic by a maximum likelihood method from a second probability distribution modeling a process of obtaining a captured image recorded in the image recording means at a level;
A filter for specifying a distribution shape of the point spread function that minimizes an expected value of an error between the true image and an image after noise removal on the condition that the statistic estimated by the statistic estimation means and the noise level Filter parameter derivation means for obtaining parameters,
With reference to a look-up table in which the relationship between the filter parameter and the focus amount of the optical system is previously associated, the focus of the optical system is adjusted with the focus amount corresponding to the filter parameter obtained by the filter parameter deriving means. Optical system control means for driving control,
The imaging recorded in the image recording unit using a point spread function specified by the filter parameter obtained by the filter parameter deriving unit on condition of the statistical amount estimated by the statistical amount estimating unit and the noise level Noise removal means for generating an image that maximizes the posterior probability of the probability distribution obtained by integrating the first and second probability distributions as an image after noise removal with respect to the image;
Noise removal program to function as.