JP2015088914A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a dynamic image in which fixed pattern noise is hardly perceived.SOLUTION: Noise dispersion is derived on the basis of an exposure condition at image capture time, a noise image is generated in which white noise corresponding to the noise dispersion is superposed on fixed pattern noise. Information about a spatial frequency having sensitivity to the moving speed of a subject in human visual perception is extracted from each of the fixed pattern noise map and the noise image, and the fixed pattern noise map and the noise image are divided for each of a plurality of spatial frequency bands. The noise image divided for each of the plurality of spatial frequency bands is masked on the basis of the fixed pattern noise map divided for each of the plurality of spatial frequency bands and synthesized, then the amount of fixed pattern noise perception is calculated.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and is particularly suitable for use in reducing fixed pattern noise of an image.

Fixed pattern noise of a solid-state imaging device such as a CCD or CMOS sensor provided in an imaging apparatus such as a digital camera or a digital video camera has a fixed position on the screen and is easily noticeable. In the case of a digital camera, a dark image is captured by blocking incident light to the solid-state image sensor during imaging, and an image of fixed pattern noise is output from the dark image. A method of removing fixed pattern noise by subtracting this fixed pattern noise image from a captured image is known (see Patent Document 1).
However, when a moving image is picked up by a digital video camera, the incident light to the solid-state image pickup device cannot be shielded by closing the shutter. Therefore, only the fixed pattern noise cannot be removed by the method described above. Therefore, there is a method of performing random noise reduction by adding random noise to fixed pattern noise (see Patent Document 2).

JP 2006-311086 A JP 2004-289241 A

S. Daly, "The Visible Differences Predictor: An algorithm for the assessment of image fidelity," in A.B.Watson, editor, Digital Image and Human Vision, pages 179-206, Cambridge, MA, MIT Press, 1993. J. Laird, M. Pelz, E. Montag, and S. Daly: "Spatio-Velocity CSF as a Function of Retinal Velocity Using Unstabilized Stimuli", Proc. Of SPIE, 6057, (2006) D. Kelly, Motion and vision. II. Stabilized spatio-temporal threshold surface, JOSA, 69, pp. 1340-1349 (1979)

However, in the moving image, the appearance of the fixed pattern noise changes according to the moving speed of the subject in the image. For this reason, solid pattern noise can be seen only by performing simple noise reduction depending on the moving speed of the subject in the image.
The present invention has been made in view of such a problem, and an object thereof is to generate a moving image in which fixed pattern noise is hardly perceived.

  The image processing apparatus according to the present invention includes a noise characteristic deriving unit that derives a noise characteristic in the moving image based on a condition relating to exposure when the moving image is captured, and a movement that acquires a moving speed of the subject in the moving image. Speed acquisition means; visual characteristic acquisition means for acquiring information representing visual sensitivity to light according to the moving speed of the subject in the moving image, acquired by the moving speed acquisition means; the moving image; When noise based on the noise characteristic is superimposed on the fixed pattern noise included in the moving image based on the noise characteristic derived by the noise characteristic deriving unit and the information acquired by the visual characteristic acquiring unit Fixed pattern noise perception amount to derive fixed pattern noise perception amount, which is an evaluation of the ease of perception of the fixed pattern noise Means out, on the basis of the fixed pattern fixed pattern noise perception values derived by the noise perceived amount deriving means, and having a noise reduction means for reducing noise included in the moving image.

  According to the present invention, it is possible to generate a moving image in which fixed pattern noise is hardly perceived.

It is a figure which shows the structure of a digital camera. It is a figure which shows the 1st example of a structure of an image process part. It is a figure showing a visual frequency response characteristic. It is a figure which shows the 1st example of a structure of a moving speed acquisition part. It is a figure which shows the structure of a fixed pattern noise characteristic acquisition part. It is a figure which shows the structure of a noise characteristic estimation part. It is a figure which shows the 1st example of a structure of a fixed pattern noise perception amount derivation | leading-out part. It is a figure which shows fan _l ((theta)). It is a figure which shows dom _k ((rho)). It is a figure which shows the 1st example of the spatial frequency area | region by which the band division | segmentation was carried out. It is a figure which shows the relationship between masking intensity | strength and a band division | segmentation image. It is a figure which shows the 1st example of a structure of a noise reduction control part. It is a figure which shows the 1st example of a structure of a noise reduction process part. It is a figure which shows the 2nd example of a structure of an image process part. It is a block diagram which shows the 2nd example of a structure of a moving speed acquisition part. It is a figure which shows the 2nd example of a structure of a noise reduction control part. It is a figure which shows the 2nd example of a structure of a noise reduction process part. It is a figure which shows the 2nd example of a structure of a fixed pattern noise perception amount deriving part. It is a figure which shows the 2nd example of the spatial frequency area | region by which the band division | segmentation was carried out.

Embodiments of the present invention will be described below with reference to the drawings.
(First embodiment)
First, the first embodiment will be described.
In the present embodiment, a fixed pattern noise perception amount that expresses the ease of perception of fixed pattern noise by a human is calculated. Here, the perceived amount of fixed pattern noise is a noise map obtained by mapping ease of perception of fixed pattern noise based on a noise sensitivity model. Based on the fixed pattern noise perception amount, processing for reducing noise of the captured image is performed. Here, regarding the noise sensitivity model, for example, there is a technique described in Non-Patent Document 1. In the present embodiment, a case where the noise sensitivity model described in Non-Patent Document 1 is used will be described as an example.

  In the conventional noise reduction process, there is a problem that fixed pattern noise becomes more conspicuous because random noise is reduced. In view of this, in the present embodiment, an optimum noise reduction process is performed in consideration of the frequency characteristics of fixed pattern noise and the frequency characteristics of random noise in accordance with changes in human visibility according to the movement of the subject in the image. As a result, it is possible to generate a moving image in which fixed pattern noise is less likely to be perceived by leaving some random noise for an arbitrary moving subject.

<System configuration>
FIG. 1 is a block diagram illustrating an example of a configuration of a digital camera (imaging device) including the image processing device of the present embodiment. In the present embodiment, the digital camera can capture a moving image.
The imaging unit 101 includes a zoom lens, a focus lens, a shake correction lens, an aperture, a shutter, an optical low-pass filter, an iR cut filter, a color filter, and a sensor (solid-state imaging device) such as a CMOS or CCD, and detects the amount of light of the subject. To do.
The A / D conversion unit 102 converts the light amount of the subject into a digital value.
The signal processing unit 103 performs demosaicing processing, white balance processing, gamma processing, and the like on the digital value converted by the A / D conversion unit 102 to generate a digital image.

The image processing unit 104 performs noise reduction processing on the digital image generated by the signal processing unit 103.
The encoder unit 105 performs processing for converting the digital image on which noise reduction processing has been performed into a video compression format such as JPEG.
A media interface (I / F) 106 is an interface for connecting to the media 107. The medium 107 is, for example, a hard disk, a memory card, a CF card, an SD card, or a USB memory.

  The CPU 108 is related to all the processes of each component of the digital camera, and sequentially reads the instructions stored in the ROM 109 and the RAM 110 and executes the processes. The CPU 108 controls each component of the digital camera via the system bus 115. The ROM 109 and the RAM 110 provide the CPU 108 with programs, data, work areas, and the like necessary for processing executed by the CPU 108.

The imaging system control unit 111 controls the imaging system instructed by the CPU 108, such as focusing, opening a shutter, and adjusting an aperture.
The operation unit 112 has buttons, a mode dial, and the like, and receives user instructions input via these buttons. User instructions include shooting settings such as ISO sensitivity setting, shutter speed setting, and F value setting. These shooting settings are reflected in the shooting conditions of the digital camera by the CPU 108 and stored in the RAM 110.

The D / A conversion unit 113 performs analog conversion on the digital image on which the noise reduction processing has been performed, and outputs the result as captured image data. The display unit 114 displays captured image data received from the D / A conversion unit 113. As the display unit 114, for example, a liquid crystal display is used.
The gyro sensor 116 is a sensor for detecting the angular velocity of the digital camera.

<Image processing unit 104>
FIG. 2 is a block diagram illustrating an example of the configuration of the image processing unit 104. In the present embodiment, the image processing unit 104 corresponds to an image processing device.
A digital image generated by the signal processing unit 103 is input from the terminal 201.
The moving speed acquisition unit 202 acquires the moving speed v _R [deg / sec] of the subject in the digital image when observing the digital image (moving image) in a standard observation environment. Specific movement speed acquisition processing will be described later. Even if the subject itself does not move, the subject moves in the digital image as the digital camera moves. In the following description, the moving speed of the subject in the digital image is referred to as the moving speed of the subject as necessary.

The visual characteristic data storage unit 203 stores visual frequency response characteristics corresponding to the moving speed v _{R of the} subject as a table or a relational expression.
The visual characteristic reference unit (visual characteristic acquisition unit) 204 is based on the moving speed v _{R of the} subject acquired from the moving speed acquisition unit 202, and the visual frequency when the human follows the moving object at the moving speed v _R. Get response characteristics.
FIG. 3 is a diagram showing changes in visual frequency response characteristics according to the moving speed of the subject. In this embodiment, the visual frequency response characteristic described in Non-Patent Document 2 is used. Specifically, the visual frequency response characteristic is defined by the following equations (1) to (4).

The movement speed v _{R of the} subject is acquired by the movement speed acquisition unit 202. U is a horizontal spatial frequency (horizontal frequency), and v is a vertical spatial frequency (vertical frequency). For example, s ₁ = 6.1, s ₂ = 7.3, p ₁ = 45.9, c ₀ = 0.6329, c ₁ = 0.8404, and c ₂ = 0.7986.

In FIG. 3, a visual frequency response characteristic 301 is a visual frequency response characteristic when the moving speed v _{R of the} subject is 0 [deg / sec], and a visual frequency response characteristic 302 is a moving speed v _{R of the} subject of 10. It is a visual frequency response characteristic when [deg / sec].
The visual frequency response characteristics are not limited to those shown in FIG. 3 (formulas (1) to (4)). The visual frequency response characteristic at the time of following vision may be a band pass characteristic in which the peak of contrast sensitivity shifts to the low frequency side as the movement of the subject increases. For example, other visual frequency response characteristics such as the visual frequency response characteristics described in Non-Patent Document 3 or actually measured data may be used as the visual frequency response characteristics.

A fixed pattern noise characteristic acquisition unit 205 acquires a characteristic of fixed pattern noise unique to the solid-state imaging device. Specific fixed pattern noise acquisition processing will be described later.
The noise characteristic estimation unit 206 estimates characteristics of noise generated in the imaging system. Specific noise characteristic estimation processing will be described later.
The fixed pattern noise perception amount deriving unit 207 derives a fixed pattern noise perception amount considering the following vision and the frequency characteristics of noise. The fixed pattern noise perception amount includes the fixed pattern noise characteristic acquired by the fixed pattern noise characteristic acquisition unit 205, the noise characteristic estimated by the noise characteristic estimation unit 206, and the visual frequency response acquired by the visual characteristic reference unit 204. Derived according to characteristics. A specific fixed pattern noise perception amount derivation process will be described later.

The noise reduction control unit 208 calculates a gain to multiply the noise reduction filter based on the fixed pattern noise perception amount derived by the fixed pattern noise perception amount deriving unit 207. The gain calculated by the noise reduction control unit 208 is transmitted to the noise reduction processing unit 209. Specific gain calculation processing will be described later.
The noise reduction processing unit 209 performs noise reduction processing on the digital image input from the signal processing unit 103 using the gain calculated by the noise reduction control unit 208. Specific noise reduction processing will be described later. The digital image subjected to the noise reduction process is output from the terminal 210.

<Movement speed acquisition unit 202>
FIG. 4 is a block diagram illustrating an example of the configuration of the moving speed acquisition unit 202. An example of processing for acquiring the moving speed v _{R of the} subject will be described with reference to FIG.
The imaging information acquisition unit 401 acquires various condition information from the RAM 110 when the digital image generated by the signal processing unit 103 is captured. Here, the various condition information at the time of imaging is, for example, focal length f [mm], sensor (solid-state imaging device) size w × h [mm], exposure time t [sec], and frame rate FR [fps]. . The acquisition method of various condition information at the time of imaging is not limited to this. For example, the condition information may be acquired from the exif information.

The gyro sensor information acquisition unit 402 acquires gyro sensor information at the time of capturing the digital image generated by the signal processing unit 103 from the gyro sensor 116. Here, the gyro sensor information is, for example, an angular velocity ω [deg / sec] of the digital camera.
The moving speed calculation unit 403 performs the following processing based on the various condition information acquired by the imaging information acquisition unit 401 and the gyro sensor information acquired by the gyro sensor information acquisition unit 402. That is, the moving speed calculation unit 403 calculates the moving speed v _R [deg / sec] of the subject in the digital image when the subject is not moving during imaging.
First, the moving speed calculation unit 403 calculates the angle of view θ _lens [deg] of the image pickup optical system of the image pickup unit 101 and the movement angle θ _gyro [deg] of the digital camera per second at the time of image pickup as follows (5 ) And (6).

At this time, if the subject is not moving, the movement angle theta _move (per unit of observation angle theta _view time change amount) per second observation angle theta in _view from the following equation (7) Calculated.

Here, it is assumed that the user observes the subject from a distance that is three times the vertical length (height) of the observation display as a standard observation distance. When the aspect ratio of the viewing angle of the observation display is 9:16, the viewing angle of _view θ _view is calculated from the following equation (8).

Therefore, the moving speed v _R [deg / sec] of the subject is defined by the following equation (9).

The moving speed calculation unit 403 calculates the moving speed v _R [deg / sec] of the subject based on the equations (5) to (9), and outputs it to the terminal 404.

<Fixed pattern noise characteristic acquisition unit 205>
FIG. 5 is a block diagram illustrating an example of the configuration of the fixed pattern noise characteristic acquisition unit 205. An example of processing for acquiring the characteristics of the fixed pattern noise will be described with reference to FIG.
The imaging condition acquisition unit 501 acquires from the RAM 110 imaging conditions at the time of imaging of the digital image generated by the signal processing unit 103. Here, the imaging condition is, for example, ISO sensitivity.
The fixed pattern noise characteristic storage unit 502 stores a relational expression or table indicating a relationship between the fixed pattern noise map and the ISO sensitivity.
The fixed pattern noise information acquisition unit 503 acquires the fixed pattern noise map FPN (x, y) corresponding to the imaging condition acquired by the imaging condition acquisition unit 501 from the fixed pattern noise characteristic storage unit 502 and outputs the fixed pattern noise map FPN (x, y) to the terminal 504. . The fixed pattern noise map FPN (x, y) is a fixed pattern noise image representing the magnitude of the fixed pattern noise at the position (pixel) at (x, y). Further, the fixed pattern noise map FPN (x, y) has the same size as the digital image i _in (x, y) generated by the signal processing unit 103, for example.

<Noise characteristic estimation unit (noise characteristic deriving unit) 206>
FIG. 6 is a block diagram illustrating an example of the configuration of the noise characteristic estimation unit 206. An example of processing for estimating noise characteristics will be described with reference to FIG.
A digital image i _in (x, y) generated by the signal processing unit 103 is input from a terminal 601. Here, x indicates a position (pixel) in the horizontal (horizontal) direction, and y indicates a position (pixel) in the vertical (vertical) direction. The digital image i _in (x, y) is, for example, a luminance value or a pixel value at a position represented by (x, y).
The average value calculation unit 602 calculates an average value avg _in (at each position (x, y)) of the digital image i _in (x, y), and transmits it to the noise information acquisition unit 604.
The imaging condition acquisition unit 603 acquires imaging conditions when the digital image i _in (x, y) is captured from the RAM 110. Here, the imaging condition is a condition related to exposure during imaging. In the present embodiment, the imaging condition acquisition unit 303 acquires ISO sensitivity, lens aperture value, and exposure time as imaging conditions.

The noise characteristic storage unit 605 stores the relationship between the imaging conditions, the average value avg _{in of} the digital image i _in (x, y), and the noise variance σ _n as a table or a relational expression.
The noise information acquisition unit 604 acquires the noise variance σ _n corresponding to the imaging condition acquired by the imaging condition acquisition unit 603 and the average value avg _in calculated by the average value calculation unit 602 from the noise characteristic storage unit 605. To do. The noise variance σ _n is output from the terminal 606 as noise characteristics.

<Fixed Pattern Noise Perception Deriving Unit 207>
FIG. 7 is a block diagram illustrating an example of the configuration of the fixed pattern noise perception amount deriving unit 207. An example of processing for deriving the fixed pattern noise perception amount will be described with reference to FIG.
A fixed pattern noise map FPN (x, y) acquired by the fixed pattern noise characteristic acquisition unit 205 is input from a terminal 701.
The noise variance σ _n obtained by the noise characteristic estimation unit 206 is input from the terminal 702.
The noise image generation unit 703 generates a white noise image n _w (x, y) having the same size as the fixed pattern noise map FPN (x, y) so that the noise variance is σ _n . Here, the white noise image n _w (x, y) represents the magnitude of white noise at the position (pixel) at (x, y). The noise image generation unit 703 generates a noise image n (x, y) by adding the fixed pattern noise map FPN (x, y) and the white noise image n _w (x, y). As described above, the noise image generation unit 703 generates an image showing white noise to be superimposed on the solid pattern noise by pseudo simulation. In addition, if the image which shows the noise superimposed on solid pattern noise can be produced | generated by pseudo simulation, it is not necessary to use white noise.

The Fourier transform unit 704 performs Fourier transform on each of the fixed pattern noise map FPN (x, y) and the white noise image n _w (x, y). Thereby, the spatial frequency information of the fixed pattern noise map FPN (x, y) and the white noise image n _w (x, y) is generated. Hereinafter, the result of Fourier transforming the fixed pattern noise map FPN (x, y) to be a function of spatial frequency is expressed as I _FPN (u, v). A result obtained by performing Fourier transform on the noise image n (x, y) to be a function of spatial frequency is denoted as N (u, v). Here, u is a spatial frequency in the horizontal direction (horizontal frequency), and v is a spatial frequency in the vertical direction (vertical frequency).

The spatial visual characteristic multiplication unit 705 receives the visual frequency response input from the terminal 711 for the fixed pattern noise map I _FPN (u, v) after the Fourier transform and the noise image N (u, v) after the Fourier transform. The characteristics CSF (ρ, v _R ) are respectively multiplied.

Hereinafter, the result of multiplying the fixed pattern noise map I _FPN (u, v) after Fourier transform by the visual frequency response characteristic CSF (ρ, v _R ) will be expressed as I _V (u, v). Further, the result of multiplying the noise image N (u, v) after Fourier transform by the visual frequency response characteristic CSF (ρ, v _R ) is denoted as N _V (u, v). I _V (u, v) represents a fixed pattern noise signal at a spatial frequency sensitive to human vision. N _V (u, v) represents a white noise signal at a spatial frequency sensitive to human vision.

The band dividing unit 706 calculates band division information Is _{k, l} (u, v) for the fixed pattern noise map expressed by the following equation (10). Further, the band dividing unit 706 calculates band division information Ns _{k, l} (u, v) for the noise image expressed by the following equation (11). Band division frequency information Is _{k, l} (u, v) divides a fixed pattern noise map I _V (u, v), which has been multiplied by Fourier transform and visual frequency response characteristics, into multiple spatial frequency bands Information. Band division frequency information Ns _{k, l} (u, v) is information _{obtained by} dividing a noise image N _V (u, v), which has been subjected to Fourier transform and multiplication of visual frequency response characteristics, for each of a plurality of spatial frequency bands. It is.

The band division filter cortex _{k, l} (u, v) is obtained by combining the characteristics in the radial direction and the characteristics in the circumferential direction in the spatial frequency domain. The following expressions (12) and (13) It is defined as an expression.

cortex_pol _{k, l} (ρ, θ) represents the band division filter cortex _{k, l} (u, v) in polar coordinates. In the equations (12) and (13), K is the number of divisions in the radial direction of the spatial frequency band, and L is the number of divisions in the circumferential direction of the spatial frequency band. In general, K = 6 and L = 6. The dom _k (ρ), base (ρ), and fan _l (θ) shown in the equation (12) are defined by the following equations (14) to (22).

FIG. 8 is a diagram schematically showing fan _l (θ) when L = 6. In FIG. 8, the horizontal axis is the angle, and the vertical axis is the response. FIG. 9 is a diagram schematically showing dom _k (ρ) when K = 6. In FIG. 8, the horizontal axis is the normalized frequency, and the vertical axis is the response. In FIG. 9, the horizontal axis is the normalized frequency, and the vertical axis is the response.
FIG. 10 is a diagram schematically showing a spatial frequency region divided by the band division filter cortex _{k, l} (u, v). In FIG. 10, the horizontal axis and the vertical axis are both normalized frequency spaces. In FIG. 10, the half-value frequency of each filter is indicated by a thick solid line.

  Here, human vision distinguishes whether the spatial frequency of the image is low frequency or high frequency, and the direction of the spatial frequency of the image (vertical direction, horizontal direction, diagonal direction). Consider and define these distinct spatial frequency regions as one region. This one area corresponds to the area surrounded by the thick line in FIG. In FIG. 10, when the fixed pattern noise map and the spatial frequency of the noise image are in the same region, white noise based on the noise image is difficult to recognize by human vision on the fixed pattern noise.

The inverse Fourier transform unit 707 performs inverse Fourier transform on the band division frequency information Is _{k, l} (u, v) and Ns _{k, l} (u, v), and is fixed into a plurality of spatial frequency bands. A pattern noise map and a noise image are generated respectively. Hereinafter, a band division image obtained as a result of the inverse Fourier transform of the band division frequency information Is _{k, l} (u, v) with respect to the fixed pattern noise map is _denoted as is _{k, l} (x, y). On the other hand, a band division image obtained as a result of the inverse Fourier transform of the band division frequency information Ns _{k, l} (u, v) for the noise image is denoted as ns _{k, l} (x, y). Each band division image is an image of each spatial frequency band divided by the band division filter cortex _{k, l} (u, v).

The noise mask unit (partial noise perception amount calculation unit) 708 is based on the band-divided image is _{k, l} (x, y) of the fixed pattern noise map, and the band-divided image ns _{k, l} (x, y) of the noise image. Apply masking to Thereby, a perceptual noise image P _{k, l} (x, y) (partial noise perception amount) that is a perceived noise image is calculated. The perceptual noise image P _{k, l} (x, y) is calculated for each spatial frequency band. Masking here refers to evaluating how hard a fixed pattern noise is perceived by humans by superimposing white noise on the fixed pattern noise.
An example of a specific process of the noise mask unit 708 will be described. First, the noise mask unit 708 calculates the masking strength Te _{k, l} (x, y) by the following equations (23) to (25). .

In the present embodiment, W = 6, Q = 0.7, b = 4, and s = 0.8 in the equations (23) to (25). FIG. 11 is a diagram illustrating an example of the relationship between the masking strength Te _{k, l} (x, y) and the band-divided image is _{k, l} (x, y) in this case. In FIG. 8, the horizontal axis represents the intensity of the band-divided image is _{k, l} (x, y) on a logarithmic scale. The vertical axis represents the intensity of the masking intensity Te _{k, l} (x, y) on a logarithmic scale. Noise mask unit 408, the masking strength Te _k, with _l (x, y), perceived noise image P _{k, l} a (x, y), and calculated by the following equation (26).

In the equation (26), m is an average value of the fixed pattern noise map i _FPN (x, y).
The fixed pattern noise perception amount calculation unit 709 synthesizes perceptual noise images P _{k, l} (x, y) for each spatial frequency band based on the following (Equation 27), and generates a fixed pattern noise perception amount P (x, y) is calculated. The fixed pattern noise perception amount P (x, y) is output from the terminal 710.

<Noise reduction control unit 208>
FIG. 12 is a block diagram illustrating an example of the configuration of the noise reduction control unit 208. An example of a process for controlling the gain for the noise reduction filter will be described with reference to FIG.
The fixed pattern noise perception amount P (x, y) derived by the fixed pattern noise perception amount deriving unit 207 is input from the terminal 1201.

The low-pass filter processing unit 1202 performs low-pass filter processing on the fixed pattern noise perception amount P (x, y). Hereinafter, the perceived amount of the fixed pattern noise that has been subjected to the low-pass filter processing is denoted as Pl (x, y).
The gain calculation unit 1203 calculates the gain g (x, y) for the noise reduction filter based on the fixed pattern noise perception amount Pl (x, y). Here, the gain g (x, y) is calculated from the following equation (28). However, the calculation method of the gain g (x, y) is not limited to the following equation (28).
g (x, y) = k ₁ × Pl (x, y) + k ₂ (28)
In equation (28), k ₁ and k ₂ are such that the gain g (x, y) decreases when the fixed pattern noise perception amount Pl (x, y) is large, and the fixed pattern noise perception amount Pl (x, y). Is a linear expression for increasing the gain g (x, y) when is small. Accordingly, noise reduction can be weakened when the fixed pattern noise perception amount Pl (x, y) is large, and masking to the fixed pattern noise perception amount Pl (x, y) can be strengthened. The gain calculation unit 1203 outputs the calculated gain g (x, y) to the terminal 1204.

<Noise reduction processing unit 209>
FIG. 13 is a block diagram illustrating an example of the configuration of the noise reduction processing unit 209. An example of a process for reducing noise _in the digital image i _in (x, y) generated by the signal processing unit 103 will be described with reference to FIG.
A digital image i _in (x, y) generated by the signal processing unit 103 is input from a terminal 1301.
The filter coefficient setting unit 1302 sets the filter coefficient of the noise reduction filter stored in the ROM 109.
A gain g (x, y) calculated by the noise reduction control unit 208 is input from a terminal 1303.
The multiplier 1304 multiplies the filter coefficient set by the filter coefficient setting unit 1302 and the gain g (x, y) input from the terminal 1303, and outputs the result to the noise reduction processing unit 1305 as a noise reduction coefficient.
The noise reduction processing unit 1305 performs noise reduction processing on the digital image i _in (x, y) using the noise reduction coefficient calculated by the multiplication unit 1304. The digital image that has been subjected to the noise reduction process is output from a terminal 1306.

As described above, in the present embodiment, the noise variance σ _n is derived based on the exposure condition during imaging, and a noise image is generated by superimposing the white noise corresponding to the noise variance σ _n on the fixed pattern noise. . From each of the fixed pattern noise map and the noise image, information on spatial frequency sensitive to human vision is extracted according to the moving speed v _{R of the} subject and divided into a plurality of spatial frequency bands. The noise image divided into multiple spatial frequency bands is masked based on the fixed pattern noise map divided into multiple spatial frequency bands and then synthesized to calculate the fixed pattern noise perception amount P (x, y) To do. When the fixed pattern noise perception amount P (x, y) is large, the fixed pattern noise is easily perceived, and thus the noise reduction amount for the digital image i _in (x, y) is suppressed. In this way, the fixed pattern noise perception amount P (x, y) that expresses the ease of perception of fixed pattern noise is calculated, and noise reduction is controlled so as to reduce this. Therefore, it is possible to realize image processing that suppresses enhancement of fixed pattern noise due to excessive noise reduction. Therefore, it is possible to generate a moving image in which fixed pattern noise is hardly perceived according to the moving speed of the subject.

(Second Embodiment)
Next, a second embodiment will be described. In the first embodiment, the case where the moving speed v _R of the subject in the image is calculated using the gyro sensor information has been described as an example. In this way, when the subject is also moving, the moving speed v _R of the subject in the image may not be calculated correctly. Therefore, in the present embodiment, the fixed pattern noise is suppressed corresponding to the movement of the subject by calculating the moving speed v _{R of the} subject using the motion vector in the image.
As described above, the present embodiment and the first embodiment are mainly different in processing and configuration when calculating the moving speed v _{R of the} subject. Therefore, in the description of the present embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals as those in FIGS.

<Image Processing Unit 1400>
FIG. 14 is a block diagram illustrating an example of the configuration of the image processing unit 1400. The image processing unit 1400 illustrated in FIG. 14 is replaced with the image processing unit 104 illustrated in FIG.
The digital image generated by the signal processing unit 103 is also input from the terminal 1401 to the moving speed acquisition unit 1402 in addition to the noise characteristic estimation unit 206 and the noise reduction processing unit 209.
The moving speed acquisition unit 1402 determines the moving speed v _R [deg / sec] of the subject in the digital image when observing the digital image in a standard observation environment based on the digital image generated by the signal processing unit 103. Get.
The configuration and processing from the visual characteristic data storage unit 203 to the noise reduction processing unit 209 are the same as those in the first embodiment.

<Movement speed acquisition unit 1402>
FIG. 15 is a block diagram illustrating an example of the configuration of the moving speed acquisition unit 1402. An example of processing for acquiring the moving speed v _{R of the} subject will be described with reference to FIG.
A digital image generated by the signal processing unit 103 is input from a terminal 1501. A digital image input from the terminal 1501 is written into the frame memory 1502.
The motion vector calculation unit 1503 receives a digital image of the current frame input from the terminal 1501 and a delayed digital image (for example, a digital image one frame before) input from the frame memory 1502, respectively. ). The size of the block to be divided is, for example, 128 × 128 pixels or 256 × 256 pixels. The motion vector calculation unit 1503 calculates a motion component d [pixel] for each block and inputs the motion component d [pixel] to the movement speed calculation unit 1504.

The moving speed calculation unit 1504 acquires information on the frame rate FR [fps] stored in the RAM 110. Then, the moving speed calculation unit 1504 calculates the moving speed v _R [deg / sec] of the subject based on the frame rate FR [fps] and the motion component d calculated by the motion vector calculation unit 1503. If the user is observing the subject from a distance that is three times the vertical length (height) of the observation display, which is a standard observation distance, the moving speed v _R [deg / sec] of the subject is as follows: (29). It is assumed that the display to be observed is FHD.

The moving speed calculation unit 1504 outputs to the terminal 1505 the moving speed v _R [deg / sec] of the subject for each block, which is calculated by calculating the equation (29).
As described above, in the present embodiment, it is possible to perform noise reduction that makes it difficult to perceive fixed pattern noise even when the subject itself moves. In addition, the moving speed v _R of the subject can be calculated for each subject. Thereby, it is possible to perform noise reduction that makes it difficult to perceive fixed pattern noise for each subject.

(Third embodiment)
Next, a third embodiment will be described. In the first and second embodiments, the case where the digital image is corrected so as to make it difficult to perceive the fixed pattern noise by controlling the gain with respect to the filter coefficient of the noise reduction filter has been described as an example. In this method, since the spatial frequency band in which noise exists cannot be suppressed, there is a possibility that noise may remain from the assumption. Therefore, in this embodiment, the perceptual amount of fixed pattern noise is controlled by controlling the spatial frequency band for suppressing noise.

  Thus, the present embodiment is different from the first and second embodiments mainly in the configuration and processing for suppressing noise. Specifically, the present embodiment is different from the first and second embodiments in the configurations and processes of the noise reduction control unit 208 and the noise reduction processing unit 209 of the image processing units 104 and 1400. Therefore, in the description of this embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals as those in FIGS.

<Noise Reduction Control Unit 1600>
FIG. 16 is a block diagram illustrating an example of the configuration of the noise reduction control unit 1600. An example of a process for controlling the cutoff frequency of the noise reduction process will be described with reference to FIG.
The fixed pattern noise perception amount P (x, y) derived by the fixed pattern noise perception amount deriving unit 207 is input from the terminal 1201.

The low-pass filter processing unit 1202 applies a low-pass filter to the fixed pattern noise perception amount P (x, y). Hereinafter, the perceived amount of the fixed pattern noise that has been subjected to the low-pass filter processing is denoted as Pl (x, y).
The cut-off frequency calculation unit 1601 calculates a cut-off frequency for the noise reduction process based on the fixed pattern noise perception amount Pl (x, y). Here, the cut-off frequency fc (x, y) is calculated from the following equation (30). However, the calculation method of the cut-off frequency fc (x, y) is not limited to the following equation (30).
fc (x, y) = k ₁ × Pl (x, y) + k ₂ (30)
k ₁ and k ₂ are when the cut-off frequency fc (x, y) is large when the fixed pattern noise perception amount Pl (x, y) is large, and when the fixed pattern noise perception amount Pl (x, y) is small. This is a linear expression for reducing the cut-off frequency fc (x, y). As a result, the cutoff frequency of noise reduction can be increased when the fixed pattern noise perception amount Pl (x, y) is large, and the masking to the fixed pattern noise perception amount Pl (x, y) can be strengthened. . Cutoff frequency calculation section 1601 outputs calculated cutoff frequency fc (x, y) to terminal 1204.

<Noise Reduction Processing Unit 1700>
FIG. 17 is a block diagram illustrating an example of the configuration of the noise reduction processing unit 1700. An example of processing for reducing noise _in the digital image i _in (x, y) generated by the signal processing unit 103 will be described with reference to FIG.
A digital image i _in (x, y) generated by the signal processing unit 103 is input from a terminal 1701.
The cutoff frequency fc (x, y) calculated by the fixed pattern noise perception amount deriving unit 207 is input from the terminal 1702.

Based on the cut-off frequency fc (x, y) input from the terminal 1702, the region dividing unit 1703 cuts off the cut-off frequency fc (x, x, y in each region of an appropriate size in the digital image i _in (x, y). The representative value of y) is calculated. The representative value of the cutoff frequency fc (x, y) in each region may be an average value or a value with the highest frequency.
The noise reduction coefficient setting unit 1704 selects one of the filter coefficient candidates stored in the ROM 109 based on the representative value of the cutoff frequency fc (x, y) for each region, and selects the selected filter coefficient as a noise. Set as a reduction factor. The ROM 109 stores a relational expression or table that represents the relationship between the filter coefficient and the representative value of the cutoff frequency fc (x, y). The noise reduction coefficient setting unit 1704 sets a filter coefficient corresponding to the representative value of the cutoff frequency fc (x, y) as a noise reduction coefficient.
The noise reduction processing unit 1705 performs noise reduction processing on the digital image i _in (x, y) using the noise reduction coefficient set by the noise reduction coefficient setting unit 1704. The digital image that has been subjected to the noise reduction process is output from a terminal 1706.

As described above, in this embodiment, the amount of noise reduction for the digital image i _in (x, y) is controlled in accordance with the representative value of the cutoff frequency fc (x, y) in each region. Therefore, the amount of noise reduction can be controlled by the spatial frequency. Thereby, noise can be controlled with higher accuracy.

(Fourth embodiment)
Next, a fourth embodiment will be described. In the first to third embodiments, the case where a filter known as a cortex filter is used for the processing in the band dividing unit 706 of the fixed pattern noise perception amount deriving unit 207 has been described as an example. However, since the cortex filter is a filter having a very special frequency characteristic, calculation processing becomes heavy. Therefore, in this embodiment, the arithmetic processing is lightened by replacing the cortex filter with a simple band division filter in which the vertical and horizontal filters in the spatial frequency domain are combined. As described above, the present embodiment and the first to third embodiments are mainly different in configuration and processing due to different band division filters. Specifically, the configuration and processing of the fixed pattern noise perception amount deriving unit 207 are different between the present embodiment and the first to third embodiments. Therefore, in the description of this embodiment, the same parts as those in the first to third embodiments are denoted by the same reference numerals as those in FIGS.

<Noise perception amount calculation processing>
FIG. 18 is a diagram illustrating a configuration of the fixed pattern noise perception amount deriving unit 1800.
A fixed pattern noise map FPN (x, y) acquired by the fixed pattern noise characteristic acquisition unit 205 is input from a terminal 701.
The noise variance σ _n obtained by the noise characteristic estimation unit 206 is input from the terminal 702.
The noise image generation unit 703 generates a white noise image n _w (x, y) having the same size as the fixed pattern noise map FPN (x, y) so that the noise variance is σ _n . Then, the noise image generation unit 703 generates the noise image n (x, y) by adding the fixed pattern noise map FPN (x, y) and the white noise image n _w (x, y).
The band division unit 1801 divides the fixed pattern noise map FPN (x, y) using a band division filter for each of a plurality of spatial frequency bands, and band division frequency information Is _{k, l} (u , v). Similarly, the band dividing unit 1801 uses a band filter to divide the noise image n (x, y) into a plurality of spatial frequency bands, and band division frequency information Ns _{k, l} (u, v) for the noise image. Is generated. The band division frequency information Is _{k, l} (u, v) and Ns _{k, l} (u, v) can be obtained by the following equations (31) and (32).

In equations (31) and (32), k is an index in the horizontal (lateral) direction, and l is an index in the vertical (vertical) direction. Further, in the equations (31) and (32), the band division filter filterbank _{k, l} (u, v) is defined as the following equation (33).

In Equation (33), K is the number of divisions in the spatial frequency band in the horizontal (lateral) direction, and L is the number of divisions in the spatial frequency band in the vertical (vertical) direction. In this embodiment, K = 6 and L = 6.
FIG. 19 is a diagram schematically showing a spatial frequency region divided into a plurality of parts by the band division filter filterbank _{k, l} (u, v). In FIG. 19, both the horizontal axis and the vertical axis are normalized frequency spaces. In FIG. 19, the half-value frequency of each filter is indicated by a thick solid line.
The noise mask unit 708 performs masking on the band division frequency information Ns _{k, l} (x, y) for the noise image based on the band division image is _{k, l} (x, y) for the fixed pattern noise map. Thereby, a perceptual noise image P _{k, l} (x, y), which is a noise image perceived by a human, is calculated. The perceptual noise image P _{k, l} (x, y) is calculated for each spatial frequency band.
The fixed pattern noise perception amount calculation unit 709 synthesizes perceptual noise images P _{k, l} (x, y) for each spatial frequency band, and calculates a fixed pattern noise perception amount P (x, y).

As described above, in the present embodiment, it is possible to realize image processing that makes it difficult to perceive fixed pattern noise by simple processing without using a special filter such as a cortex filter.
In the present embodiment, band division is performed by performing multiplication in the frequency domain. However, it is also possible to realize band division by performing a convolution operation in the spatial domain. In addition, a known band division filter such as a wavelet can be applied to the present embodiment. In the present embodiment, the case of changing the band division filter of the first embodiment has been described as an example, but the present embodiment can also be applied to the second and third embodiments.

  Each of the above-described embodiments is merely an example of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

(Other examples)
The present invention is also realized by executing the following processing. That is, first, software (computer program) for realizing the functions of the above embodiments is supplied to a system or apparatus via a network or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the computer program.

  101 imaging unit, 104 image processing unit, 108 CPU

Claims (15)

A noise characteristic deriving unit for deriving a characteristic of noise in the moving image based on a condition relating to exposure when the moving image is captured;
Moving speed acquisition means for acquiring the moving speed of the subject in the moving image;
Visual characteristic acquisition means for acquiring information representing visual sensitivity to light according to the movement speed of the subject in the moving image, acquired by the movement speed acquisition means;
Based on the moving image, the characteristic of the noise derived by the noise characteristic deriving unit, and the information acquired by the visual characteristic acquiring unit, the fixed pattern noise included in the moving image is based on the characteristic of the noise. Fixed pattern noise perception amount derivation means for deriving a fixed pattern noise perception amount, which is a value obtained by evaluating the ease of perception of the fixed pattern noise when noise is superimposed;
An image processing apparatus comprising: noise reduction means for reducing noise contained in the moving image based on the fixed pattern noise perception amount derived by the fixed pattern noise perception amount deriving means. The fixed pattern noise perception amount derivation means includes:
A noise image generating means for generating a noise image, which is an image in which the noise of the characteristic derived by the noise characteristic deriving means is superimposed on the fixed pattern noise;
Conversion means for converting the fixed pattern noise image representing the fixed pattern noise and the noise image into a function of spatial frequency, respectively;
Spatial visual characteristic multiplying means for multiplying the fixed pattern noise image and the noise image converted into a function of the spatial frequency by a spatial visual characteristic representing a relationship between contrast sensitivity and spatial frequency, respectively.
Band division means for dividing the fixed pattern noise image multiplied by the spatial visual characteristic and the noise image for each spatial frequency band;
Based on the fixed pattern noise image divided for each spatial frequency and the noise image divided for each spatial frequency band, a partial noise perception amount that is the fixed pattern noise perception amount for each spatial frequency band is determined. A partial noise perception amount calculating means for calculating;
Fixed pattern noise perception amount calculation means for calculating the fixed pattern noise perception amount by combining the partial noise perception amount calculated by the partial noise perception amount calculation means;
The image processing apparatus according to claim 1, further comprising:   The band dividing means is a band dividing filter whose spatial frequency characteristic is defined by a combination of a circumferential characteristic and a radial characteristic in the spatial frequency domain, or the spatial frequency characteristic is a vertical frequency in the spatial frequency domain. 3. The fixed pattern noise image and the noise image are divided for each spatial frequency band using a band division filter defined by a combination of a directional characteristic and a horizontal characteristic. Image processing apparatus.   The said moving speed acquisition means derives | leads-out the moving speed of the to-be-photographed object in the said moving image based on the angular velocity of the imaging device at the time of the said moving image being imaged. The image processing apparatus according to item.   The moving speed acquisition unit derives a motion vector in the moving image, and derives a moving speed of a subject in the moving image based on the derived motion vector. The image processing apparatus according to item. The noise reduction means includes
Noise reduction control means for deriving a gain for a filter for reducing noise included in the moving image based on the fixed pattern noise perception amount derived by the fixed pattern noise perception amount deriving means;
Noise reduction processing means for multiplying the filter coefficient of the filter by the gain derived by the noise reduction control means, and performing processing for reducing noise included in the moving image based on the filter coefficient multiplied by the gain; The image processing apparatus according to claim 1, wherein the image processing apparatus includes: The noise reduction means includes
Noise reduction control means for deriving a cutoff frequency in the moving image based on the fixed pattern noise perception amount derived by the fixed pattern noise perception amount deriving means;
6. Noise reduction processing means for performing processing for reducing noise contained in the moving image based on a cut-off frequency derived by the noise reduction control means. The image processing apparatus according to item 1. A noise characteristic deriving step for deriving a noise characteristic in the moving image based on a condition relating to exposure when the moving image is captured;
A moving speed acquisition step of acquiring a moving speed of a subject in the moving image;
A visual characteristic acquisition step of acquiring information representing visual sensitivity to light according to a moving speed of a subject in the moving image acquired by the moving speed acquisition step;
Based on the moving image, the noise characteristic derived by the noise characteristic deriving step, and the information acquired by the visual characteristic acquiring step, the fixed pattern noise included in the moving image is based on the noise characteristic. A fixed pattern noise perception amount derivation step for deriving a fixed pattern noise perception amount, which is a value obtained by evaluating the ease of perception of the fixed pattern noise when noise is superimposed;
A noise reduction step of reducing noise included in the moving image based on the fixed pattern noise perception amount derived by the fixed pattern noise perception amount derivation step. The fixed pattern noise perception amount derivation step includes:
A noise image generation step for generating a noise image, which is an image in which the noise of the characteristic derived by the noise characteristic derivation step is superimposed on the fixed pattern noise;
A conversion step of converting the fixed pattern noise image representing the fixed pattern noise and the noise image into a function of spatial frequency, respectively;
A spatial visual characteristic multiplication step of multiplying the fixed pattern noise image and the noise image converted into a function of the spatial frequency by a spatial visual characteristic representing a relationship between contrast sensitivity and spatial frequency, respectively.
A band dividing step of dividing the fixed pattern noise image multiplied by the spatial visual characteristic and the noise image into spatial frequency bands, respectively.
Based on the fixed pattern noise image divided for each spatial frequency and the noise image divided for each spatial frequency band, a partial noise perception amount that is the fixed pattern noise perception amount for each spatial frequency band is determined. A partial noise perception amount calculating step to calculate;
A fixed pattern noise perception amount calculation step of calculating the fixed pattern noise perception amount by combining the partial noise perception amount calculated by the partial noise perception amount calculation step;
The image processing method according to claim 8, further comprising:   In the band dividing step, a spatial frequency characteristic is defined by a combination of a circumferential characteristic and a radial characteristic in the spatial frequency domain, or a spatial frequency characteristic is a vertical frequency in the spatial frequency domain. 10. The fixed pattern noise image and the noise image are divided for each spatial frequency band using a band division filter defined by a combination of a directional characteristic and a lateral characteristic. Image processing method.   11. The moving speed acquisition step of deriving a moving speed of a subject in the moving image based on an angular velocity of an imaging device when the moving image is captured. The image processing method according to item.   11. The moving speed acquisition step of deriving a motion vector in the moving image and deriving a moving speed of a subject in the moving image based on the derived motion vector. The image processing method according to item. The noise reduction step includes
A noise reduction control step for deriving a gain for a filter for reducing noise included in the moving image based on the fixed pattern noise perception amount derived by the fixed pattern noise perception amount deriving step;
A noise reduction processing step of multiplying a gain derived by the noise reduction control step by a filter coefficient of the filter, and performing a process of reducing noise included in the moving image, based on the filter coefficient multiplied by the gain; The image processing method according to claim 8, wherein the image processing method includes: The noise reduction step includes
A noise reduction control step of deriving a cutoff frequency in the moving image based on the fixed pattern noise perception amount derived by the fixed pattern noise perception amount deriving step;
13. A noise reduction processing step of performing a process of reducing noise included in the moving image based on the cut-off frequency derived by the noise reduction control step. 2. The image processing method according to item 1.   A program for causing a computer to execute each step of the image processing method according to any one of claims 8 to 14.

Images