JP6642570B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to a technique for reducing noise included in an image.

  A technique for reducing noise included in an image is an indispensable technique not only for reproducing a captured image more clearly but also for preventing noise enhancement in various image processing. In the following, noise refers to additive white Gaussian noise.

  In order to realize effective noise removal, information on the intensity (variance and standard deviation) of the noise included in the image to be processed is required in addition to improving the performance of the noise removal technology. When the intensity of the noise included in the processing target image is estimated to be small, the noise remains in the image resulting from the noise removal. On the other hand, if the noise intensity is largely estimated, the image resulting from the noise removal becomes blurred, and the texture and edges become unclear. That is, there is a need for a technique for more accurately estimating the intensity of noise included in an image to be processed.

  As a technique for estimating the intensity of noise included in an image, the following method is generally considered. The first method is to measure the relationship between the parameters of the camera gain and the sensor noise in advance when the camera sensor is known, and make a table. This method estimates the noise intensity in a captured image by referring to a table during actual operation. The second is a method of extracting a region where no edge or texture exists from an image, and regarding the fluctuation of the pixel value in the extracted region as being derived from noise. In this method, the variance of the pixel values in the area is regarded as the noise intensity of the target image. A third method is to use two images that are temporally continuous at short time intervals in a situation where the camera is fixed. In this method, a value obtained by dividing a variance of pixel values of a difference image between two images by two is regarded as a noise intensity.

  A related technique for estimating the intensity of noise included in an image is described in Patent Document 1. This related technique calculates a difference between corresponding pixels between a first image and a second image that is continuous with the first image. Then, the related technique detects a noise level in the first image based on a variance of a difference corresponding to each pixel forming a flat region in the first image. Then, the related technique detects a motion region in the first image based on the variance of the difference corresponding to all pixels forming the first image and the detected noise level.

  Another related technique for estimating the intensity of noise included in an image is described in Patent Document 2. This related technique calculates a noise level within a frame and a noise level between frames. This related technique selects a noise level in a frame for a moving part in an image. This related technique selects a noise level between frames for a static portion in an image.

JP 2012-231389 A JP 2013-258513 A

  However, the above-mentioned general technology and related technologies have the following problems.

  In the first general method, it is necessary to measure a noise variance table for each camera. For this reason, there is a problem that this method cannot be applied when the information of the camera sensor cannot be obtained. In addition, this method has a problem in that if a table is generated for each sensor of a camera in consideration of individual differences between sensors, an enormous work cost is required in advance.

  Further, the second general method adds the variance of the pixel value of the texture to the variance of the noise for an image in which the entire surface of the target image is covered with the texture. In such a case, this method has a problem that the variance value is estimated to be larger than the true value, and accurate noise intensity estimation cannot be performed.

  Further, the third general method cannot be applied when the camera moves. In addition, this method is used even when the camera is fixed, when there is a moving object in the image, or when the image fluctuates greatly due to scene change, illumination fluctuation of the shooting environment, fluctuation of camera gain, etc. Estimates the variance value larger than the true value. In such a case, this method has a problem that accurate noise intensity estimation cannot be performed.

  In the related art described in Patent Document 1, the noise level is calculated using a flat area excluding the influence of texture and edges. However, since the noise level is calculated to detect a moving area, the noise level is calculated. In calculating the level, the influence of a moving area is not taken into account. For this reason, this related art has a problem that when a moving area is included in an image, accurate noise intensity estimation cannot be performed.

  Further, the related art described in Patent Literature 2 does not disclose taking into account the influence of a moving area when calculating an inter-frame noise level applied to a still part. For this reason, this related technique has a problem that when a moving area is included in an image, it is not possible to accurately estimate a noise level for a stationary portion.

  The present invention has been made to solve the above problems. That is, an object of the present invention is to provide a technique for more accurately estimating the intensity of noise even in an image including a texture or a region with a lot of motion.

  In order to achieve the above object, an image processing apparatus according to the present invention is configured such that, in a frame image included in a video, a change in pixel value is a variance of noise in the frame image based on a pixel value belonging to a flat area. An intra-frame noise variance estimating unit for estimating an intra-frame noise variance, and a difference image representing a difference between the frame image and a neighboring frame image temporally close to the frame image, wherein the difference between the frame image and the neighboring frame image is An inter-frame noise variance estimating means for estimating an inter-frame noise variance, which is a variance of noise between the frame image and the adjacent frame image, based on a pixel value belonging to a non-moving area, and the intra-frame noise variance and the frame The frame image based on the smaller of the inter-noise variances. Comprising a noise intensity determination means for determining the intensity of that noise, the.

  Further, in the image processing method of the present invention, in the frame image included in the video, based on the pixel value belonging to a flat area where the change of the pixel value is a flat area, the intra-frame image is a variance of noise in the frame image. Estimating the noise variance, in the difference image representing the difference between the frame image and the adjacent frame image temporally adjacent to the frame image, the pixel value belonging to an area where there is no motion between the frame image and the adjacent frame image The inter-frame noise variance, which is the variance of noise between the frame image and the adjacent frame image, based on the smaller of the intra-frame noise variance and the inter-frame noise variance. Determine the intensity of the

Further, the image processing program according to the present invention estimates an intra-frame noise variance, which is a variance of noise in the frame image, based on a pixel value belonging to a flat region where a change in pixel value is flat in a frame image included in the video. In a noise variance estimating step, and a difference image representing a difference between the frame image and a nearby frame image temporally adjacent to the frame image, wherein the difference image belongs to a region where there is no motion between the frame image and the nearby frame image. An inter-frame noise variance estimating step of estimating an inter-frame noise variance, which is a variance of noise between the frame image and the adjacent frame image, based on a pixel value; and a small one of the intra-frame noise variance and the inter-frame noise variance. Of the frame image based on the A noise intensity determining step of determining the intensity of the figure, Ru cause the computer to execute an apparatus.

  The present invention can provide a technique for more accurately estimating the intensity of noise even in an image including a texture or an area with a lot of motion.

FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a hardware configuration of the image processing apparatus according to the first embodiment of the present invention. 5 is a flowchart illustrating an operation of the image processing apparatus according to the first embodiment of the present invention. FIG. 11 is a block diagram illustrating a configuration of an image processing device according to a second embodiment of the present invention. FIG. 13 is a block diagram illustrating a detailed configuration of an intra-frame noise variance estimating unit according to a second embodiment of the present invention. FIG. 11 is a block diagram illustrating a detailed configuration of an inter-frame noise variance estimating unit according to a second embodiment of the present invention. 11 is a flowchart illustrating an operation of the image processing apparatus according to the second embodiment of the present invention for estimating noise variance in a frame. 11 is a flowchart illustrating an operation of the image processing apparatus according to the second embodiment of the present invention for estimating an inter-frame noise variance. FIG. 13 is a block diagram illustrating a configuration of an image processing device according to a third embodiment of the present invention. FIG. 14 is a block diagram illustrating a detailed configuration of an inter-frame noise variance estimating unit according to a third embodiment of the present invention. 13 is a flowchart illustrating an operation of the image processing apparatus according to the third embodiment of the present invention for estimating an inter-frame noise variance. FIG. 14 is a block diagram illustrating a configuration of an image processing apparatus according to a fourth embodiment of the present invention. 13 is a flowchart illustrating an operation of the image processing apparatus according to the fourth embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First Embodiment)
FIG. 1 shows a functional block configuration of an image processing apparatus 1 according to a first embodiment of the present invention. 1, the image processing apparatus 1 includes an intra-frame noise variance estimating unit 11, an inter-frame noise variance estimating unit 12, and a noise intensity determining unit 13. The image processing apparatus 1 inputs an image of a frame number t (frame image u _t ) included in the video and an image of a frame number t-1 adjacent to the frame of the frame number t (proximal frame image u _t-1 ). To get as The image processing apparatus 1, using these two images, and outputs performs estimation of noise intensity sigma ² _t included in the frame image u _t.

The frame image u _t-1 is preferably that are continuous in time before or after the frame image u _t in the video may not be contiguous. Proximity frame image u _t-1 has only to satisfy the predetermined temporal proximity to the frame image u _t.

  Here, the image processing apparatus 1 can be configured by hardware elements as shown in FIG. 2, the image processing apparatus 1 is configured by a computer device including a CPU (Central Processing Unit) 1001 and a memory 1002. The memory 1002 includes a random access memory (RAM), a read only memory (ROM), an auxiliary storage device (such as a hard disk), and the like. The CPU 1001 reads and executes a computer program stored in the memory 1002. In this case, each functional block of the image processing apparatus 1 is configured by the CPU 1001 that reads and executes a computer program stored in the memory 1002. Note that the hardware configuration of the image processing apparatus 1 and its functional blocks is not limited to the above-described configuration.

The image processing device 1 is connected to the image input device 901. For example, the image input device 901 is configured by an imaging device such as a camera or a scanner, or an image database or the like in which image data is stored by being captured by the imaging device. The image processing device 1 and the image input device 901 may be connected by a peripheral device connection interface, or may be connected via a network. The image processing apparatus 1, from the image input unit 901, a frame image u _t and close the frame image u _t-1 described above is input.

The image processing device 1 is connected to the output device 902. For example, the output device 902 is configured by a display, a printer, or a storage medium such as a hard disk or a memory card that stores output information. The image processing device 1 and the output device 902 may be connected by a peripheral device connection interface or may be connected via a network. The image processing device 1 outputs the above-described noise intensity σ ² _t to the output device 902.

Frame noise variance estimation unit 11, from the frame image u _t, change in pixel value is extracted flat planar region. Then, the intra-frame noise variance estimation unit 11 based on the extracted pixel values belonging to a flat area, estimates the variance of the noise in the frame image u _t (frame in the noise variance).

Interframe noise variance estimation unit 12, based on the difference image representing the difference between the frame image u _t and close the frame image u _t-1, no motion between the frame images u _t and close the frame image u _t-1 region Is extracted. Then, the inter-frame noise variance estimation unit 12 based on the extracted pixel values of the difference images belonging to the area without motion, dispersion of the frame image u _t and close the frame image u _t-1 noise during (inter noise variance ).

Noise intensity determination unit 13, based on the smaller of the intra-frame noise variance and the inter-frame noise variance, determines the noise intensity sigma ² _t in the frame image u _t. Then, the noise intensity determination unit 13 outputs the determined noise intensity σ ² _t .

  The operation of the image processing apparatus 1 configured as described above will be described with reference to FIG.

First, the image processing apparatus 1 acquires the frame image _{u t} and close the frame image _{u t-1} included in the image (step S1).

Next, the intra-frame noise variance estimation unit 11, from the frame image u _t, change in pixel value is extracted flat planar region. Then, the intra-frame noise variance estimation unit 11 based on the extracted pixel values belonging to a flat area, estimates the frame noise variance in the frame image u _t (Step S2).

Then, the interframe noise variance estimation unit 12, based on the difference image representing the difference between the frame image u _t and close the frame image u _t-1, motion between the frame images u _t and close the frame image u _t-1 Extract the region without the. Then, the inter-frame noise variance estimation unit 12 based on the extracted pixel values of the motionless difference images belonging to the region, estimates the inter-frame noise variance in the frame image u _t and close the frame image u _t-1 between ( Step S3).

Next, the noise intensity determination unit 13 determines and outputs the noise intensity σ ² _t in the frame image based on the smaller of the intra-frame noise variance and the inter-frame noise variance (step S4).

  Thus, the operation of the image processing apparatus 1 ends.

  Next, effects of the first exemplary embodiment of the present invention will be described.

  The image processing apparatus according to the first embodiment of the present invention can more accurately estimate the noise intensity even for an image including a texture or an area with a lot of motion.

  The reason will be described. In the present embodiment, the intra-frame noise variance estimating unit estimates the intra-frame noise variance using the pixel values of the flat region in the frame image. Further, the inter-frame noise variance estimating unit estimates the inter-frame noise variance by using the pixel values of the difference image between the frame image and the adjacent frame image in the area where there is no motion. Then, the noise intensity determination unit determines the noise intensity of the frame image based on the smaller of the intra-frame noise variance and the inter-frame noise variance.

  Here, in the case of a frame image containing many textures, the noise variance in the frame becomes larger than the true value. In this case, in the present embodiment, if there is no motion in the video, the noise intensity of the frame image is reduced based on the inter-frame noise variance that is smaller than the intra-frame noise variance and is close to the true value. It can be estimated with high accuracy. Further, between frame images including moving objects, and between frame images in which the entire screen greatly fluctuates due to scene changes, illumination fluctuations, camera gain fluctuations, the inter-frame noise variance becomes larger than the true value. In this case, in the present embodiment, if a flat region exists in the image, the noise intensity of the frame image is increased based on the intra-frame noise variance that is smaller than the inter-frame noise variance and is close to the true value. It can be estimated with high accuracy.

(Second embodiment)
Next, a second embodiment of the present invention will be described in detail with reference to the drawings. In each of the drawings referred to in the description of the present embodiment, steps having the same configuration and the same operation as those in the first embodiment of the present invention are denoted by the same reference numerals, and detailed description in the present embodiment will be given. Description is omitted.

First, FIG. 4 shows a configuration of an image processing apparatus 2 according to a second embodiment of the present invention. 4, the image processing apparatus 2 includes an intra-frame noise variance estimating unit 21, an inter-frame noise variance estimating unit 22, and a noise intensity determining unit 23. The image processing apparatus 2, as in the first embodiment of the present invention, to obtain the frame image u _t and close the frame image u _t-1 included in the image as input. The image processing apparatus 2, as in the first embodiment of the present invention, by using these two images, but outputs perform estimation of the noise intensity sigma ² _t included in the frame image u _t , The details are different.

  Here, the image processing apparatus 2 and its functional blocks can be configured by the same hardware elements as the image processing apparatus 1 according to the first embodiment of the present invention described with reference to FIG. However, the hardware configuration of the image processing apparatus 2 and its functional blocks is not limited to the above-described configuration.

  Next, the configuration of each functional block of the image processing device 2 will be described.

First, the configuration of the intra-frame noise variance estimating unit 21 will be described in detail. As shown in FIG. 5, the intra-frame noise variance estimating unit 21 includes a block dividing unit 211, an intra-block variance calculating unit 212, a variance histogram generating unit 213, and an intra-frame noise variance calculating unit 214. Frame noise variance estimation unit 21 uses these function blocks, the intra-frame noise variance sigma ² _t frame image _{u t,} estimates the _intra.

Specifically, the block dividing unit 211 divides the frame image u _t into blocks of a predetermined size. For example, _{assume that} the width of the frame image ut is 640 pixels and the height is 480 pixels. It is also assumed that a height of 8 pixels and a width of 8 pixels are determined as the size of a block to be divided. In this case, the block division unit 211 divides the frame image _{u t (640 × 480) /} (8 × 8) = 4800 blocks. Here, “/” represents division.

Block variance calculation unit 212, for each block of the block position frame image _{u t} is divided (i, j), calculates the variance _{v intra} (i, j) of the pixel values. Here, i is an integer of 1 or more and n or less, and j is an integer of 1 or more and m or less. Here, n is a number of blocks arranged in the width direction of the frame image u _t. Further, m is a number of blocks arranged in the height direction of the frame image u _t.

The variance histogram generating unit 213 generates a histogram from the variance v _intra (i, j) of each block.

The in-frame noise variance calculation unit 214 regards the lower _γsmooth percentage of the histogram generated by the variance histogram generation unit 213 as the variance calculated in a block whose pixel value change belongs to a flat area. Here, γ _smooth is a predetermined positive number of 100 or less. Then, the intra-frame noise variance calculation unit 214 sets the bin value of the histogram of the lower _γsmooth percentage of the histogram as the flatness determination threshold _τsmooth . For example, the intra-frame noise variance calculation unit 214 may set the value included in the width of the smallest bin exceeding the _γsmooth percentage in the total number of bins in the histogram from the lower order as the flatness determination threshold _τsmooth . Alternatively, the intra-frame noise variance calculation unit 214 may set a value included in the width of the largest bin whose sum from the lowest frequency of bins in the histogram does not exceed _γsmooth percentage as the flatness determination threshold _τsmooth . The value included in the bin width may be a median value, a minimum value, a maximum value, or the like of the bin width. Then, the _intra- frame noise variance calculation unit 214 calculates the frame based on the statistic of the variance value v _intra (i, j) _satisfying v _intra (i, j) < _τsmooth (for example, an average value or a median value). frame noise variance sigma ² _t images _{u t,} calculates the _intra.

Next, the configuration of the inter-frame noise variance estimating unit 22 will be described in detail. As shown in FIG. 6, the inter-frame noise variance estimating unit 22 includes a differential image generating unit 221, a differential image block dividing unit 222, a differential image block variance calculating unit 223, a motion determining unit 224, and an inter-frame noise And a variance calculation unit 225. The inter-frame noise variance estimating unit 22 uses these functional blocks to set an area where the pixel value variance of each block of the difference image is smaller than the motion determination threshold τ _nomv based on the intra-frame noise variance as a non-motion area. The _inter- frame noise variance σ ² _{t, inter} is estimated.

Specifically, the difference image generation unit 221 from the frame image _{u t} and close the frame image _{u t-1,} to generate a difference image d. For example, the difference image generation unit 221, each pixel (x, y) constituting the frame image _{u t for, d (x, y) =} u t (x, y) -u t-1 (x, y) and What is necessary is just to calculate.

  The difference image block dividing unit 222 divides the difference image d into blocks of a predetermined size. The dividing process is the same as that of the block dividing unit 211 in the intra-frame noise variance estimating unit 21. Note that the size of the block may be the same as or different from the block divided by the block dividing unit 211.

The difference image intra-block variance calculation unit 223 calculates the variance v _inter (i, j) of the pixel value of each block at the block position (i, j) into which the difference image d is divided. Here, i is an integer of 1 or more and n 'or less, and j is an integer of 1 or more and m' or less. Note that n ′ is the number of blocks arranged in the width direction of the difference image d. M ′ is the number of blocks arranged in the height direction of the difference image d.

The motion determination unit 224 determines a motion determination threshold τ _nomv based on _{the intra-} frame noise variance σ ² _{t, intra} . For example, the motion determination unit 224 may calculate the motion determination threshold τ _nomv using the following equation (1).

Here, α is a parameter for adjusting the strictness of the motion determination. In principle, α = 1.0. However, when the strictness of the motion determination is reduced (a slight motion is allowed), a value larger than 1.0 is set to α. To increase the strictness of the motion determination, a value smaller than 1.0 is set for α.

Then, the motion determining unit 224 determines a block in which v _inter (i, j) <τ _nomv as a region with no motion. In addition, the motion determining unit 224 determines a block that does not _satisfy v _inter (i, j) <τ _nomv as a motion area. For example, the motion determination unit 224 may set the motion determination result m (i, j) = 1 for a block in which v _inter (i, j) <τ _nomv . Further, the motion determination unit 224 may set the motion determination result m (i, j) = 0 for a block where v _inter (i, j) <τ _nomv . In this case, the fact that the motion determination result is 1 indicates that there is no motion. A motion determination result of 0 indicates that there is a motion.

The inter-frame noise variance calculation unit 225 calculates an _inter- frame noise variance σ ² _{t, inter} based on the noise variance v _inter (i, j) and the motion determination result m (i, j) of each block of the difference image d. I do.

Specifically, the inter-frame noise variance calculation unit 225 calculates the total number n _nomv of blocks without motion using the following equation (2).

Here, if n _nomv = 0, the _inter -frame noise variance calculation unit 225 _assumes that the entire screen greatly fluctuates, and sets σ ² _{t, inter} = σ ² _{t, intra} . When _nnomvｖ0 , the inter-frame noise variance calculation unit 225 calculates σ ² _{t, inter} using the following equation (3).

Incidentally, the inter-frame noise variance estimation unit 22, if the neighboring frame image _{u t-1} for the frame image _{u t} does not exist, it is impossible to estimate the inter-frame noise variance sigma ² _t, the _inter. In this case, the inter-frame noise variance estimating unit 22 may use _{the intra-} frame noise variance σ ² _{t, intra} to set σ ² _{t, inter} = σ ² _{t, intra} .

  Next, the configuration of the noise intensity determination unit 23 will be described in detail.

Noise intensity determining unit 23, the intra-frame noise variance sigma ² _t, and _intra, inter-frame noise variance sigma ² _t, the smaller of the _inter, determined as a noise intensity of the frame image _{u t.} That is, the noise intensity determination unit 23 calculates the noise strength of the frame image _{^{u t, σ 2 t = min}} (σ 2 t, inter, σ 2 t, intra) the.

  The operation of the image processing apparatus 2 configured as described above will be described. Here, the outline of the operation of the image processing apparatus 2 is substantially the same as the operation of the image processing apparatus 1 according to the first embodiment of the present invention described with reference to FIG. However, details of the operations in steps S2 to S4 are different.

  First, a detailed operation in the present embodiment of the intra-frame noise variance estimation processing in step S2 will be described with reference to FIG.

In Figure 7, first, the block dividing unit 211, a frame image _{u t,} is divided into blocks of a predetermined size (step S21).

Next, the intra-block variance calculation unit 212 calculates the variance v _intra (i, j) of the pixel value for each of the divided blocks (step S22).

Next, the histogram generation unit 213 generates a histogram from the variance v _intra (i, j) of each block calculated in step S12 (step S23).

Next, the noise variance calculating unit 214 frame, based on the value of the lower gamma _{smooth smooth} percent of bins of the histogram, calculates a flatness determination threshold tau _{smooth smooth} (step S24).

Next, the _intra- frame noise variance calculation unit 214 regards a block whose variance v _intra (i, j) is smaller than the flatness determination threshold _τsmooth as belonging to a flat area. Then, the intra-frame noise variance calculation unit 214 calculates the statistic of the variance of the blocks (for example, an average value or a median value) as _{the intra-} frame noise variance σ ² _{t, intra} (step S25).

  Next, a detailed operation in the present embodiment of the inter-frame noise variance estimation processing in step S3 will be described with reference to FIG.

In Figure 8, first, the difference image generation unit 221 from the frame image _{u t} and close the frame image _{u t-1,} to generate a difference image d (step S31).

  Next, the difference image block dividing unit 222 divides the difference image d into blocks of a predetermined size (Step S32).

Next, the intra-difference-image-block variance calculation unit 223 calculates the variance v _inter (i, j) of the pixel value of each divided block (step S33).

Next, the motion determination unit 224 calculates a motion determination threshold τ _nomv based on _{the intra-} frame noise variance σ ² _{t, intra} . Then, the motion determination unit 224 _obtains a motion determination result m (i, j) for each block of the difference image based on whether or not the variance v _inter (i, j) is smaller than the motion determination threshold τ _nomv. (Step S34).

For example, as described above, the motion determination unit 224 may calculate the motion determination threshold τ _nomv using Expression (1).

Next, based on the motion determination result m (i, j), the _inter- frame noise variance calculation unit 225 calculates a statistic (an average value, a median value, or the like) of the variance v _inter (i, j) of a block in a region without motion. ) _Is calculated as _{the inter-} frame noise variance σ ² _{t, inter} (step S35).

For example, as described above, the inter-frame noise variance calculation unit 225 may calculate _{the inter-} frame noise variance σ ² _{t, inter} using Expression (3).

As described above, when the number n _{nomv of} blocks determined to be a region without motion (m (i, j) = 1) is 0, the inter-frame noise variance calculation unit 225 determines the intra-frame noise variance σ ² _t, the _intra, inter-frame noise variance sigma ² _t, may be _inter.

Next, the processing of the noise intensity determination unit 23 in step S4 will be described. In the present embodiment, the noise intensity determination unit 23 determines the smaller one of σ ² _{t, intra} and σ ² _{t, inter} calculated in steps S1 and S2 as the noise intensity σ ² _t of the frame image ut. Output.

  This is the end of the description of the operation of the image processing apparatus 2.

  Next, effects of the second exemplary embodiment of the present invention will be described.

  The image processing apparatus according to the second embodiment of the present invention can more accurately estimate the noise intensity even for an image including a texture or an area with a lot of motion.

  The reason will be described. In the present embodiment, the intra-frame noise variance estimating unit generates a histogram for the variance of pixel values in each block obtained by dividing the frame image, and extracts a flat region using a flatness determination threshold determined based on the histogram. . Then, the intra-frame noise variance estimating unit estimates the statistic of the variance of the pixel values in the flat region as the intra-frame noise variance. Further, the inter-frame noise variance estimating unit calculates the variance of the pixel value in each block obtained by dividing the difference image between the frame image and the adjacent frame image, and compares the calculated variance with a motion determination threshold based on the intra-frame noise variance. By doing so, an area without motion is extracted. Then, the inter-frame noise variance estimating unit estimates the statistic of the variance of the pixel values in the area without motion as the inter-frame noise variance. Then, the noise intensity determination unit determines the smaller one of the intra-frame noise variance and the inter-frame noise variance as the noise intensity of the frame image.

  As described above, in the present embodiment, the smaller one of the intra-frame noise variance that becomes larger than the true value in the case of an image with many textures and the inter-frame noise variance that becomes larger than the true value in the case of an image with many motions Is selected as the noise intensity of the frame image. As a result, the present embodiment can determine a noise intensity closer to a true value for an image having a large amount of motion but a small amount of texture, or an image having a large amount of texture but a small amount of movement, and thus the accuracy of the noise intensity estimation can be determined. Improve.

  Further, in the present embodiment, when calculating the inter-frame noise variance, the motion determination threshold is calculated using the intra-frame noise variance. Therefore, there is no need for the user to adjust the motion determination threshold for each image. Further, in the present embodiment, the inter-frame noise variance is calculated using only blocks in which the noise variance of each block is smaller than the intra-frame noise variance. For this reason, in the present embodiment, when it can be determined that there is an area without motion, it is possible to obtain a noise variance estimation result closer to the true value than the intra-frame variance.

(Third embodiment)
Next, a third embodiment of the present invention will be described in detail with reference to the drawings. In each of the drawings referred to in the description of the present embodiment, steps having the same configuration and the same operation as those in the second embodiment of the present invention are denoted by the same reference numerals, and detailed descriptions in the present embodiment will be given. Description is omitted.

First, FIG. 9 shows a configuration of an image processing device 3 according to a third embodiment of the present invention. In FIG. 9, the image processing apparatus 3 is different from the image processing apparatus 2 according to the second embodiment of the present invention in that an inter-frame noise variance estimating unit 32 is provided instead of the inter-frame noise variance estimating unit 22. Are different. As shown in FIG. 10, the inter-frame noise variance estimating unit 32 includes a motion determining unit 324 in place of the motion determining unit 224 in the inter-frame noise variance estimating unit 22 according to the second embodiment of the present invention. Further, the difference is that a variance histogram generation unit 326 and a motion determination threshold value calculation unit 327 are included. The inter-frame noise variance estimating unit 32 uses these functional blocks, based on a first threshold based on the intra-frame noise variance and a second threshold based on the variance of pixel values of each block of the difference image, Determine a motion determination threshold. Then, the inter-frame noise variance estimating unit 32 estimates the _inter- frame noise variance σ ² _{t, inter} with an area where the variance of the pixel value of each block of the difference image is smaller than the motion determination threshold as an area without motion.

Specifically, the variance histogram generation unit 326 generates a histogram of the variance value v _inter (i, j) of each block in the difference image d.

The motion determination threshold calculator 327 calculates a motion determination threshold τ _nomv based on a histogram of the variance v _inter (i, j) of each block of the difference image d _{and the intra-} frame noise variance σ ² _{t, intra.} . Specifically, motion determination threshold value calculation unit 327 calculates a first candidate motion determination threshold tau _NOMV, the intra-frame noise variance sigma ² _t, the first threshold tau _Nomv1 based on _intra. Equation (1) may be used to calculate the first threshold value _τnomv1 .

The motion determination threshold value calculation unit 327 calculates a second candidate motion determination threshold tau _NOMV, variance _{v inter} (i, j) of each block of the difference image d of the second threshold tau _Nomv2 based on . Specifically, the motion determination threshold value calculation unit 327, the dispersion value v _{inter (i, j)} the lower gamma _NOMV percent of the histogram of considered to be calculated dispersed with no moving area, the value of that bin The second threshold value τ _nomv2 may be used. Here, γ _nomv is a predetermined positive number of 100 or less. For example, the motion determination threshold value calculation unit 327 may set a value included in the width of the smallest bin whose sum from the _lowermost frequency of the histogram exceeds γ _nomv percent as the second threshold value τ _nomv2 . In addition, the motion determination threshold value calculation unit 327 may set the value included in the width of the largest bin whose sum from the _lowermost frequency of the histogram does not exceed γ _nomv percent as the second threshold value τ _nomv2 . The value included in the bin width may be a median value, a minimum value, a maximum value, or the like of the bin width. Then, the motion determination threshold calculation unit 327 sets the smaller one of the first threshold τ _nomv1 and the second threshold τ _nomv2 as the motion determination threshold τ _nomv . That is, the motion determination threshold calculator 327 calculates the motion determination threshold τ _nomv using the following equation (4).

The motion determining unit 324 is configured substantially in the same manner as the motion determining unit 224 according to the second embodiment of the present invention, _except that the threshold value calculated by the motion determining threshold calculating unit 327 is used as the motion determining threshold τ _nomv. Are different. Specifically, the motion determination unit 324 determines, among the blocks of the difference image d, a block whose variance v _inter (i, j) is smaller than the motion determination threshold τ _nomv calculated by the motion determination threshold calculation unit 327. It is determined that there is no motion. Then, as in the second embodiment of the present invention, the motion determination unit 324 determines 1 (no motion) or 0 (no motion) as the motion determination result m (i, j) at the block position (i, j). Is set).

  The operation of the image processing device 3 configured as described above will be described. The operation of the image processing apparatus 3 is substantially the same as the operation of the image processing apparatus 2 according to the second embodiment of the present invention, except for the details of the inter-frame noise variance estimation processing described with reference to FIG. .

  FIG. 11 shows the operation of the inter-frame noise variance estimation processing in the present embodiment.

In FIG. 11, first, the inter-frame noise variance estimating unit 32 operates in steps S31 to S33 in the same manner as the second embodiment of the present invention, and performs variance v _inter (i, j) is calculated.

Next, the motion determination threshold calculator 327 calculates a first threshold τ _nomv1 based on _{the intra-} frame noise variance σ ² _{t, intra} (step S44).

As described above, the motion determination threshold value calculation unit 327 may calculate the first threshold value τ _nomv1 using Expression (1).

Next, the variance histogram generation unit 326 generates a histogram for the variance v _inter (i, j) of each block. Then, the motion determination threshold calculator 327 calculates a second threshold τ _nomv2 based on the bin value of the lower γ _nomv percent of the histogram (step S45).

Next, the motion determination threshold calculation unit 327 compares the first threshold τ _nomv1 and the second threshold τ _nomv2, and _sets the smaller one as the motion determination threshold τ _nomv (step S46).

Next, the inter-frame noise variance estimating unit 32 executes step S35 in the same manner as in the second embodiment of the present invention _, and estimates the _inter- frame noise variance σ ² _{t, inter} .

  Thus, the inter-frame noise variance estimating unit 32 ends the operation of the inter-frame noise variance estimation process.

  Next, effects of the third exemplary embodiment of the present invention will be described.

  The image processing apparatus according to the third embodiment of the present invention can more accurately estimate the noise intensity for an image having many textures.

  The reason will be described. This is because, in the present embodiment, in addition to the same configuration as the second embodiment of the present invention, the inter-frame noise variance estimating unit is configured as follows. That is, the inter-frame noise variance estimating unit sets a smaller one of the first threshold based on the intra-frame noise variance and the second threshold calculated based on the variance of each block of the difference image as the motion determination threshold. . Then, the inter-frame noise variance estimating unit estimates an area without motion in the frame image using the motion determination threshold, and calculates an inter-frame noise variance based on the pixel value of the area without motion. .

  For this reason, in the present embodiment, it is possible to make a more precise motion determination, and to more accurately estimate the inter-frame noise variance. As a result, the present embodiment can more accurately estimate the inter-frame noise variance even when the image has many textures.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described in detail with reference to the drawings. In each of the drawings referred to in the description of the present embodiment, steps having the same configuration and the same operation as those in the third embodiment of the present invention are denoted by the same reference numerals, and detailed description in the present embodiment will be given. Description is omitted.

  First, FIG. 12 shows a configuration of an image processing apparatus 4 according to a fourth embodiment of the present invention. In FIG. 12, the image processing apparatus 4 further includes a noise suppression unit 44 in addition to the same configuration as the image processing apparatus 3 according to the third embodiment of the present invention.

Noise suppressing section 44, the frame image u _t, by applying the noise suppression process using the noise strength sigma ² _t which is determined by the noise intensity determination unit 23 generates an image z _t where noise is suppressed. The noise suppressing section 44 outputs the image _{z t.}

  Here, various known methods can be applied to the noise suppression processing. As an example, an example of noise suppression (noise removal) by an adaptive Wiener filter will be described.

The adaptive Wiener filter will be described. Here, u is an input image, and the variance of noise included in u is σ ² _n . Also, a region of 5 pixels × 5 pixels centering on the pixel position (x, y) is used as a local region. In this case, the local average value μ (x, y) of the pixel values in this local area is represented by the following equation (5).

Further, the local variance σ ² _v (x, y) of the pixel values in this area is represented by the following equation (6).

At this time, the processing result z (x, y) by the adaptive Wiener filter is expressed by the following equation (7).

In the adaptive Wiener filter, when the local variance σ ² _v (x, y) at the pixel of interest is approximately _equal to the noise variance σ ² _n , that is, when the pixel of interest is in a flat region, z (x, y) is It approaches the local average value μ (x, y). On the other hand, when the local variance σ ² _v (x, y) is in a large area, such as when the target pixel is an edge, z (x, y) approaches u (x, y) as the input signal. .

Noise suppressing section 44, such to apply the frame image u _t as the input image u of the adaptive Wiener filter, as a dispersion sigma ² _n of the noise, to apply the noise strength sigma ² _t calculated in the noise intensity determination unit 23 it allows to generate an image z _t where noise is suppressed. When the noise suppression unit 44 employs the adaptive Wiener filter, the above-described local region is not limited to 5 pixels × 5 pixels.

  The operation of the image processing apparatus 4 configured as described above will be described with reference to FIG.

In Figure 13, first, the intra-frame noise variance estimation unit 21, the inter-frame noise variance estimation unit 32 and the noise intensity determination unit 23 is input with frame image u _t and close the frame image u _t-1, frame image u _t The noise intensity σ ² _t is estimated (step S51).

  The noise intensity estimation operation in step S51 is the same as the operation of the image processing device 3 according to the third embodiment of the present invention described with reference to FIGS. 3, 7, and 11.

Next, the noise suppressing section 44, a local average of the frame image _{u t} at the target pixel position (x, y) μ (x , y) and the local variance σ ² _v (x, y) is calculated (step S52).

  Next, the noise suppression unit 44 calculates a noise-removed image z (x, y) at the target pixel position (x, y) using, for example, an adaptive Wiener filter (step S53).

Here, if there is a pixel that is not completed for all pixels out process of a frame image u _t (No in step S54), the noise suppression unit 44 continues the processing returns to step S52.

On the other hand, if completed the steps S52~S53 for all pixels of the frame image _{u t} (Yes in step S54), the noise suppressing unit 44 ends the operation.

  Thus, the operation of the image processing apparatus 4 ends.

  Next, effects of the fourth embodiment of the present invention will be described.

  The image processing apparatus according to the fourth embodiment of the present invention executes more appropriate noise suppression processing on an image including a texture or an area with a lot of motion by using a more appropriate noise intensity for each frame image. be able to.

  The reason will be described. This is because, in the present embodiment, the noise suppression unit performs the noise suppression process for each frame image using the noise intensity determined by the same configuration as in the third embodiment of the present invention.

  Note that, as the present embodiment, an image processing apparatus in which a noise suppression unit is added to the configuration of the third embodiment of the present invention has been described. However, the present invention is not limited thereto, and an image processing apparatus in which the noise suppression unit of the present embodiment is added to the configuration of the first or second embodiment of the present invention also has the same effect as that of the present embodiment.

  Further, in the above-described second to fourth embodiments of the present invention, the noise intensity determination unit determines that the smaller of the intra-frame noise variance and the inter-frame noise variance is determined as the noise intensity in the frame image. did. However, the noise intensity determination unit of each embodiment does not necessarily have to apply the smaller noise intensity itself. For example, the noise intensity determination unit may determine the noise intensity by performing a predetermined operation using the smaller of the intra-frame noise variance and the inter-frame noise variance.

  Also, in each of the embodiments of the present invention described above, each functional block of the image processing apparatus has been described mainly with respect to an example in which the functional blocks are realized by a CPU that executes a computer program stored in a storage device or a ROM. However, the present invention is not limited to this, and some, all, or a combination of the functional blocks may be realized by dedicated hardware.

  Further, in each of the embodiments of the present invention described above, the functional blocks of the image processing device may be implemented by being distributed to a plurality of devices.

  In each of the embodiments of the present invention described above, the operation of the image processing apparatus described with reference to each flowchart is stored in a storage device (storage medium) of a computer device as a computer program of the present invention. Then, the CPU may read and execute the computer program. In such a case, the present invention is configured by the code of the computer program or the storage medium.

  In addition, the above-described embodiments can be implemented in appropriate combinations.

  The present invention has been described above using the above-described embodiment as a typical example. However, the invention is not limited to the embodiments described above. That is, the present invention can apply various aspects that can be understood by those skilled in the art within the scope of the present invention.

  This application claims priority based on Japanese Patent Application No. 2015-101272 filed on May 18, 2015, the disclosure of which is incorporated herein in its entirety.

1, 2, 3, 4 Image processing device 11, 21 In-frame noise variance estimating unit 12, 22, 32 Inter-frame noise variance estimating unit 13, 23 Noise intensity determining unit 44 Noise suppressing unit 211 Block dividing unit 212 In-block variance calculation Unit 213 variance histogram generation unit 214 intra-frame noise variance calculation unit 221 difference image generation unit 222 difference image block division unit 223 difference image block variance calculation unit 224, 324 motion determination unit 225 inter-frame noise variance calculation unit 326 variance histogram generation unit 327 Motion determination threshold value calculation unit 901 Image input device 902 Output device 1001 CPU
1002 memory

Claims (6)

In a frame image included in the video, an intra-frame noise variance estimating unit that estimates an intra-frame noise variance, which is a variance of noise in the frame image, based on a pixel value belonging to a flat region where a change in pixel value is flat,
In a difference image representing a difference between the frame image and a neighboring frame image temporally close to the frame image, the frame image is based on a pixel value belonging to an area where there is no movement between the frame image and the neighboring frame image. And an inter-frame noise variance estimating means for estimating an inter-frame noise variance that is a variance of noise between the adjacent frame images,
A noise intensity determining unit that determines a noise intensity in the frame image based on a smaller one of the intra-frame noise variance and the inter-frame noise variance,
An image processing device comprising:   The inter-frame noise variance estimating means may set the area where the variance of the pixel value for each local area of the difference image is smaller than a motion determination threshold based on the intra-frame noise variance as the non-motion area, and The image processing apparatus according to claim 1, wherein:   The inter-frame noise variance estimating means sets the motion determination threshold based on a first threshold based on the intra-frame noise variance and a second threshold based on a variance of pixel values for each local region of the difference image. The image processing apparatus according to claim 2, wherein the determination is made.   4. The apparatus according to claim 1, further comprising: a noise suppression unit configured to execute a noise suppression process on the frame image based on the noise intensity determined by the noise intensity determination unit. 5. The image processing apparatus according to any one of the preceding claims. Computer equipment
In a frame image included in a video, a change in pixel value is based on a pixel value belonging to a flat area, and an intra-frame noise variance that is a variance of noise in the frame image is estimated.
In a difference image representing a difference between the frame image and a neighboring frame image temporally close to the frame image, the frame image is based on a pixel value belonging to an area where there is no movement between the frame image and the neighboring frame image. Estimating the inter-frame noise variance, which is the variance of noise between the adjacent frame images,
An image processing method for determining a noise intensity in the frame image based on a smaller one of the intra-frame noise variance and the inter-frame noise variance. In a frame image included in the video, an intra-frame noise variance estimating step of estimating an intra-frame noise variance that is a variance of noise in the frame image based on a pixel value belonging to a flat region where a change in pixel value is flat,
In a difference image representing a difference between the frame image and a neighboring frame image temporally close to the frame image, the frame image is based on a pixel value belonging to an area where there is no movement between the frame image and the neighboring frame image. And an inter-frame noise variance estimating step of estimating an inter-frame noise variance that is a variance of noise between the adjacent frame images,
A noise intensity determining step of determining an intensity of noise in the frame image based on a smaller one of the intra-frame noise variance and the inter-frame noise variance;
The image processing program causing a computer to execute the device.

Images