JP2017130709A - Image processing apparatus, image processing method and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of detecting a position of generation or a strength of mosquito noise easily, through processing with a little capacity of a memory to be used, without using coded information and while using only a decoded image, and capable of effectively cancelling the noise on the basis of a result of the detection.SOLUTION: An image processing apparatus 1 comprises: a local feature amount generation part 10 for generating multiple local feature amounts including at least a relative flat pixel count feature amount Frc indicating the number of peripheral pixels in the case where a center pixel in a local region of a decoded image is flatter than each of peripheral pixels in the local region; an evaluation value generation part 20 which weights and merges the multiple local feature amounts to calculate a noise evaluation value E for each pixel of the decoded image; a mixture ratio generation part 30 which generates a mixture ratio R on the basis of the noise evaluation value E; a filter processing part 40 which generates a smoothed image of the decoded image; and an image mixture processing part 50 which mixes the decoded image and the smoothed image on the basis of the mixture ratio R, thereby generating an output image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

  The amount of data of high-resolution video such as full HD (1920 x 1080 pixels) and 4K (3840 x 2160 pixels) is enormous, and image quality is maintained as much as possible in order to realize broadcasting, Internet distribution, storage in recording media, etc. Therefore, a compression encoding technique that reduces the amount of data while maintaining it is indispensable.

  In lossy compression encoding, image quality degradation such as noise increases as the compression rate is increased, that is, the amount of data is significantly reduced. For this reason, display devices such as televisions and smartphones generally improve image quality by applying correction processing for removing noise to received video.

  In particular, in JPEG and MPEG-2, high-frequency noise called mosquito noise that appears to fly around the edge is generated. Therefore, techniques for removing mosquito noise by image processing have been studied for a long time.

  Mosquito noise is high-frequency component noise, and generally a method of removing noise by smoothing an image with a low-pass filter such as a bilateral filter or an epsilon filter. However, when smoothing the entire image uniformly, there is a side effect that important image structures such as texture as well as noise are lost at the same time. Therefore, many methods have been proposed in which a region where mosquito noise is generated in advance is detected and smoothing processing is performed only on that region.

  For example, in Patent Document 1, an edge pixel is identified by edge detection and an edge map is generated around the edge pixel, a flat pixel is identified by flat detection and a flat map is generated around the flat pixel, and the edge of the target pixel is detected. It describes that noise is detected using a product obtained by multiplying a map, a flat map, and a secondary differential value as a mosquito noise level, and an edge preserving smoothing filter is applied to the detected region.

  Also, using the encoded information (picture type, transform coefficient, quantization parameter, motion vector, etc.) obtained when the encoded video data is decoded on the receiver side, the location and intensity of mosquito noise are identified. There is also technology to do.

Special table 2012-502540 gazette

  However, the technology according to Patent Document 1 does not consider the visibility of noise depending on the strength and complexity of the edge, and since a flat map is essential, the character with complicated edges is complicated. There is a problem that noise in areas where flats cannot be detected, such as gaps, cannot be removed.

  Also, the technology for identifying the location and intensity of mosquito noise using encoded information is different when the receiver and the display device are different (for example, an HDMI (registered trademark) cable for a video signal received and decoded by an STB). For example, when the information is transferred to a television and displayed, the encoded information cannot be used on the display device side.

  For this reason, there is a need for a technique for detecting the position and intensity of mosquito noise using only decoded images without using encoded information, and effectively removing noise based on the detection results.

  In addition, when the above noise removal is realized by hardware such as an image processor, an increase in the amount of computation and memory usage increases the circuit scale and increases the product cost. Therefore, it is as simple as possible and uses less memory. There is a need for a method for detecting and removing noise by processing.

  The present invention has been made to solve such a problem, and is a simple process with a small memory usage, and the position and intensity of occurrence of mosquito noise can be determined using only decoded images without using encoded information. An object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program capable of detecting and effectively removing noise based on the detection result.

  An image processing apparatus according to the present invention includes a plurality of features including at least a relative flat pixel number feature amount indicating the number of peripheral pixels when a central pixel of a local region of a decoded image is flatter than individual peripheral pixels of the local region. A local feature value generating unit that generates a local feature value, an evaluation value generating unit that calculates a noise evaluation value for each pixel of the decoded image by weighting and integrating a plurality of local feature values, and a mixing ratio based on the noise evaluation value A mixing rate generation unit to generate, a filter processing unit to generate a smoothed image of the decoded image, and an image mixing processing unit to mix the decoded image and the smoothed image based on the mixing rate and generate an output image Is.

  Further, the image processing method according to the present invention includes at least a relative flat pixel number feature amount indicating the number of peripheral pixels when the central pixel of the local region of the decoded image is flatter than each peripheral pixel of the local region. A step of generating a plurality of local feature values, a step of calculating a noise evaluation value for each pixel of the decoded image by weighted integration of the plurality of local feature values, a step of generating a mixing ratio based on the noise evaluation value, The method includes a step of generating a smoothed image of the decoded image, and a step of mixing the decoded image and the smoothed image based on the mixing ratio to generate an output image.

  In addition, the image processing program according to the present invention allows the computer to display a relative flat pixel number feature amount indicating the number of peripheral pixels when the central pixel of the local region of the decoded image is flatter than each peripheral pixel of the local region. For generating a plurality of local feature quantities including at least, a procedure for calculating a noise evaluation value for each pixel of a decoded image by weighted integration of the plurality of local feature quantities, and generating a mixing ratio based on the noise evaluation value This is a procedure for executing a procedure, a procedure for generating a smoothed image of the decoded image, and a procedure for mixing the decoded image and the smoothed image based on the mixing ratio to generate an output image.

  According to the present invention, the generation position and intensity of mosquito noise are detected using only decoded images without using encoded information, and processing is simple and requires less memory, and noise is effectively removed based on the detection results. An image processing apparatus, an image processing method, and an image processing program can be provided.

FIG. 5 is a diagram for explaining an overview of image processing according to the first embodiment. 4 is an example of a decoded image and an output image of image processing according to Embodiment 1. 1 is a diagram illustrating a schematic configuration of an image processing apparatus 1 according to Embodiment 1. FIG. 3 is a diagram illustrating a configuration of a local feature value generation unit 10 according to Embodiment 1. FIG. FIG. 3 is a diagram illustrating a configuration of a first local feature quantity generation unit 11 according to the first embodiment. 6 is a diagram for explaining a directional index k according to Embodiment 1. FIG. It is an example of what each imaged the 1st local feature-value which concerns on Embodiment 1. FIG. 6 is an example of a mask value Bmask according to the first embodiment. It is a figure which shows the structure of the 2nd local feature-value production | generation part 12 which concerns on Embodiment 1. FIG. It is an example of what imaged the 2nd local feature-value which concerns on Embodiment 1. FIG. 3 is an example of a weight function according to the first embodiment. It is an example of what imaged the noise evaluation value E which concerns on Embodiment 1. FIG. It is an example of the weight function regarding the mixture rate R which concerns on Embodiment 1. FIG. It is an example of what imaged the mixture rate R which concerns on Embodiment 1. FIG. 6 is an example of filter processing by the bilateral filter according to the first embodiment. 3 is an example of an output image by the image processing apparatus 1 according to the first embodiment. 3 is a diagram illustrating a schematic configuration of an image processing apparatus 201 according to Embodiment 2. FIG. FIG. 10 is a diagram illustrating a configuration of a global feature value generation unit 260 according to Embodiment 2. 10 is an example of a weight function wtx according to the second embodiment. FIG. 10 is a diagram illustrating a schematic configuration of a part of an image processing apparatus 301 according to a third embodiment. 10 is an example of a weight function wns according to the third embodiment. FIG. 10 is a diagram illustrating a schematic configuration of a part of an image processing apparatus 401 according to a fourth embodiment. 14 is a flowchart illustrating a calculation procedure of a still edge pixel value Bstill according to the fourth embodiment. It is an example of the subtitle appearance position which concerns on Embodiment 4. FIG. It is an example of the weight function wst regarding the subtitle feature-value Fst which concerns on Embodiment 4. FIG.

Hereinafter, an image processing apparatus, an image processing method, or an image processing program according to each embodiment will be described with reference to the drawings.
The “image” described in this specification includes “still image” and “moving image (video)”.

(Embodiment 1)
First, an outline of image processing for detecting and removing mosquito noise according to the first embodiment will be described.

FIG. 1 is a diagram for explaining an overview of image processing according to the first embodiment.
The input image is digital image data before compression acquired by an imaging device such as a digital camera or a digital video camera. On the transmission side, in order to transmit or store the input image, the input image is compressed and encoded so as to be within a desired data amount, and is transmitted or stored.

  On the receiving side, the compressed and encoded image data is received and decoded to generate a decoded image. A digital television normally includes a decoding device, and can perform all processes from reception / decoding to display within the device. On the other hand, only a video signal may be acquired and displayed via a HDMI cable or the like from a set top box (STB) or a hard disk recorder (HDD).

  In the image processing according to the first embodiment, it is assumed that processing from obtaining a decoded image to displaying an output image is performed inside the display device. Further, mosquito noise is detected and removed from the acquired decoded image by an image processing device inside the display device, and the corrected image is displayed as an output image.

  In the image processing according to the first embodiment, the decoded image is represented by a YCbCr color system composed of three components, a luminance component (Y) and two color difference components (Cb, Cr). 1 pixel is expressed by 8 bits (an integer value of 0 to 255).

FIG. 2 is an example of a decoded image and an output image of the image processing according to the first embodiment.
In the decoded image, mosquito noise is generated due to compression coding so that moyamoya and mosquitoes appear to fly around edges (for example, around characters such as program telops). On the other hand, in the output image, the mosquito noise is removed and a clear image is obtained.

FIG. 3 is a diagram showing a schematic configuration of the image processing apparatus 1 according to the first embodiment.
The image processing apparatus 1 inputs a decoded image, corrects it, and outputs an output image. For this purpose, the image processing apparatus 1 includes a local feature value generation unit 10, an evaluation value generation unit 20, a mixing rate generation unit 30, a filter processing unit 40, an image mixing processing unit 50, and the like.

  The local feature generation unit 10 inputs the decoded image and specifies the possibility of occurrence of mosquito noise and the visibility using only pixels within a predetermined range (for example, a local region of 15 × 15 pixels). A plurality of local feature amounts, which are local (local) image feature amounts, are generated and output to the evaluation value generation unit 20.

  The evaluation value generation unit 20 obtains a plurality of local feature quantities, weights them, and integrates them, thereby calculating a noise evaluation value representing the possibility of occurrence of mosquito noise and visibility for each pixel, and the mixing ratio Output to the generation unit 30.

The mixing rate generation unit 30 acquires a noise evaluation value, generates a mixing rate for mixing a smoothed image and a decoded image, which will be described later, and outputs the mixing rate to the image mixing processing unit 50.
The filter processing unit 40 receives the decoded image, smoothes the decoded image using an edge preserving smoothing filter (for example, a bilateral filter, an epsilon filter, etc.), and outputs the smoothed image to the image mixing processing unit 50.

  The image mixing processing unit 50 receives the decoded image, the mixing rate, and the smoothed image, mixes the decoded image and the smoothed image according to the mixing rate, and generates and outputs an output image from which mosquito noise is removed.

  Each component realized by the image processing device 1 can be realized by executing a program under the control of an arithmetic device (not shown) included in the image processing device 1 which is a computer, for example.

  More specifically, the image processing apparatus 1 is realized by loading a program stored in a storage unit (not shown) into a main storage device (not shown) and executing the program under the control of an arithmetic device. Each component is not limited to being realized by software by a program, and may be realized by any combination of hardware, firmware, and software.

  The above-described program can be stored using various types of non-transitory computer readable media and supplied to the image processing apparatus 1. Non-transitory computer readable media include various types of tangible storage media.

  Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD-ROMs. R, CD-R / W, semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)).

  Further, the program may be supplied to the image processing apparatus 1 by various types of temporary computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the image processing apparatus 1 via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

Next, each configuration of the image processing apparatus 1 according to the first embodiment will be described in detail.
FIG. 4 is a diagram illustrating a configuration of the local feature value generation unit 10 according to the first embodiment.
The local feature value generation unit 10 receives the decoded image and outputs a plurality of local feature values. For this purpose, the local feature value generation unit 10 includes a first local feature value generation unit 11, a second local feature value generation unit 12, and the like.

FIG. 5 is a diagram illustrating a configuration of the first local feature value generation unit 11 according to the first embodiment.
The first local feature value generation unit 11 acquires the luminance value Y of the decoded image (only the luminance component is used in the first embodiment), and the first local feature value is a mask value Bmask, a standard deviation value Vstdv, The edge pixel value Bedge, the flat pixel value Bflat, the adjacent pixel difference absolute value sum Vsad1, the minute variation pixel value Bsmall, and the adjacent pixel secondary difference absolute value sum Vsad2 are output.
Here, the variance value of the pixel value, the standard deviation value Vstdv, the adjacent pixel difference absolute value sum Vsad1 or the adjacent pixel secondary difference absolute value sum Vsad2 may be referred to as a basic feature amount.

  Here, V means multivalue, and B means binary (0 or 1). The unit to be output is a local region (for example, 15 × 15 pixels) of a predetermined size centered on the pixel of interest, and is output as a vector having one value for one pixel in the local region. When implemented by software, values for all the pixels of the feature quantity other than the mask value Bmask are calculated and stored in advance, and only the area used for the second local feature quantity to be described later is read and used. May be.

  For this purpose, the first local feature generation unit 11 includes a standard deviation calculation unit 111, an edge pixel generation unit 112, a flat pixel generation unit 113, an adjacent pixel difference absolute value sum calculation unit 114, and an adjacent pixel secondary difference absolute value sum. A calculation unit 115, a minute variation pixel generation unit 116, a local uniform region mask generation unit 117, and the like are provided.

  Hereinafter, each configuration of the first local feature value generation unit 11 and a calculation method of each feature value will be described. Here, the coordinate of the pixel of interest in the decoded image is (x, y), and the luminance value of the pixel of interest is Y (x, y).

The standard deviation calculation unit 111 calculates the standard deviation value Vstdv according to the following equation (1).

  Here, an approximate standard deviation based on an absolute value is simply used to reduce the calculation cost, but an ordinary standard deviation based on a square root of a square may be used, and a variance value may be used to reduce the calculation of the square root. May be used. However, the standard deviation value Vstdv varies depending on the calculation method, and the range of values that can be taken differs. Therefore, it is necessary to adjust the threshold value for calculating the edge pixel value Bedge and the flat pixel value Bflat described later according to the calculation method. is there. In addition, here, the standard deviation of the 3 × 3 pixel area centered on the target pixel is used, but an area such as 5 × 5 pixel or 7 × 7 pixel may be used.

The edge pixel generation unit 112 calculates the edge pixel value Bedge according to the following equation (2). Here, Tedge indicates a threshold value.

The flat pixel generation unit 113 calculates a flat pixel value Bflat according to the following equation (3). Here, Tflat represents a threshold value.

The adjacent pixel difference absolute value sum calculation unit 114 calculates the adjacent pixel difference absolute value sum Vsad1 according to the following equation (4).
Here, the direction index k is used.
FIG. 6 is a diagram for explaining the direction index k according to the first embodiment.

The adjacent pixel secondary difference absolute value sum calculation unit 115 calculates the adjacent pixel secondary difference absolute value sum Vsad2 according to the following equation (5). Again, k indicates the direction index and follows FIG.

The minute variation pixel generation unit 116 calculates the minute variation pixel value Bsmall according to the following equation (6). Here, Thigh and Tlow represent upper and lower thresholds, respectively.

  FIG. 7 is an example of images of the first local feature values (excluding the mask value Bmask) according to the first embodiment. The upper left is a decoded image for reference. B of binary data is shown as 0 for black, 1 is shown as white, and V of multi-value data is shown as gray when the value is 0, and becomes gray as the value increases.

  The local uniform region mask generation unit 117 calculates the mask value Bmask by the following equation (7) using the luminance value Y and the standard deviation value Vstdv. When a 15 × 15 pixel centered on the pixel of interest (x, y) is a local region, the coordinates (i, j) of peripheral pixels that are pixels other than the pixel of interest (x, y) in the local region are Each takes a value from -7 to 7.

Here, Tmask indicates a threshold for wmask, s indicates a position, and || x || indicates a Euclidean distance. Gσ represents a Gaussian distribution with a standard deviation σ, and is represented by Gσ (x) = exp (−x / 2σ ² ). σs, σr, and σd are parameters that determine the standard deviation of each Gaussian distribution.
FIG. 8 is an example of the mask value Bmask according to the first embodiment. 4 on the left shows four types of local area images, 4 in the center show the weight wmask calculated based on each local area image, and 4 on the right shows the mask value Bmask calculated based on each local area image. Indicates. Pixels with a mask value Bmask of 0 are painted black.

  The weight wmask has a large value in a pixel similar to the target pixel in three dimensions of a spatial position, a luminance value, and a standard deviation value, and is a locally uniform region that does not exceed the edge by a mask value Bmask obtained by thresholding this. (Hereinafter referred to as “local uniform region”) can be selected.

  In other words, the local uniform region is a local region having a predetermined size (for example, 15 × 15 pixels), and the spatial position, luminance value, and variance value of the local region with the target pixel (center pixel) are similar. A higher weight is calculated for each pixel, and the weight is greater than a predetermined threshold value.

  Specifically, in FIG. 8, a region having a high similarity to the target pixel and a mask value Bmask of 1 is a local uniform region. By using the local uniform region, for example, the reliability of each local feature quantity such as the flat pixel number feature quantity calculated by the second local feature quantity generation unit 12 can be improved.

FIG. 9 is a diagram illustrating a configuration of the second local feature value generation unit 12 according to the first embodiment.
The second local feature quantity generation unit 12 acquires each feature quantity output from the first local feature quantity generation unit 11, and obtains an edge strength feature quantity Fes, an edge pixel number feature quantity Fec, a relative flat pixel number feature quantity Frc, A flat pixel number feature amount Ffc, a minute variation intensity feature amount Fms, a minute variation pixel number feature amount Fmc, and a minute variation dispersion feature amount Fmv are calculated and output. The second local feature value generation unit 12 calculates and outputs one feature value for each pixel of interest.

  For this purpose, the second local feature value generation unit 12 includes an edge strength feature value generation unit 121, an edge pixel number feature value generation unit 122, a flat pixel number feature value generation unit 123, a minute variation strength feature value generation unit 124, A variation pixel number feature amount generation unit 125, a minute variation dispersion feature amount generation unit 126, a relative flat pixel number feature amount generation unit 127, and the like are provided.

  Hereinafter, each configuration of the second local feature quantity generation unit 12 and a calculation method of each feature quantity will be described. Again, the coordinates of the pixel of interest in the decoded image are (x, y), and the luminance value of the pixel of interest is Y (x, y). Further, the size of the local region is 15 × 15 pixels, and the coordinates (i, j) of the peripheral pixels of the target pixel (x, y) take values of −7 to 7, respectively.

The edge strength feature value generation unit 121 calculates the edge strength feature value Fes according to the following equation (8).

The edge pixel number feature quantity generation unit 122 calculates the edge pixel number feature quantity Fec according to the following equation (9).

The flat pixel number feature quantity generating unit 123 calculates the flat pixel number feature quantity Ffc according to the following equation (10).

The minute fluctuation strength feature quantity generation unit 124 calculates the minute fluctuation strength feature quantity Fms according to the following equation (11).

The minute variation pixel number feature amount generation unit 125 calculates the minute variation pixel number feature amount Fmc according to the following equation (12).

The minute fluctuation dispersion feature quantity generation unit 126 calculates the minute fluctuation dispersion feature quantity Fmv according to the following equation (13).

The relative flat pixel number feature quantity generation unit 127 calculates a relative flat pixel number feature quantity Frc according to the following equation (14).

  Here, the threshold Trelf is a relative coefficient and takes a value from 0.0 to 1.0. The relative coefficient is a value that determines a ratio that the average standard deviation aveVstdv of the local uniform region is considered to be flat with respect to the standard deviation Vstdv of each peripheral pixel of the local region.

  Further, the relative flat pixel number feature amount Frc is that the target pixel is flatter than the peripheral pixel when the average standard deviation aveVstdv of the local uniform region is smaller than the value obtained by multiplying the standard deviation Vstdv of each peripheral pixel by the relative coefficient. It is the value counted as being.

  The standard deviation Vstdv of the pixel of interest can be used instead of the average standard deviation aveVstdv of the local uniform region, but it is better to use the average standard deviation aveVstdv of the local uniform region composed of pixels similar to the pixel of interest. The relative flat pixel number feature quantity Frc can be calculated with high accuracy even in an image in which some noise is on the target pixel.

  In a region including an edge having a large standard deviation Vstdv such as captions in the vicinity, even if there is noise or texture, the relative flat pixel number feature is provided if the average standard deviation aveVstdv of the local uniform region is sufficiently smaller than the standard deviation Vstdv of the edge The amount Frc is a large value.

  Also, even in an area including an edge with a small standard deviation Vstdv, such as a thin character, if the average standard deviation aveVstdv of the local uniform area is sufficiently smaller than the standard deviation Vstdv of the edge, the relative flat pixel number feature amount Frc is Larger value.

  By calculating the relative flat pixel number feature quantity Frc, it is possible to detect a mosquito noise region in consideration of noise visibility based on the relationship of the relative noise intensity to the edge intensity.

  FIG. 10 is an example of images of the second local feature values according to the first embodiment. The upper left is a decoded image for reference. When the value is 0, it is shown as black, and it is shown in a gray scale that becomes white as the value increases.

The evaluation value generation unit 20 illustrated in FIG. 3 calculates a noise evaluation value E by weighted integration of a plurality of local feature amounts acquired from the local feature amount generation unit 10.
Here, among the second local feature amounts, the edge strength feature amount Fes, the edge pixel number feature amount Fec, and the relative flat pixel number feature amount Frc are referred to as local main feature amounts, the flat pixel number feature amount Ffc, and the minute variation. The intensity feature amount Fms, the minute variation pixel number feature amount Fmc, and the minute variation dispersion feature amount Fmv are referred to as local auxiliary feature amounts.

  First, the evaluation value generation unit 20 sets an initial value E0 as the noise evaluation value E (E = E0). Here, the noise evaluation value E is in the range of 0 to 1, and the initial value E0 is 0.5.

Next, the evaluation value generation unit 20 calculates the first evaluation value Em using the three local main feature amounts according to the following equation (15).
Here, w () is a weight function, and the local main feature amount is converted into a weight coefficient w (F (x, y)) in the range of 0 to 1 based on a preset weight function. The weight function of each feature amount will be described later.

Next, the evaluation value generation unit 20 calculates the second evaluation value Es according to the following equation (16) using the four local auxiliary feature amounts.

Finally, the evaluation value generation unit 20 multiplies a weight coefficient wstdv (Vstdv (x, y)) based on the standard deviation Vstdv to obtain a final noise evaluation value E in order to keep the noise evaluation value E on the edge low. It calculates with the following formula | equation (17).

  Through the above processing, the evaluation value generation unit 20 can calculate the noise evaluation value E for each pixel from a plurality of local feature amounts.

FIG. 11 is an example of a weight function according to the first embodiment.
The weighting factor of the local main feature amount is designed so that the weighting factor increases as the feature amount increases. The weighting factor of the local auxiliary feature amount is designed so that the weighting factor is small outside the predetermined range so that the weighting factor is large when the feature amount is within the predetermined range. The weight coefficient of the standard deviation is designed so that the weight coefficient decreases as the standard deviation increases.

FIG. 12 is an example of an image of the noise evaluation value E according to the first embodiment.
The noise evaluation value E is a value of 0.0 to 0.75, and the image is normalized and displayed such that 0 is black and 0.75 is white. The reason why the maximum value of the noise evaluation value E is set to 0.75 is that the maximum value becomes 1.0 when a subtitle feature amount according to Embodiment 3 described later is included. .

The mixing rate generation unit 30 acquires the noise evaluation value E, and generates a mixing rate R used in the image mixing processing unit 50 described later. The mixing ratio R is calculated by the following equation (18).

  Here, wratio () is a weighting function relating to the mixing ratio R, and the noise evaluation value E is converted to a mixing ratio R in the range of 0.0 to 1.0 based on this weighting function. This weighting function is set so that 0.0 is below the lower limit value, 1.0 is above the upper limit value, and the value therebetween is monotonically increasing from 0.0 to 1.0.

FIG. 13 is an example of a weighting function related to the mixing ratio R according to the first embodiment.
FIG. 14 is an example of an image of the mixing ratio R according to the first embodiment. The mixing ratio R calculated using the weight function shown in FIG. 13 is imaged with respect to the image of the noise evaluation value E on the left.

The filter processing unit 40 receives the decoded image, performs a smoothing process using an edge-preserving smoothing filter (for example, a bilateral filter or an epsilon filter), and outputs a smoothed image. Using the luminance value Y of the decoded image, the luminance value f [Y] after filtering, the position p of the target pixel in the image, and the position q in the region centered on the position p, the bilateral filter is expressed by the following formula ( 19).

  Here, the function Gσ represents a Gaussian distribution with a standard deviation σ. The weighting factor wp, q is determined based on the Euclidean distance between the position p and the position q and the absolute value of the luminance difference. The normalization coefficient Wp is such that the sum of weights in the region Ω is 1.

FIG. 15 is an example of filter processing by the bilateral filter according to the first embodiment. For the regions (a) and (b), the decoded image and the smoothed image are respectively shown.
In the area (a), the mosquito noise around the character generated in the decoded image before the filter processing can be removed while the edges are preserved in the smoothed image after the filter processing. On the other hand, in the region (b), the texture of the tree before the filtering process has disappeared after the filtering process. That is, in a smoothed image obtained by smoothing the entire image, not only noise but also important image structures such as textures are lost.

The image mixing processing unit 50 acquires the luminance value Y of the decoded image, the luminance value f [Y] of the smoothed image generated by the filter processing unit, and the mixing rate R generated by the mixing rate generating unit, and the following equation (20 ), The decoded image and the smoothed image are mixed according to the mixing ratio R to generate and output an output image Yout from which noise has been removed.

FIG. 16 is an example of an output image by the image processing apparatus 1 according to the first embodiment. For regions (a) and (b), a decoded image, a smoothed image, and an output image are shown.
In the prior art, at the boundary (edge) between the tree (texture) and the sky (flat) as in the region (a), the texture of the tree is erroneously detected as noise, and the texture is erroneously smoothed. there were. In addition, there is a problem that mosquito noise around a thin character such as the character under the area (b) cannot be detected. Further, when mosquito noise is generated while the characters are dense like the characters on the region (b), there is a problem that the mosquito noise cannot be detected because there is no flat region between the characters.

  In the output image according to the first embodiment, these problems are solved and mosquito noise can be effectively removed with few side effects.

  As described above, the image processing apparatus 1 or the image processing method according to the first embodiment weights and integrates a plurality of local feature amounts such as edges, relative flats, flats, and minute fluctuations calculated only from the decoded image. A noise evaluation value E considering the possibility of occurrence of mosquito noise and visibility is calculated for each pixel, and the smoothing intensity is adaptively controlled based on the noise evaluation value E.

  As a result, mosquito noise is detected and smoothed even in situations where it was difficult to detect with conventional techniques such as between characters with complex edges or around weak edges, while retaining image structures other than noise, such as textures. It is possible to effectively and appropriately remove mosquito noise that can be easily perceived by human eyes while reducing side effects such as excessive blur due to conversion.

  In addition, since a plurality of local feature amounts do not require preprocessing such as complicated edge detection and can be calculated by simple processing using only local region pixels such as 15 × 15 pixels, the memory usage amount It can be realized with hardware with a relatively small circuit scale, and the product cost can be reduced.

  As described above, the image processing apparatus 1 according to the first embodiment has a relative value indicating the number of peripheral pixels when the central pixel of the local region of the decoded image is flatter than the individual peripheral pixels of the local region. A local feature quantity generation unit 10 that generates a plurality of local feature quantities including at least a flat flat pixel feature quantity Frc, and an evaluation that calculates a noise evaluation value E for each pixel of the decoded image by weighted integration of the plurality of local feature quantities The value generation unit 20, the mixing rate generation unit 30 that generates the mixing rate R based on the noise evaluation value E, the filter processing unit 40 that generates a smoothed image of the decoded image, and the decoded image and the smoothed image are mixed. An image mixing processing unit 50 that mixes based on the rate R and generates an output image is provided.

  With such a configuration, the generation position and intensity of mosquito noise are detected using only the decoded image without using encoded information, and processing is simple and requires less memory, and noise is effectively reduced based on the detection result. Can be removed.

Further, in the image processing apparatus 1 according to the first embodiment, the local feature value generation unit 10 uses a plurality of local feature values as a basic feature value, a variance value of pixel values in a local region, which is a basic feature value, and a standard deviation of pixel values. It is preferable to calculate each based on at least one of the value Vstdv or the adjacent pixel difference absolute value Vsad.
With such a configuration, it is possible to remove noise with simpler processing with less memory usage.

Further, in the image processing device 1 according to the first embodiment, when the local feature value generation unit 10 generates the relative flat pixel number feature value Frc, the spatial position, luminance value, or pixel value of the center pixel is changed. It is preferable to compare the average value of the basic feature values of the pixels in the local area where the similarity of the variance values is equal to or greater than a predetermined value with the basic feature values of the individual surrounding pixels.
With such a configuration, it is possible to detect the generation position and intensity of mosquito noise even in a decoded image in which noise is on the center pixel.

Further, in the image processing apparatus 1 according to the first embodiment, the plurality of local feature amounts is the edge strength feature amount Fes that is the average value of the basic feature amounts of the edge pixels in the local region or the edge pixel number feature in the local region. Preferably it further comprises at least one of the quantities Fec.
With such a configuration, it is possible to more accurately detect the generation position and intensity of mosquito noise.

In addition, the image processing apparatus 1 according to the first embodiment has a feature quantity Fms related to a minute variation pixel in which a plurality of local feature quantities fall between a lower limit value whose basic feature quantity is larger than 0 and an upper limit value. , Fmc, and Fmv.
With such a configuration, it is possible to more accurately detect the generation position and intensity of mosquito noise.

  In addition, the image processing method according to the first embodiment uses the relative flat pixel number feature amount indicating the number of peripheral pixels when the central pixel of the local region of the decoded image is flatter than the individual peripheral pixels of the local region. A step of generating a plurality of local feature amounts including at least Frc, a step of calculating a noise evaluation value E for each pixel of the decoded image by weighted integration of the plurality of local feature amounts, and a mixing ratio based on the noise evaluation value E A step of generating R, a step of generating a smoothed image of the decoded image, and a step of mixing the decoded image and the smoothed image based on the mixing ratio R to generate an output image.

  The image processing program according to the first embodiment also causes the computer to display a relative flat pixel indicating the number of peripheral pixels when the central pixel of the local region of the decoded image is flatter than each peripheral pixel of the local region. A procedure for generating a plurality of local feature amounts including at least several feature amounts Frc, a procedure for calculating a noise evaluation value E for each pixel of a decoded image by weighted integration of the plurality of local feature amounts, and a noise evaluation value E To generate a mixing rate R, a procedure for generating a smoothed image of the decoded image, and a procedure for mixing the decoded image and the smoothed image based on the mixing rate R to generate an output image. belongs to.

(Embodiment 2)
In the image processing apparatus or the image processing method according to the first embodiment, the noise evaluation value E is calculated by weighted integration of the local feature amounts. However, in the image processing apparatus or the image processing method according to the second embodiment, the local feature is calculated. The noise evaluation value E is calculated by weighted integration of the amount and a global feature amount described later.

Since the local feature amount is calculated using only the luminance value of the local region in the decoded image, the local feature amount is calculated using only one frame of the still image or the moving image. However, the global feature amount is a feature used for the moving image. It is a quantity, and is calculated based on the local feature quantity of the previous frame in time or its statistical quantity.
Here, the imaging time of the frame to be processed is t, and the imaging time of the immediately preceding frame is t-1.

FIG. 17 is a diagram illustrating a schematic configuration of the image processing apparatus 201 according to the second embodiment.
The image processing apparatus 201 includes a local feature value generation unit 210, an evaluation value generation unit 220, a mixing rate generation unit 230, a filter processing unit 240, an image mixing processing unit 250, a global feature value generation unit 260, and the like. The local feature value generation unit 210, the mixing rate generation unit 230, the filter processing unit 240, and the image mixing processing unit 250 have the same configuration as that according to the first embodiment and operate in the same manner. Omitted.

FIG. 18 is a diagram illustrating a configuration of the global feature value generation unit 260 according to the second embodiment.
The global feature amount generation unit 260 receives the decoded image including color information (YCbCr), the edge pixel number feature amount Fec and the minute variation pixel number feature amount Fmc output from the local feature amount generation unit 210, and receives the global feature. A texture feature amount Ftx as a quantity is generated and output to the evaluation value generation unit 220.

  Therefore, the global feature value generation unit 260 includes a color conversion unit 261, a color class classification unit 262, a color-specific texture probability generation unit 263, a delay unit 264, a texture feature value generation unit 265, and the like.

  When the decoded image at time t−1 is input, the color conversion unit 261 calculates the hue H and the saturation S according to a general color conversion formula from the YCbCr color system to the HSV color system, and sets the luminance Y of YCbCr. The YCbCr color system is converted into the HSY color system and output to the color class classification unit 262.

  The color class classification unit 262 classifies each pixel into a color class based on the values of hue H, saturation S, and luminance Y. In the second embodiment, for a pixel whose luminance Y falls within the range of Tdark ≦ Y ≦ Tbright, the value of the saturation S is a chromatic color when the value is greater than or equal to the threshold value Tsat, and the achromatic color is less than Tsat. In addition, for a chromatic color, a value obtained by quantizing a hue H expressed by 8 bits from 0 to 255 with a constant QH is defined as a class value C. Similarly, for the achromatic color, a value obtained by quantizing the luminance Y expressed by 8 bits from 0 to 255 with a constant QY is defined as a class value C.

As described above, by quantizing the colors, adjacent pixels can be easily classified into the same class, the spatial variation of the texture feature amount Ftx can be suppressed, and the number of histogram bins described later is reduced, and the histogram is stored. Memory capacity can be reduced.
The color class classification unit 262 outputs the class value C to the color-specific texture probability generation unit 263 and the texture feature amount generation unit 265.

  The color-specific texture probability generation unit 263 calculates the texture probability Ptexture for each color class value C using the edge pixel number feature amount Fec and the minute variation pixel number feature amount Fmc.

First, the color-specific texture probability generation unit 263 calculates a texture pixel value Btexture according to the following equation (21). Here, Tec and Tmc indicate threshold values.

Next, the color-specific texture probability generation unit 263 sets the texture pixel value of the pixel of coordinates (x, y) and color class value C to Btexture (x, y, C), and sets the size of the area for obtaining the texture probability to N A histogram of texture pixel values is created for each color class value C as xM pixels and divided by the size of the area according to the following equation (22), and a texture probability Ptexture for each color class value C is calculated. To do. The area for calculating the texture probability Ptexture may be the entire decoded image, or the image may be divided into blocks of a predetermined size, and the texture probability Ptexture may be calculated for each block.

The delay unit 264 stores the texture probability Ptexture for each color calculated at time t−1 in a buffer for use by the texture feature value generation unit 265 at time t.
On the other hand, when the decoded image at time t is input, HSY is similarly generated in the color conversion unit 261, and the color class value C is calculated for each pixel in the color class classification unit 262.

  The texture feature quantity generation unit 265 refers to the texture probability Ptexture (C) calculated for the color class value C of the decoded image at time t−1 for the color class value C of the target pixel of the decoded image at time t, This texture probability Ptexture (C) is output as the texture feature amount Ftx of the target pixel of the decoded image at time t.

  The evaluation value generation unit 220 illustrated in FIG. 17 calculates the noise evaluation value E by weighting and integrating the plurality of local feature amounts acquired from the local feature amount generation unit 210 and the texture feature amount Ftx that is the global feature amount. To do.

The evaluation value generation unit 220 integrates the texture feature amount Ftx according to the following equation (23) with respect to the second evaluation value Es of equation (16), and calculates a third evaluation value Eg.
Here, wtx () is a weighting function related to the texture feature amount Ftx.

FIG. 19 shows an example of the weight function wtx according to the second embodiment.
The weighting function wtx is designed so that the weighting factor increases as the feature amount increases. However, since the weight coefficient wtx (Ftx (x, y)) is subtracted from the second evaluation value Es in Expression (23), the third evaluation value Eg decreases as the texture feature amount Ftx increases.

Finally, the evaluation value generation unit 220 multiplies the weight coefficient wstdv (Vstdv (x, y)) based on the standard deviation Vstdv in order to keep the noise evaluation value E on the edge low, as in the equation (17). The final noise evaluation value E is calculated by the following equation (24).

  As described above, in the image processing apparatus 201 or the image processing method according to the second embodiment, the noise evaluation value E can be calculated for each pixel from the plurality of local feature amounts and texture feature amounts Ftx.

  Thereby, even in a region where it is difficult to distinguish between noise and texture with local feature amounts, it is possible to suppress erroneous detection of the texture and reduce the side effect of erroneously smoothing the texture. Further, since the texture probability Ptexture is calculated using the previous frame, an increase in memory usage can be suppressed as compared with the first embodiment. Further, the texture feature amount Ftx at time t can be obtained in units of one pixel and does not require a line buffer or the like, so that an increase in hardware cost can be suppressed.

  As described above, the image processing apparatus 201 according to the second embodiment has a global feature amount generation unit that generates a global feature amount based on a local feature amount of a decoded image temporally prior to the decoded image. 260, the global feature amount includes a texture feature amount Ftx indicating that it is a texture, and the evaluation value generation unit 220 integrates a plurality of local feature amounts and the global feature amount to obtain a noise evaluation value E. It is preferable to calculate.

  With such a configuration, it is possible to suppress erroneous detection of texture even in an area where it is difficult to distinguish between noise and texture with local feature amounts, and to reduce the side effect of erroneously smoothing the texture.

(Embodiment 3)
In the image processing apparatus or image processing method according to the second embodiment, the texture feature quantity Ftx is used as the global feature quantity. However, in the image processing apparatus or image processing method according to the third embodiment, the noise scene is used as the global feature quantity. Use features.

FIG. 20 is a diagram illustrating a schematic configuration of a part of the image processing apparatus 301 according to the third embodiment.
The image processing apparatus 301 includes a local feature value generation unit 210, an evaluation value generation unit 320, a mixing rate generation unit 230, a filter processing unit 240, an image mixing processing unit 250, a global feature value generation unit 360, and the like. Since the mixing rate generation unit 230, the filter processing unit 240, and the image mixing processing unit 250 have the same configuration as that according to the second embodiment and operate in the same manner, illustration and description thereof are omitted here.

  The global feature value generation unit 360 uses the decoded image and the flat pixel value Bflat calculated by the local feature value generation unit 210 (the threshold value Tflat for calculating the flat pixel value Bflat shown in Expression (3)) as the noise scene feature value. And a flat pixel number feature value Fec are obtained, and the noise scene feature value Fsc is output to the evaluation value generation unit 320 as a global feature value. To do. For this reason, the global feature value generation unit 360 includes a code change calculation unit 361, a noise scene feature value generation unit 362, a delay unit 363, and the like.

When the decoded image at time t−1 is input, the code change calculation unit 361 calculates a code change pixel value B ^X sc in the horizontal direction and a code change pixel value B ^Y sc in the vertical direction for each pixel, and a noise scene The data is output to the feature value generation unit 362. The sign change pixel value B ^X sc in the horizontal direction is calculated according to the following equation (25).

The sign change pixel value B ^Y sc in the vertical direction is also calculated using three pixels of Y (x, y + 1), Y (x, y), and Y (x, y−1), as in Expression (25).

Noise scene feature amount generating unit 362, based on a flat pixel number feature quantity Ffc and flat pixel value Bflat obtained from the local feature amount generating unit 210, a sign change pixel value B ^X sc, in the B ^Y sc, the following formula The noise scene feature value Fns is calculated according to (26) and output to the delay unit 363. Here, N and M indicate the number of pixels in the horizontal and vertical directions of the decoded image, and Tfc indicates a threshold value.

As can be seen from the equation (26), the noise scene feature amount Fns represents a code change amount of the flat area of the entire image. Accordingly, one noise scene feature amount Fns is calculated for one image.
The delay unit 363 enables the noise scene feature amount Fns calculated in the frame at time t−1 to be used in the frame at time t.

  The evaluation value generation unit 320 applies the same noise scene feature amount Fns to all the pixels in one image. The evaluation value generation unit 320 calculates a noise evaluation value E by weighting and integrating a plurality of local feature amounts acquired from the local feature amount generation unit 210 and a noise scene feature amount Fns that is a global feature amount, and a mixing ratio The data is output to the generation unit 230.

Specifically, the evaluation value generation unit 320 changes the equation (17) in the first embodiment to the following equation (27), and calculates the noise evaluation value E.

Here, wns () is a weighting function related to the noise scene feature amount Fns.
FIG. 21 shows an example of the weight function wns according to the third embodiment.
The weighting function wns is designed so that the weighting factor wns (Fns) decreases as the noise scene feature amount Fns increases.

  As described above, the evaluation value generation unit 320 calculates the noise evaluation value E for each pixel from the plurality of local feature amounts and the noise scene feature amount Fns. As a result, it is possible to suppress false detection even in scenes including film grain, camera noise, etc., which are difficult to distinguish from mosquito noise with local features, and smooth out noise intentionally added by the producer. Unevenness caused by smoothing side effects and camera noise in only a part of the region can be reduced.

  As described above, the image processing apparatus 301 according to the third embodiment has a global feature amount generation unit that generates a global feature amount based on the local feature amount of the decoded image temporally before the decoded image. 360, the global feature amount includes a noise scene feature amount Fsc indicating the noise amount of the image, and the evaluation value generation unit calculates a noise evaluation value E by integrating a plurality of local feature amounts and the global feature amount. It is preferable to do.

  With such a configuration, it is possible to suppress erroneous detection even in a scene including film grain, camera noise, and the like, which are difficult to distinguish from mosquito noise with local feature amounts.

(Embodiment 4)
In the image processing apparatus or the image processing method according to the second or third embodiment, the texture feature quantity or the noise scene feature quantity is used as the global feature quantity, but the image processing apparatus or the image processing method according to the fourth embodiment is used. The subtitle feature value is used as the global feature value.

FIG. 22 is a diagram showing a schematic configuration of a part of the image processing apparatus 401 according to the fourth embodiment.
The image processing apparatus 401 includes a local feature value generation unit 210, an evaluation value generation unit 420, a mixing rate generation unit 230, a filter processing unit 240, an image mixing processing unit 250, a global feature value generation unit 460, and the like. Since the mixing rate generation unit 230, the filter processing unit 240, and the image mixing processing unit 250 have the same configuration as that according to the second embodiment and operate in the same manner, illustration and description thereof are omitted here.

  The global feature value generation unit 460 acquires the edge pixel value Bedge and the edge strength feature value Fes calculated by the local feature value generation unit 210, calculates a subtitle feature value Fst as the global feature value, and sends it to the evaluation value generation unit 420. Output. Therefore, the global feature value generation unit 460 includes a still edge pixel generation unit 461, a still edge pixel number feature value generation unit 462, a subtitle feature value generation unit 463, and the like.

  The still edge pixel generation unit 461 acquires the edge pixel value Bedge from the local feature value generation unit 210, calculates the still edge pixel value Bstill, and outputs it to the still edge pixel number feature value generation unit 462. When the edge pixel value Bedge (x, y) becomes 1 continuously for a predetermined number of frames at a certain pixel position (x, y), the still edge pixel generation unit 461 performs the still edge pixel value Bstill (x, y). ) Is 1.

FIG. 23 is a flowchart showing a calculation procedure of the still edge pixel value Bstill according to the fourth embodiment.
When the decoded edge image is input (step S10), the still edge pixel generation unit 461 determines whether it is the first frame of the input video, that is, the first frame (time t = 0) (step S20). When it is the first frame (Yes in step S20), the count numbers cnt (x, y) of all pixel positions are initialized to 0 (step S30), and the process proceeds to the next procedure (step S40). If it is not the first frame (No in step S20), the process proceeds to the next procedure (step S40) as it is.

  Next, the still edge pixel generation unit 461 performs the following processing (step S40 to step S90) on all the pixel positions (x, y) in the frame image, so that the still edge pixel value Bstill (x, y y) is calculated.

First, it is determined whether the edge pixel value Bedge (x, y) is 0 (step S40).
When the edge pixel value Bedge (x, y) is not 0, that is, when the edge pixel value Bedge (x, y) is 1 (No in step S40), the count number cnt (x, y) is increased by 1. (Step S50).

Further, it is determined whether the count number cnt (x, y) is greater than or equal to a predetermined threshold value Tcnt (step S60).
When the count number cnt (x, y) is equal to or larger than the predetermined threshold Tcnt (Yes in step S60), the still edge pixel value Bstill (x, y) is set to 1 (step S70). The threshold value Tcnt is a value for determining that “the edge is stationary” when the edge is continuously generated for several frames.

  When the count number cnt (x, y) is less than the predetermined threshold Tcnt (No in step S60), the still edge pixel value Bstill (x, y) is set to 0 (step S80).

  On the other hand, when the edge pixel value Bedge (x, y) is 0 (Yes in step S40), the still edge pixel value Bstill (x, y) is set to 0 and the count number cnt (x, y) is set to 0. (Step S90).

  Then, the still edge pixel generation unit 461 calculates the still edge pixel value Bstill (x, y) for all the pixel positions (x, y) in the frame image (steps S40 to S90), and then the still edge Pixel value Bstill (x, y) is output to still edge pixel number feature quantity generation section 462.

Next, the still edge pixel generation unit 461 determines whether it is the last frame (step S110). When it is not the last frame (No in step S110), the process returns to step S10. When it is the last frame (Yes in step S110), the process is terminated.
In this way, the still edge pixel generation unit 461 can calculate the still edge pixel value Bstill (x, y) at time t.

The still edge pixel number feature quantity generation unit 462 acquires the still edge pixel value Bstill (x, y), calculates the still edge pixel number feature quantity Fse (x, y) according to the following equation (28), and obtains the subtitle feature. The data is output to the quantity generation unit 463. In the fourth embodiment, as in the first embodiment, the size of the local area is 15 × 15 pixels, and the coordinates (i, j) of the peripheral pixels of the pixel of interest (x, y) are from −7 to 7 Take the value.

The subtitle feature value generation unit 463 calculates the subtitle feature value Fst according to the following equation (29) using the three feature values of the still edge pixel number feature value Fse, the edge strength feature value Fes, and the subtitle position feature value Fsp. The result is output to the evaluation value generation unit 420.

  The subtitle position feature amount Fsp (x, y) is a value representing the ease of appearance of the subtitle at the coordinates (x, y). For example, captions are likely to appear at the bottom of the screen, and broadcast station watermarks and sports scores are likely to appear at the top of the screen.

FIG. 24 is an example of the subtitle appearance position according to the fourth embodiment. In general, subtitles are likely to appear in hatched areas at the top or bottom of the screen 6.
The subtitle feature quantity generation unit 463 sets the subtitle position feature quantity Fsp (x, y) in advance so that the hatched portion has a high value. Further, the subtitle feature quantity generation unit 463 may collect a large amount of program data, calculate the subtitle occurrence probability for each pixel position (x, y) in advance, and use it as the subtitle position feature quantity Fsp (x, y).

The evaluation value generation unit 420 calculates the noise evaluation value E by weighting and integrating the plurality of local feature amounts acquired from the local feature amount generation unit 210 and the subtitle feature amount Fsp that is a global feature amount.
Specifically, the third evaluation value Eg is calculated by integrating the subtitle feature quantity Fst and the second evaluation value Es calculated by the expression (16) according to the following expression (30).
Here, wst () is a weighting function related to the subtitle feature quantity Fst.

FIG. 25 shows an example of the weighting function wst related to the subtitle feature quantity Fst according to the fourth embodiment.
The weighting function wst is set so that the weighting coefficient wst (Fst (x, y)) increases as the subtitle feature amount Fst increases.

Then, in the same way as Expression (17), the evaluation value generation unit 420 suppresses the noise evaluation value E on the edge to be low, according to the following Expression (31), the weight coefficient wstdv (Vstdv (x, Multiply y)) to calculate the noise evaluation value E.

  As described above, the evaluation value generation unit 420 can calculate the noise evaluation value E for each pixel from the plurality of local feature values and the subtitle feature value Fst. Thereby, it is possible to increase the noise evaluation value E in subtitles, program telops, etc. in which mosquito noise is conspicuous, and to detect mosquito noise in consideration of visibility.

As described above, the image processing apparatus 401 according to the fourth embodiment has a global feature amount generation unit that generates a global feature amount based on a local feature amount of a decoded image temporally prior to the decoded image. 460 further includes a subtitle feature value Fsp indicating that the global feature value is a stationary character region, and the evaluation value generation unit 420 integrates a plurality of local feature values and the global feature value to obtain a noise evaluation value. It is preferable to calculate E.
With such a configuration, it is possible to detect mosquito noise in consideration of visibility.

  In the second to fourth embodiments, the global feature amount calculated based on the local feature amount of the previous frame or its statistical amount is used when calculating the noise evaluation value E of the current frame. This can be realized even in the case where sequential line processing is performed by hardware, and has an advantage that frame delay or the like does not occur.

  However, when real-time processing is not required, when frame delay is acceptable, or when realized by software, the global feature is calculated based on the local feature of the current frame instead of the previous frame or its statistics, Two-pass processing for calculating the noise evaluation value E of the current frame using the global feature amount can also be performed.

  Further, the noise evaluation value E can be calculated using all of the plurality of local feature values and the plurality of global feature values according to the first to fourth embodiments, or the noise evaluation value E can be calculated using a part of the feature values. It can also be calculated.

  The image processing apparatus or image processing method according to each embodiment relates to a technique for improving image quality by detecting and removing mosquito noise in an image / video deteriorated by compression coding in the technical field. It can be used for mosquito noise removal processing in an image processing chip mounted on a display device such as a digital television. Since there are few false detections or detection leaks of noise and the hardware cost is low, it is excellent in terms of performance and cost.

Regarding the above embodiments, the following additional notes are disclosed.
(Appendix 1)
In a noise removal apparatus that removes mosquito noise generated by compression coding using only a decoded image,
A local feature amount generating unit that generates a plurality of local feature amounts related to edges, relative flats, flats, and minute variations using pixels in a predetermined local region in the decoded image;
An evaluation value generation unit that calculates a noise evaluation value for each pixel by weighted integration processing of a plurality of local feature amounts;
A filter processing unit that applies a smoothing filter to the decoded image to generate a smoothed image;
A mixing rate generator that generates a mixing rate based on the noise evaluation value;
A noise removal apparatus comprising: an image mixing processing unit that generates an output image by mixing a decoded image and a smoothed image based on a mixing ratio.

(Appendix 2)
When a decoded image is one frame of a moving image in a compression-encoded moving image,
A global feature generation unit that generates a global feature using a local feature in a frame temporally prior to the processing target frame;
The noise removal apparatus according to appendix 1, wherein the evaluation value calculation unit calculates a noise evaluation value by weighted integration processing using a global feature amount in addition to a plurality of local feature amounts.

(Appendix 3)
The noise removal device according to supplementary note 1, wherein the plurality of local feature amounts are calculated based on a basic feature amount calculated based on a pixel value of a local region (15 × 15 pixels or the like) in the decoded image.
(Appendix 4)
The noise removal device according to appendix 3, wherein the basic feature amount is a dispersion value or a standard deviation value in the local region.

(Appendix 5)
The noise removal device according to supplementary note 3, wherein the basic feature amount is an adjacent pixel difference absolute value in the local region.
(Appendix 6)
The noise removal device according to supplementary note 3, wherein the plurality of local feature amounts includes at least one of edge strength, edge pixel number, and relative flat pixel number calculated based on the basic feature amount.

(Appendix 7)
The noise removal apparatus according to supplementary note 3, wherein the plurality of local feature amounts includes a flat pixel number, a minute variation intensity, a minute variation variance, and a minute variation pixel number calculated based on the basic feature amount.
(Appendix 8)
The noise removal apparatus according to appendix 2, wherein the global feature amount includes a texture feature amount determined based on a color.

(Appendix 9)
The noise removal apparatus according to appendix 8, wherein the texture feature amount is calculated based on a color-dependent minute variation pixel histogram in a temporally previous frame.
(Appendix 10)
The noise removal apparatus according to appendix 2, wherein the global feature amount includes a noise scene feature amount representing a noise amount of the entire image.

(Appendix 11)
The noise removal device according to appendix 10, wherein the noise scene feature amount is calculated based on the number of changes in luminance difference code of the flat portion in the previous frame in time.
(Appendix 12)
The noise removal device according to attachment 2, wherein the global feature amount includes a subtitle feature amount indicating that the character region is a stationary character region.

(Appendix 13)
The subtitle feature amount is calculated from the number of still edge pixels calculated based on the edge pixels in the previous frame in time and the edge pixels in the current frame, the edge strength of the current frame, and the character appearance frequency by region. 12 noise removers.
(Appendix 14)
The noise removal device according to appendix 1, wherein the noise evaluation value increases as the edge strength increases.

(Appendix 15)
The noise removal device according to appendix 1, wherein the noise evaluation value increases as the number of edge pixels increases.
(Appendix 16)
The noise removal device according to appendix 1, wherein the noise evaluation value becomes larger as the number of relative flat pixels increases.

(Appendix 17)
The noise removal device according to supplementary note 1, wherein the noise evaluation value becomes a large value when the number of flat pixels takes a value within a predetermined range under a condition that the main local feature is not zero.
(Appendix 18)
The noise removal device according to supplementary note 1, wherein the noise evaluation value is a large value when the value of the minute luminance fluctuation takes a value within a predetermined range under a condition that the main local feature is not zero.

(Appendix 19)
The noise removal device according to supplementary note 1, wherein the noise evaluation value becomes a large value when a value within a predetermined range having a minute luminance fluctuation dispersion is obtained under a condition that the main local feature amount is not zero.
(Appendix 20)
The noise removal device according to appendix 1, wherein the noise evaluation value becomes a large value when the number of minute luminance variation pixels takes a value within a predetermined range under a condition that the main local feature is not zero.

(Appendix 21)
The noise removal device according to attachment 2, wherein the noise evaluation value becomes smaller as the texture feature amount becomes larger.
(Appendix 22)
The noise removal device according to appendix 10, wherein the noise evaluation value becomes smaller as the noise scene feature amount becomes larger.

(Appendix 23)
The noise removal device according to attachment 12, wherein the noise evaluation value increases as the subtitle feature amount increases.
(Appendix 24)
The noise removal device according to supplementary note 3, wherein the noise evaluation value becomes smaller as the basic feature value increases when the basic feature value exceeds a certain threshold.

(Appendix 25)
The noise removal device according to appendix 14, wherein the edge strength is an average value of basic feature values in edge pixels in the local region.
(Appendix 26)
The noise removal device according to supplementary note 15, wherein the number of edge pixels is the number of edge pixels in the local region.

(Appendix 27)
The relative flat pixel count is obtained by multiplying the basic feature value of each pixel in the local area by a predetermined flat rate, and the flat threshold value. The basic feature value of the central pixel or the basic feature value average value in the local uniform area is The noise removal device according to supplement 16, wherein the number of pixels equal to or less than the flat threshold is counted.
(Appendix 28)
The noise removal device according to appendix 17, wherein the number of flat pixels is the number of flat pixels in the local uniform region.

(Appendix 29)
The noise removal device according to appendix 18, wherein the minute variation intensity is an average value of basic feature amounts in minute variation pixels within a local uniform region.
(Appendix 30)
The noise removal device according to appendix 19, wherein the minute variation variance is a variance value of a basic feature amount in a minute variation pixel in a local uniform region.

(Appendix 31)
The noise removal device according to appendix 20, wherein the number of minute variation pixels is the number of minute variation pixels in a local uniform region.
(Appendix 32)
The noise removal device according to attachment 25, wherein the edge pixel is a pixel whose basic feature amount is equal to or greater than a predetermined threshold value.

(Appendix 33)
The noise removal device according to appendix 28, wherein the flat pixel is a pixel whose basic feature amount is equal to or less than a predetermined threshold value.
(Appendix 34)
The noise removal device according to supplementary note 29, wherein the minute variation pixel is a pixel whose basic feature amount falls within a range of two threshold values indicating an upper limit and a lower limit.

(Appendix 35)
The local uniform area is a weight that increases as the degree of similarity of the spatial position, luminance value, and variance value with the central pixel increases with respect to the pixels in the local area of a predetermined size (15 × 15 pixels or the like). The noise removal apparatus according to supplementary note 28, wherein the pixel is calculated for each pixel, and only pixels whose weight is equal to or greater than a predetermined threshold are used as the local uniform region.

1, 201, 301, 401 Image processing device 10, 210 Local feature generation unit 11 First local feature generation unit 12 Second local feature generation unit 20, 220, 320, 420 Evaluation value generation unit 30, 230 Mixing rate Generation unit 40, 240 Filter processing unit 50, 250 Image mixing processing unit 111 Standard deviation calculation unit 114 Adjacent pixel difference absolute value sum calculation unit 116 Slight variation pixel generation unit 121 Edge strength feature quantity generation unit 122 Edge pixel number feature quantity generation unit 127 Relative flat pixel number feature generation unit 260, 360, 460 Global feature generation unit 265 Texture feature generation unit 362 Noise scene feature generation unit 463 Subtitle feature generation unit

Claims (8)

A local feature that generates a plurality of local feature amounts including at least a relative flat pixel number feature amount indicating the number of peripheral pixels when the central pixel of the local region of the decoded image is flatter than the individual peripheral pixels of the local region A quantity generator;
An evaluation value generation unit that calculates a noise evaluation value for each pixel of the decoded image by weight-integrating the plurality of local feature amounts;
A mixing rate generation unit that generates a mixing rate based on the noise evaluation value;
A filter processing unit for generating a smoothed image of the decoded image;
An image processing apparatus comprising: an image mixing processing unit that mixes the decoded image and the smoothed image based on the mixing ratio and generates an output image. The local feature value generation unit is configured to determine the plurality of local feature values based on at least one of a variance value of pixel values in the local region, a standard deviation value of pixel values, or an adjacent pixel difference absolute value, which is a basic feature value. The image processing device according to claim 1, wherein the image processing device calculates each of the images. When the local feature amount generating unit generates the relative flat pixel number feature amount,
The average value of the basic feature values of the pixels in the local region whose spatial position, brightness value, or dispersion value of pixel values with the central pixel is equal to or greater than a predetermined value, and the basic features of the individual peripheral pixels The image processing apparatus according to claim 2, wherein the amount is compared. The plurality of local feature amounts further include at least one of an edge strength feature amount that is an average value of the basic feature amounts of edge pixels in the local region or an edge pixel number feature amount of the local region. The image processing apparatus according to claim 3. 5. The plurality of local feature amounts further include a feature amount relating to a minute variation pixel in which the basic feature amount falls between a lower limit value greater than 0 and an upper limit value. The image processing apparatus according to item. A global feature quantity generating unit that generates a global feature quantity based on the local feature quantity of the decoded image temporally prior to the decoded image;
The global feature amount includes at least one of a texture feature amount indicating texture, a noise scene feature amount indicating image noise amount, or a subtitle feature amount indicating a stationary character region,
The image processing apparatus according to any one of claims 1 to 5, wherein the evaluation value generation unit calculates the noise evaluation value by integrating the plurality of local feature values and the global feature value. Generating a plurality of local feature amounts including at least a relative flat pixel number feature amount indicating the number of peripheral pixels when a central pixel of a local region of the decoded image is flatter than each peripheral pixel of the local region; ,
Calculating a noise evaluation value for each pixel of the decoded image by weight-integrating the plurality of local feature amounts;
Generating a mixing ratio based on the noise evaluation value;
Generating a smoothed image of the decoded image;
An image processing method comprising: mixing the decoded image and the smoothed image based on the mixing ratio to generate an output image. On the computer,
A procedure for generating a plurality of local feature amounts including at least a relative flat pixel number feature amount indicating the number of peripheral pixels when a central pixel of a local region of a decoded image is flatter than each peripheral pixel of the local region; ,
A procedure for calculating a noise evaluation value for each pixel of the decoded image by weight-integrating the plurality of local feature amounts;
Generating a mixing ratio based on the noise evaluation value;
Generating a smoothed image of the decoded image;
An image processing program for executing a procedure of mixing the decoded image and the smoothed image based on the mixing ratio and generating an output image.