JP5634494B2 - Image processing apparatus, image processing method, and computer program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a computer program that can determine the direction of an edge component included in an input image and remove or reduce noise that stores the edge portion.

  2. Description of the Related Art Conventionally, there is an image processing apparatus that smoothes pixel values by filtering processing on an input image and reduces noise included in the input image. In such an image processing apparatus, since the smoothing process is performed without distinguishing between a flat part and an edge part included in the input image, the edge part may be deteriorated. In view of this, image processing apparatuses that store and smooth the edge portion of an input image have been studied.

  Patent Document 1 describes an image processing apparatus that performs a process of reducing noise included in an input image. The image processing apparatus separates an input image into a first component that is a skeleton component and a second component obtained from a residual of the first component with respect to the input image, and from the first component to an edge direction (horizontal or vertical) and Get edge information such as edge strength. The image processing device performs noise reduction processing of the second component based on the acquired edge information, and synthesizes the first component and the second component after noise reduction into an output image.

JP 2009-020605 A

  However, the image processing apparatus described in Patent Document 1 determines the direction of the edge portion in two directions, horizontal or vertical. For this reason, when an edge portion in another direction is included in the input image, the edge portion may be deteriorated. Further, Patent Document 1 suggests that an anisotropic bilateral filter is used. However, when the anisotropic bilateral filter is mounted, a multiplier is required. There is a problem that increases.

  Further, the image processing apparatus described in Patent Document 1 may cause an error in the determination of the edge direction or the like when the input image includes, for example, a diagonal line having a width of one pixel, and stores an edge in the wrong direction. There is a risk of smoothing processing. Further, the configuration using the anisotropic bilateral filter suggested in Patent Document 1 has a problem that it cannot cope with the case where the input image includes block noise or the like and an edge shift occurs at the block noise boundary.

  The present invention has been made in view of such circumstances, and an object of the present invention is to perform image processing that can detect edges in more directions and perform edge-preserving smoothing with high accuracy. An apparatus, an image processing method, and a computer program are provided.

  An image processing apparatus according to the present invention includes a specific area extracting unit that extracts a specific area including a target pixel from an input image composed of a plurality of pixels arranged in a matrix, and the intensity of an edge component in each specific direction. A Sobel filter storage means for storing a plurality of Sobel filters to be detected in different directions, and a filter process by a plurality of Sobel filters stored by the Sobel filter storage means for the specific area extracted by the specific area extraction means And, based on the plurality of intensities calculated by the edge strength calculating means, one or more different from the plurality of directions, the edge strength calculating means for calculating the strength of the edge component included in the specific region in a plurality of directions Edge strength estimating means for estimating the strength of the edge component for the direction, the strength calculated by the edge strength calculating means, and the Based on Tsu di strength intensity estimation means has estimated the edge component included in the specific area is characterized by comprising an edge direction determination means for determining the direction of extension.

  Further, in the image processing apparatus according to the present invention, the edge strength calculating means calculates the strength of the edge component in at least four directions, and the edge strength estimating means has 2 in the vicinity of the direction to be estimated. For the direction interpolated by linear interpolation based on the two strengths calculated by the edge strength calculation means for the direction, linear for the two neighboring directions based on the strengths of the four directions calculated by the edge strength calculation means. Each error between the intensity interpolated by interpolation and the intensity in the two neighboring directions calculated by the edge intensity calculating means is weighted according to the difference between the direction to be estimated and the two neighboring directions. A value obtained by adding the average values is calculated as an estimated value of the intensity of the edge component in the direction to be estimated.

  In the image processing apparatus according to the present invention, when the Sobel filter storage means sets the horizontal angle of the input image to 0 [rad], the angles 0, π / 4, π / 2, and 3π / 4 are used. A basic Sobel filter for four directions corresponding to, and a basic Sobel filter corresponding to angle 0 as A, a basic Sobel filter corresponding to angle π / 4 as B, and a basic Sobel filter corresponding to angle π / 2. When the bell filter is C, the basic Sobel filter corresponding to the angle 3π / 4 is D, and the Sobel filter corresponding to the angle π is E (= −A), the following (1) to (4) One or a plurality of extended Sobel filters X relating to the direction of the angle x (where 0 <x <π) given by the equation is stored.

When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)

  Further, the image processing apparatus according to the present invention includes a specific area extracting unit that extracts a specific area including a target pixel from an input image including a plurality of pixels arranged in a matrix, and each of edge components related to a specific direction. A Sobel filter storage means for storing a plurality of Sobel filters for detecting the intensity in different directions, and a plurality of Sobel filters stored by the Sobel filter storage means for the specific area extracted by the specific area extraction means. An edge strength calculation unit that performs filtering to calculate the strength of the edge component included in the specific region in a plurality of directions, and the edge component included in the specific region is extended based on the strength calculated by the edge strength calculation unit. Edge direction determining means for determining an existing direction, and the Sobel filter storage means includes a lateral angle of the input image. Is 0 [rad], A is a basic Sobel filter for four directions corresponding to angles 0, π / 4, π / 2, and 3π / 4, and a basic Sobel filter corresponding to angle 0, and the angle The basic Sobel filter corresponding to π / 4 is B, the basic Sobel filter corresponding to the angle π / 2 is C, the basic Sobel filter corresponding to the angle 3π / 4 is D, and the sorbor corresponding to the angle π is D. When the bell filter is E (= −A), one or a plurality of extended Sobels regarding the direction of the angle x (where 0 <x <π) given by the following equations (1) to (4) The filter X is stored.

When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)

  In the image processing apparatus according to the present invention, when the plurality of Sobel filters stored in the Sobel filter storage unit is a 3 × 3 matrix and the horizontal direction of the input image is 0 [rad], the angle The Sobel filter relating to the x direction is given by the following equations (5) to (8).

  The image processing apparatus according to the present invention also stores an edge-preserving smoothing filter that stores a plurality of edge-preserving smoothing filters that store edge components in a specific direction and smooth pixel values, respectively, in different directions. And one edge preserving type smoothing filter is selected from the edge preserving type smoothing filter storing means according to the direction determined by the edge direction determining means and the pixel value of the pixel of interest in the specific region is smoothed And a smoothing means for converting to a smoothing means.

  The image processing apparatus according to the present invention further includes an edge strength difference determination unit that determines whether or not a difference between a maximum value and a minimum value of the plurality of edge strengths calculated by the edge strength calculation unit exceeds a threshold value. When the edge strength difference determination unit determines that the difference does not exceed the threshold, the smoothing unit smoothes the pixel value of the target pixel in the specific region using a smoothing filter that is not an edge-preserving type. It is characterized by that.

  The image processing apparatus according to the present invention further includes an application determination unit that determines whether or not to apply a smoothing result by the smoothing unit, and the pixel value of each pixel of the input image is determined by the smoothing unit. An output image having either the pixel value smoothed in step 1 or the original pixel value not smoothed is generated.

  Further, the image processing apparatus according to the present invention performs second filtering on the specific area extracted by the specific area extraction unit using a Laplacian filter, and calculates an intensity of an edge component included in the specific area. Calculating means, and edge strength determining means for determining whether or not the strength calculated by the second edge strength calculating means exceeds a threshold, and the application determining means is configured to detect the edge when the strength exceeds the threshold. When the strength determining means determines, it determines that the smoothing result by the smoothing means is applied, and when the edge strength determining means determines that the intensity does not exceed the threshold, the smoothing result by the smoothing means is It is characterized in that it is determined not to apply.

  Further, the image processing apparatus according to the present invention includes an increase / decrease number calculation unit that calculates the increase / decrease number of pixel values between pixels adjacent in a specific direction in the specific region extracted by the specific region extraction unit, and the increase / decrease number calculation. An increase / decrease count determination unit that determines whether the increase / decrease count calculated by the means exceeds a threshold, and the application determination unit determines that the increase / decrease count does not exceed the threshold, When the smoothing result by the smoothing means is determined to be applied, and the increase / decrease count determination means determines that the increase / decrease count exceeds the threshold value, the smoothing result by the smoothing means is determined not to be applied. It is characterized by being.

  Also, the image processing apparatus according to the present invention is configured to calculate a difference between a pixel value of a pixel of interest included in the specific area extracted by the specific area extraction unit and a pixel value smoothed by the smoothing unit. Calculating means, and smoothing difference determining means for determining whether or not the difference calculated by the smoothing difference calculating means exceeds a threshold value, and the application determining means determines the smoothing difference determination if the difference does not exceed the threshold value. When the means determines, it is determined that the smoothing result by the smoothing means is applied, and when the difference exceeds the threshold, the smoothing difference determination means determines that the smoothing result by the smoothing means is not applied. It is characterized in that it is determined.

  In addition, the image processing method according to the present invention includes a specific region extraction step of extracting a specific region including a target pixel from an input image composed of a plurality of pixels arranged in a matrix, and each of edge components related to a specific direction. A plurality of Sobel filters for detecting the intensity are stored in different directions, and the specific region extracted in the specific region extraction step is subjected to filter processing by the plurality of stored Sobel filters, and is included in the specific region. Edge strength calculation step for calculating the strength of the edge component to be calculated in a plurality of directions, and the strength of the edge component in one or a plurality of directions different from the plurality of directions based on the plurality of strengths calculated in the edge strength calculation step An edge strength estimation step for estimating the edge strength, the strength calculated in the edge strength calculation step, and the edge strength estimation Estimated intensity based on at step, characterized in that the edge components included in the specific region and an edge direction determination step of determining a direction of extension.

  Further, in the image processing method according to the present invention, in the edge strength calculating step, the edge component strength is calculated in at least four directions, and in the edge strength estimating step, the edge is detected in two directions in the vicinity of the direction to be estimated. For the intensity interpolated by linear interpolation based on the two intensities calculated by the intensity calculating means, the neighboring two directions are interpolated by linear interpolation based on the intensity in the four directions calculated by the edge intensity calculating means. An average value weighted according to the difference between the direction to be estimated and the two directions in the vicinity is calculated for each error between the intensity and the intensity in the two directions in the vicinity calculated by the edge strength calculation means. The added value is calculated as an estimated value of the intensity of the edge component in the direction to be estimated.

  Further, in the image processing method according to the present invention, when the lateral angle of the input image is set to 0 [rad] as a Sobel filter used for filter processing in the edge intensity calculation step, the angle is 0, π / 4. , Π / 2 and 3π / 4, the basic Sobel filter corresponding to the four directions, the basic Sobel filter corresponding to the angle 0 as A, the basic Sobel filter corresponding to the angle π / 4 as B, the angle When a basic Sobel filter corresponding to π / 2 is C, a basic Sobel filter corresponding to an angle 3π / 4 is D, and a Sobel filter corresponding to an angle π is E (= −A), One or a plurality of extended Sobel filters X relating to the direction of the angle x (where 0 <x <π) given by the equations (1) to (4) are stored.

When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)

  In addition, the image processing method according to the present invention includes a specific region extraction step of extracting a specific region including a target pixel from an input image composed of a plurality of pixels arranged in a matrix, and each of edge components related to a specific direction. A plurality of Sobel filters for detecting the intensity are stored in different directions, and the specific region extracted in the specific region extraction step is subjected to filter processing by the plurality of stored Sobel filters, and is included in the specific region. Edge strength calculating step for calculating the strength of the edge component to be determined in a plurality of directions, and edge direction determination for determining the direction in which the edge component included in the specific region extends based on the strength calculated in the edge strength calculating step And the input as a Sobel filter used for filter processing in the edge strength calculation step When the lateral angle of the image is 0 [rad], a basic Sobel filter for four directions corresponding to angles 0, π / 4, π / 2 and 3π / 4, and a basic Sobel corresponding to angle 0 The filter is A, the basic Sobel filter corresponding to the angle π / 4 is B, the basic Sobel filter corresponding to the angle π / 2 is C, the basic Sobel filter corresponding to the angle 3π / 4 is D, When the Sobel filter corresponding to the angle π is set to E (= −A), one is related to the direction of the angle x (where 0 <x <π) given by the following equations (1) to (4). Alternatively, a plurality of extended Sobel filters X are stored.

When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)

  In the image processing method according to the present invention, the plurality of Sobel filters to be stored are 3 × 3 matrices, and when the horizontal direction of the input image is set to 0 [rad], The bell filter is given by the following equations (5) to (8).

  Further, the image processing method according to the present invention stores a plurality of edge-preserving smoothing filters each storing edge components in a specific direction and smoothing pixel values for different directions, and said edge direction determining step A smoothing step of smoothing the pixel value of the pixel of interest in the specific region by selecting one edge preserving type smoothing filter from the stored plurality of edge preserving type smoothing filters according to the direction determined in step Is further included.

  The image processing method according to the present invention further includes an edge strength difference determination step for determining whether or not the difference between the maximum value and the minimum value of the plurality of edge strengths calculated in the edge strength calculation step exceeds a threshold value. In the smoothing step, when the edge strength difference determining step determines that the difference does not exceed the threshold value, the pixel value of the pixel of interest in the specific region is smoothed using a smoothing filter that is not an edge-preserving type. It is characterized by doing.

  The image processing method according to the present invention further includes an application determination step for determining whether or not to apply a smoothing result obtained by the smoothing step, and the pixel value of each pixel of the input image is determined by the smoothing step. An output image having either the pixel value smoothed in step 1 or the original pixel value not smoothed is generated.

  In the image processing method according to the present invention, a second edge that calculates a strength of an edge component included in the specific region by performing filter processing using a Laplacian filter on the specific region extracted in the specific region extraction step. An intensity calculation step, and an edge intensity determination step for determining whether the intensity calculated in the second edge intensity calculation step exceeds a threshold value. In the application determination step, when the intensity exceeds the threshold value If it is determined in the edge strength determination step, it is determined that the smoothing result in the smoothing step is applied, and if it is determined in the edge strength determination step that the strength does not exceed the threshold, the smoothing step It is determined that the smoothing result is not applied.

  Further, the image processing method according to the present invention includes an increase / decrease number calculation step for calculating the increase / decrease number of pixel values between pixels adjacent in a specific direction in the specific region extracted in the specific region extraction step, and the increase / decrease number An increase / decrease count determination step for determining whether or not the increase / decrease count calculated in the calculation step exceeds a threshold value. In the application determination step, the increase / decrease count determination step determines that the increase / decrease count does not exceed the threshold value. If it is determined that the smoothing result by the smoothing means is to be applied, and if the number of increase / decrease times exceeds the threshold value and the determination is made in the increase / decrease number determination step, the smoothing result by the smoothing means is not applied. It is characterized by doing.

  In addition, the image processing method according to the present invention calculates a difference between a pixel value of a target pixel included in the specific area extracted in the specific area extraction step and a pixel value smoothed in the smoothing step. A difference calculating step, and a smoothing difference determining step for determining whether or not the difference calculated in the smoothing difference calculating step exceeds a threshold value. In the application determining step, if the difference does not exceed the threshold value, the smoothing When determined in the difference determining step, it is determined that the smoothing result by the smoothing unit is applied, and when the difference exceeds the threshold, the smoothing result by the smoothing unit is determined when determined in the smoothing difference determining step. Is determined not to apply.

  Further, the computer program according to the present invention includes a specific area extracting means for extracting a specific area including a pixel of interest from an input image composed of a plurality of pixels arranged in a matrix, and an edge for each specific direction. A plurality of Sobel filters for storing different Sobel filters for detecting the intensity of the component, and a plurality of Sobels stored by the Sobel filter storage for the specific area extracted by the specific area extraction means. Based on the plurality of intensities calculated by the edge strength calculating means that performs filtering by a filter and calculates the strength of the edge component included in the specific region in a plurality of directions, the plurality of directions are different from the plurality of directions. Edge strength estimating means for estimating the strength of edge components in one or more directions, and the edge strength Strength calculating means has calculated strength and the edge strength estimation means has estimated based on the edge components included in the specific region is equal to or to operate as an edge direction determination means for determining the direction of extension.

  Further, the computer program according to the present invention includes a specific area extracting means for extracting a specific area including a pixel of interest from an input image composed of a plurality of pixels arranged in a matrix, and an edge for each specific direction. A plurality of Sobel filters for storing different Sobel filters for detecting the intensity of the component, and a plurality of Sobels stored by the Sobel filter storage for the specific area extracted by the specific area extraction means. An edge strength calculation unit that performs filtering using a filter and calculates the strength of the edge component included in the specific region in a plurality of directions, and the edge component included in the specific region based on the strength calculated by the edge strength calculation unit Operating as edge direction determining means for determining the extending direction of the Sobel filter memory The stage includes a basic Sobel filter for four directions corresponding to angles 0, π / 4, π / 2, and 3π / 4, and an angle of 0 when the lateral angle of the input image is 0 [rad]. The corresponding basic Sobel filter is A, the basic Sobel filter corresponding to the angle π / 4 is B, the basic Sobel filter corresponding to the angle π / 2 is C, and the basic Sobel corresponding to the angle 3π / 4. When the filter is D and the Sobel filter corresponding to the angle π is E (= −A), the angle x given by the following equations (1) to (4) (where 0 <x <π) ), One or a plurality of extended Sobel filters X are stored.

When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)

In the present invention, the strength of edge components included in a specific region is calculated in a plurality of directions using a plurality of Sobel filters, and the strength of edge components in directions that cannot be calculated by the Sobel filter based on the calculated strengths Is estimated. The extending direction of the edge component in the specific region is determined based on the calculated and estimated strength of the edge component.
As a result, it is possible to estimate and obtain the strength of the edge component in the direction that the Sobel filter does not support, and to determine the direction of the edge component in more directions. By performing smoothing using an edge-preserving smoothing filter corresponding to the determined direction, it is possible to select an edge-preserving smoothing filter that is more suitable for the edge component and perform smoothing with high accuracy. Become.

In the present invention, the edge component strength is calculated in at least four directions by the Sobel filter, and the strength of the edge component in the other direction is estimated based on the calculated four strengths.
The strength of the edge component is estimated by the following algorithm. First, the intensity in the direction to be estimated is obtained by linear interpolation based on the two intensities calculated in the two neighboring directions. Next, intensities in two directions in the vicinity are obtained by linear interpolation based on the four intensities calculated by the Sobel filter. Next, an error between the intensity calculated by the Sobel filter and the intensity obtained by linear interpolation is calculated for the two adjacent directions. Next, an average value weighted to each calculated error is calculated according to the difference between the direction to be estimated and the two neighboring directions. A value obtained by adding the calculated average value to the intensity in the direction to be estimated obtained by linear interpolation is set as an estimated value of the edge component intensity in this direction.
In addition, it is not necessary to perform the calculation of the above algorithm every time of estimation, and an arithmetic expression for calculating the estimated value according to the above algorithm may be derived in advance. When estimating the strength of the edge component, the estimated value can be calculated by substituting the strength of the edge component calculated by the Sobel filter into an arithmetic expression derived in advance.

In the present invention, the Sobel filters stored for calculating the strength of the edge component in a specific direction are four basic Sobel filters and one or a plurality of extended Sobel filters. The basic Sobel filter is a four-direction Sobel filter corresponding to angles 0, π / 4, π / 2, and 3π / 4 when the horizontal direction of the input image is 0 [rad]. The extended Sobel filter X in the direction of the angle x (where 0 <x <π) is A, and the basic Sobel filter corresponding to the angle 0 is A, and the basic Sobel filter corresponding to the angle π / 4 is B. When the basic Sobel filter corresponding to the angle π / 2 is C, the basic Sobel filter corresponding to the angle 3π / 4 is D, and the Sobel filter corresponding to the angle π is E (= −A). Are given by the following equations (1) to (4).
When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)
In this way, the edge component direction is determined for more directions by calculating the strength of the edge component using an extended Sobel filter with an angle x in addition to the conventional four-direction basic Sobel filter. be able to. By performing smoothing using an edge-preserving smoothing filter corresponding to the determined direction, it is possible to select an edge-preserving smoothing filter that is more suitable for the edge component and perform smoothing with high accuracy. Become.

  In the present invention, the Sobel filter for calculating the strength of the edge component in the specific direction is calculated by the following equations (5) to (8). As a result, the Sobel filter can be expanded in an arbitrary direction (an arbitrary angle x when the horizontal direction of the input image is 0 [rad]).

  In the present invention, an edge-preserving type smoothing filter that stores edge components in a specific direction and smoothes pixel values is stored for a plurality of directions, and the direction of edge components determined using a Sobel filter The specific area is smoothed by selecting an edge-preserving smoothing filter according to the above. As a result, the edge component can be stored and noise or the like can be removed or reduced from the specific region.

  In the present invention, the maximum value and the minimum value are selected from the calculated plurality of edge strengths, and it is determined whether or not the difference between them exceeds a threshold value. When the difference does not exceed the threshold value, the specific region can be regarded as a substantially flat image that does not include an edge component, and thus smoothing can be performed using a smoothing filter that is not an edge-preserving type.

In the present invention, whether to apply a smoothing processing result using an edge-preserving smoothing filter is determined based on one or more conditions.
For example, if the input image is an image having a clear edge or pattern, the original pixel value is used without applying the result of the smoothing process because the image quality may be deteriorated by performing the smoothing process. To do.

  For example, filter processing using a Laplacian filter is performed on the extracted specific region, the strength of the edge component included in the specific region is calculated, and it is determined whether or not the calculated strength exceeds a threshold value. When the edge component intensity exceeds the threshold value, the smoothing process result is applied. When the edge component intensity does not exceed the threshold value, the smoothing process result is not applied.

  Also, for example, if the input image is an image containing fine textures such as one pixel unit, the edge strength is determined to be small depending on the arrangement, etc., and the edges are smoothed without being saved by the edge preserving type smoothing filter. There is a possibility that the texture is unclear due to the processing. Therefore, the frequency of the pixels in the specific region (the number of increase / decrease of the pixel value) is calculated. If the calculated frequency is larger than the threshold value, the specific region is regarded as a texture region, and the result of the smoothing process is calculated. Does not apply. Thereby, it is possible to prevent the texture from being deteriorated by the smoothing process.

  Further, for example, when only the target pixel in the specific area is greatly different from other pixels, it is difficult to determine whether this is texture or noise. In such a case, in order to reduce the influence of smoothing, the difference between the pixel values of the target pixel before and after the smoothing process is calculated. When the calculated difference exceeds the threshold value, the result of the smoothing process is not applied. Thereby, the influence of smoothing can be suppressed within the threshold value.

  In the present invention, the strength of edge components in a plurality of directions is calculated by a Sobel filter, and the strength of edge components in other directions is estimated based on the calculated strength. In the present invention, in addition to the four-direction basic Sobel filters corresponding to the angles 0, π / 4, π / 2, and 3π / 4, an extended Sobel filter corresponding to the direction of the angle x is used. Calculate the intensity. As a result, the strength of the edge component in more directions can be obtained, and the direction of the edge component can be determined more accurately. Therefore, it is possible to select an edge-preserving smoothing filter that is more suitable for the edge component and perform smoothing with high accuracy, and thus it is possible to generate a high-quality output image in which noise is accurately removed or reduced.

It is a block diagram which shows the structure of the display apparatus which concerns on this Embodiment. It is a block diagram which shows the example of 1 structure of an image process part. It is a flowchart which shows an example of the procedure of the noise removal process by an image process part. It is a flowchart which shows the outline of the procedure of the smoothing determination process which an image process part performs. It is a schematic diagram for demonstrating the horizontal direction determination process performed in step S21. It is a flowchart which shows the procedure of the horizontal direction determination process which an image processing apparatus performs. It is a schematic diagram for demonstrating the vertical direction determination process performed in step S22. It is a schematic diagram for demonstrating a noise boundary direction determination process. It is a schematic diagram for demonstrating a noise boundary position determination process. It is a schematic diagram which shows an example of a smoothing filter. It is a schematic diagram which shows an example of a smoothing filter. It is a schematic diagram which shows an example of the edge preservation | save type smoothing filter. It is a schematic diagram which shows an example of the edge preservation | save type smoothing filter. It is a schematic diagram which shows an example of the edge preservation | save type smoothing filter. It is a schematic diagram which shows an example of the Sobel filter for direction determination. It is a graph which shows an example of a response | compatibility with the value calculated by a Sobel filter, and an angle. It is a schematic diagram for demonstrating the interpolation process for intensity | strength estimation. It is a graph which shows an example of a response | compatibility with the value calculated by a Sobel filter, and an angle. It is a flowchart which shows the procedure of an edge preservation | save type smoothing process. It is a schematic diagram for demonstrating edge amount determination using a Laplacian filter. It is a schematic diagram for demonstrating the application determination based on a frequency. It is a schematic diagram for demonstrating the application determination based on a frequency. It is a flowchart which shows the procedure of an application determination process. It is a flowchart which shows the procedure of an application determination process. It is a block diagram which shows the structure of PC concerning a modification.

  Hereinafter, the present invention will be specifically described with reference to the drawings showing embodiments thereof. In the present embodiment, a display device that displays an image on a display unit such as a liquid crystal panel by performing image processing for removing or reducing noise on an input image from an external device such as a PC (Personal Computer) is taken as an example. Next, the configuration of the image processing apparatus, the image processing method, and the computer program will be described. FIG. 1 is a block diagram illustrating a configuration of a display device according to the present embodiment. In the figure, reference numeral 1 denotes a display device, which is a device that performs various image processing on a still image or a moving image input from an external device such as the PC 5 and displays it on the liquid crystal panel 13.

  The display device 1 includes an image input unit 16 for driving the liquid crystal panel 13 based on an input image from the PC 5, an image expansion unit 17, an image processing unit 20, a panel drive unit 18, and the like. The display device 1 also includes a backlight 14 that irradiates light for display onto the back surface of the liquid crystal panel 13 and a light driving unit 15 that drives the backlight 14. In addition, the display device 1 includes an operation unit 12 that receives a user operation, and a control unit 11 that controls the operation of each unit in the device according to the received operation.

  The control part 11 is comprised using arithmetic processing units, such as CPU (Central Processing Unit) or MPU (Micro Processing Unit). The operation unit 12 includes one or a plurality of switches arranged on the front peripheral edge or the side surface of the casing of the display device 1. The operation unit 12 receives a user operation using these switches, and the received operation content is displayed on the control unit 11. To notify. For example, the user can perform an operation for changing the brightness setting or the color balance setting related to the image display using the operation unit 12, and the control unit 11 can change each unit in the apparatus according to the setting content received by the operation unit 12. To control the operation.

  The image input unit 16 has a connection terminal for connecting an external device, and an external device such as the PC 5 is connected via a cable for video signals. In the present embodiment, image data compressed by a compression method such as MPEG or JPEG is input from the PC 5 to the display device 1 as an input image. The image input unit 16 gives an input image from the PC 5 to the image expansion unit 17. The image expansion unit 17 expands the input image from the image input unit 16 by a method corresponding to each compression method, and gives the image to the image processing unit 20.

  The image processing unit 20 can perform various image processes on the input image given from the image expansion unit 17. In the present embodiment, the image processing unit 20 can perform image processing to remove (or reduce) stepped noise such as block noise included in the input image. Details of image processing for noise removal performed by the image processing unit 20 will be described later. The image processing unit 20 gives the image subjected to the image processing to the panel driving unit 18.

  The panel drive unit 18 generates and outputs a drive signal for driving each pixel constituting the liquid crystal panel 13 in accordance with the input image given from the image processing unit 20. The liquid crystal panel 13 is a display device that displays an image by arranging a plurality of pixels in a matrix and changing the transmittance of each pixel according to a drive signal from the panel drive unit 18.

  The backlight 14 is configured using a light source such as an LED (Light Emitting Diode) or a CCFL (Cold Cathode Fluorescent Lamp), and irradiates the back surface of the liquid crystal panel 13 with light. The backlight 14 emits light by a driving voltage or a driving current given from the light driving unit 15. The light driving unit 15 generates a driving voltage or a driving current according to a control signal from the control unit 11 and outputs the driving voltage or driving current to the backlight 14. For example, the control unit 11 determines the drive amount of the backlight 14 according to the brightness setting received by the operation unit 12, and outputs a control signal corresponding to the determined drive amount to the light drive unit 15.

  FIG. 2 is a block diagram illustrating a configuration example of the image processing unit 20, and illustrates blocks related to noise removal processing from an input image. The image processing unit 20 includes a specific region extracting unit 21 that extracts a region of a specific size from the input image. The specific area extraction unit 21 performs a process of extracting a specific area of, for example, 5 × 5 pixels centered on the target pixel for one pixel (hereinafter referred to as the target pixel) in the input image. The specific region extracted by the specific region extraction unit 21 is given to the smoothing determination unit 22, the first smoothing unit 23, the second smoothing unit 24, the application determination unit 25, and the pixel value selection unit 26.

  Whether the smoothing determination unit 22 performs the smoothing of the pixel of interest included in the specific region by examining the pixel values of the plurality of pixels included in the specific region extracted by the specific region extraction unit 21 and changes thereof. The process which determines is performed. The smoothing determination unit 22 notifies the pixel value selection unit 26 of whether or not smoothing processing can be performed. Moreover, the smoothing determination part 22 performs the process which determines the direction and position of a noise boundary in a specific area, when a noise boundary exists in a specific area. The smoothing determination unit 22 gives the determined direction and position of the noise boundary to the first smoothing unit 23.

  The first smoothing unit 23 stores a plurality of smoothing filters, and smoothes the image by selecting one of the smoothing filters and performing a filtering process on a specific region. The first smoothing unit 23 selects one smoothing filter according to the direction and position of the noise boundary given from the smoothing determination unit 22. The first smoothing unit 23 supplies the pixel value selection unit 26 with the result of smoothing the specific area by the smoothing filter, that is, the pixel value of the pixel of interest in the smoothed specific area.

  The second smoothing unit 24 performs a filtering process using an edge-preserving smoothing filter on the specific region. An edge-preserving smoothing filter can store high-frequency components (such as edges) included in a specific region to smooth pixel values, and can perform filtering processing that does not cause significant deterioration in image quality. . The second smoothing unit 24 stores a plurality of edge-preserving smoothing filters corresponding to the edge direction, determines the direction of the edge included in the specific region, and smoothes using the filter corresponding to the edge direction. I do. The second smoothing unit 24 gives the pixel value selection unit 26 the result of smoothing the specific region by the edge-preserving smoothing filter, that is, the pixel value of the target pixel in the smoothed specific region. The smoothing result by the second smoothing unit 24 is also given to the application determining unit 25.

  The application determining unit 25 determines whether to apply the smoothing result by the second smoothing unit 24 based on the feature of the pixel value of the specific region and / or the smoothing result by the second smoothing unit 24, or the like. Do. The application determination unit 25 determines, for example, based on the amount of edge component in the specific region, the change pattern of the pixel value in the specific region, and / or the difference in pixel values before and after smoothing by the second smoothing unit 24. I do. The application determination unit 25 gives the determination result to the pixel value selection unit 26.

  In the pixel value selection unit 26, the pixel value smoothed by the first smoothing unit, the pixel value smoothed by the second smoothing unit, and the smoothing are performed on the target pixel of the input image. Three pixel values of the original pixel values that have not been performed are input. The pixel value selection unit 26 selects one pixel value from the three input pixel values according to whether or not the smoothing process determined by the smoothing determination unit 22 is performed and the determination result by the application determination unit 25. Output.

  When it is determined that the smoothing determination unit 22 performs the smoothing process, the pixel value selection unit 26 selects and outputs the pixel value smoothed by the first smoothing unit 23. When it is determined that the smoothing determination unit 22 does not perform the smoothing process and the application determination unit 25 determines to apply, the pixel value selection unit 26 is smoothed by the second smoothing unit 24. Select and output pixel values. When it is determined that the smoothing determination unit 22 does not perform the smoothing process and the application determination unit 25 determines that the smoothing process is not performed, the pixel value selection unit 26 selects the original pixel value that has not been smoothed. Select to output.

  The image processing unit 20 can generate an output image by performing the processing of the specific area extraction unit 21 to the pixel value selection unit 26 for all the pixels of the input image, and outputs the generated image to the panel drive unit 18. To do. The image processing unit 20 shown in the block diagram of FIG. 2 has a configuration in which the first smoothing unit 23, the second smoothing unit 24, and the like perform processing in parallel and finally select the processing result of each processing. However, it is not limited to this. The image processing unit 20 may sequentially determine whether or not to perform the smoothing process as described below, and may perform one of the smoothing processes when the condition is satisfied.

  FIG. 3 is a flowchart illustrating an example of a procedure of noise removal processing by the image processing unit 20. The image processing unit 20 selects one target pixel from the input image from the image expansion unit 17 (step S1), and extracts a specific area having a predetermined size including the selected target pixel (step S2). Next, the image processing unit 20 performs a smoothing determination process on the extracted specific area (step S3), and determines whether or not it is determined in the smoothing determination process to perform the smoothing process on the target pixel in the specific area. Check (step S4).

  When it is determined that the smoothing process is to be performed (S4: YES), the image processing unit 20 performs the filtering process for the specific region using the smoothing filter (step S5). At this time, the image processing unit 20 performs filtering processing by selecting one smoothing filter from a plurality of prestored smoothing filters based on the direction and position of the noise boundary determined in the smoothing determination processing. The image processing unit 20 outputs the pixel value of the target pixel in the specific region that has been subjected to the filtering process using the smoothing filter as a processing result (step S6).

  When it is determined that the smoothing process is not performed (S4: NO), the image processing unit 20 performs the filtering process for the specific region using the edge preserving type smoothing filter (step S7), and the filtering process. An application determination process related to the result is performed (step S8). At this time, the image processing unit 20 performs filtering processing by selecting one of a plurality of edge-preserving smoothing filters stored in advance based on the direction of the edge included in the specific region. Further, the image processing unit 20 performs application determination processing based on one or more conditions among the amount of edge components in the specific region, the change pattern of the pixel value in the specific region, the smoothing result in step S7, and the like. .

  Through the application determination process, the image processing unit 20 determines whether to apply the result of the filtering process of the edge-preserving smoothing filter (step S9). When it determines with applying (S9: YES), the image process part 20 outputs the pixel value of the attention pixel of the specific area | region which was filtered by the edge preservation | save type smoothing filter as a processing result (step S10). If it is determined not to be applied (S9: NO), the image processing unit 20 outputs the pixel value of the target pixel selected in step S1 (step S11).

  The image processing unit 20 can generate an output image by repeatedly performing the processes in steps S1 to S11 described above for all the pixels of the input image. Each pixel of the output image to be generated is a pixel obtained by performing filtering using a smoothing filter on each pixel of the input image, a pixel obtained by performing filtering using an edge preserving smoothing filter, or smoothing. Any pixel that has not been performed (the same pixel as the input image), and the output image is an image in which staircase noise such as block noise has been removed or reduced from the input image.

<1. Smoothing judgment processing>
<1-1. Process Overview>
Next, the smoothing determination process performed by the image processing unit 20 will be described. The smoothing determination process is a process performed by the smoothing determination unit 22 in FIG. 2, and is a process performed in step S3 in FIG. FIG. 4 is a flowchart illustrating an outline of the procedure of the smoothing determination process performed by the image processing unit 20. First, the image processing unit 20 performs a horizontal direction (horizontal direction) determination process (step S21) and a vertical direction (longitudinal direction) determination process (step S22) on the specific area extracted from the input image, and sets the specific area. It is determined whether or not the included pixel of interest is to be smoothed (step S23). If it is determined that the target pixel is not subject to the smoothing process (S23: NO), the image processing unit 20 ends the smoothing determination process.

  When it is determined that the target pixel is the target of the smoothing process (S23: YES), the image processing unit 20 performs a direction determination process for the noise boundary existing in the specific area (step S24). From the result of the direction determination process, the image processing unit 20 determines whether or not a noise boundary extending in one of the horizontal direction and the vertical direction exists in the specific region (step S25).

  When it is determined that there is no noise boundary in the horizontal direction or the vertical direction (S25: NO), the image processing unit 20 determines whether there is a noise boundary extending in another direction in the specific region ( Step S26). Here, the noise boundary extending in the other direction is, for example, a noise boundary extending in a specific direction such as 45 ° or 135 ° in the specific region, or extending in the horizontal direction and the vertical direction in the specific region, for example. It is a noise boundary such as an existing L-shape, T-shape, or cross shape. If it is determined that there is a noise boundary in another direction (S26: YES), the image processing unit 20 ends the smoothing determination process.

  If no noise boundary exists in the specific area, the specific area can be regarded as corresponding to an internal area of block noise. Therefore, when it is determined that there is no noise boundary in another direction in the specific area (S26: NO), the image processing unit 20 determines whether or not the size of the specific area is a predetermined size (step S27). . If the specific area is not the predetermined size (S27: NO), the image processing unit 20 enlarges the specific area (step S28), returns to step S24, and performs the noise boundary direction determination process again. If the specific area has a predetermined size (S27: YES), the image processing unit 20 ends the smoothing determination process.

  If it is determined that a horizontal or vertical noise boundary exists in the specific area (S25: YES), the image processing unit 20 performs a noise boundary position determination process in the specific area (step S29), and smoothes it. The determination process ends.

<1-2. Horizontal / Vertical judgment>
FIG. 5 is a schematic diagram for explaining the horizontal direction determination processing performed in step S21. The image processing unit 20 selects one target pixel from the input image, and extracts a specific area 100 having a predetermined size (5 × 5 pixels in the illustrated example) including the target pixel. In the drawing, the pixel of interest in the specific area 100 is shown with hatching. In the horizontal direction determination process, the image processing unit 20 first calculates a primary differential value of a pixel value between pixels adjacent in the horizontal direction in the specific region 100 (that is, a difference in pixel value between adjacent pixels). As a result, the image processing unit 20 obtains a horizontal primary differential value matrix 101 composed of 4 × 5 primary differential values. The image processing unit 20 binarizes the horizontal primary differential value matrix 101 by comparing the absolute value of each primary differential value of the horizontal primary differential value matrix 101 with a predetermined threshold value (for example, the primary differential value matrix). 1 when the absolute value of the value ≧ the threshold value, and 0 when the absolute value of the primary differential value <the threshold value). Next, the image processing unit 20 performs an OR (logical sum) operation on the four binarized primary differential values arranged in the horizontal direction, and a horizontal primary differential value OR sequence 102 composed of five calculation results. Get.

  Further, the image processing unit 20 calculates a differential value (that is, a secondary differential value) of primary differential values adjacent in the horizontal direction in the horizontal primary differential value matrix 101. As a result, the image processing unit 20 obtains a horizontal secondary differential value matrix 103 composed of 3 × 5 secondary differential values. The image processing unit 20 binarizes the horizontal secondary differential value matrix 103 by comparing the absolute value of each secondary differential value of the horizontal secondary differential value matrix 103 with a predetermined threshold (for example, secondary differential value). If the absolute value of the value ≧ the threshold value, the absolute value of the first and second differential values <0 if the threshold value is satisfied). Next, the image processing unit 20 performs an OR operation on the three binarized secondary differential values arranged in the horizontal direction, and obtains a horizontal secondary differential value OR sequence 104 composed of five calculation results. The threshold for binarizing the primary differential value and the threshold for binarizing the secondary differential value may be the same value or different values, and the design stage of the display device 1 And so on.

  Next, the image processing unit 20 performs an OR operation of two values at corresponding positions of the horizontal primary differential value OR sequence 102 and the horizontal secondary differential value OR sequence 104, and a horizontal OR sequence 105 constituted by five calculation results. Get. The image processing unit 20 performs an OR operation on the upper three values (that is, the first to third values) of the horizontal OR column 105 to obtain the horizontal upper OR value 106, and the lower three values (that is, the third to fifth values). The horizontal lower OR value 107 is obtained by performing an OR operation of the (th value). Further, the image processing unit 20 performs an AND operation on the horizontal upper OR value 106 and the horizontal lower OR value 107 to obtain a horizontal direction determination result 108.

  The horizontal direction determination result 108 obtained in this way is 1-bit information whose value is “0” or “1”. The horizontal direction determination result 108 indicates whether or not the target pixel in the specific area 100 is included in a block having a low frequency component (a change in the pixel value is constant and small in the horizontal direction). When the value of the horizontal direction determination result 108 is “0”, it indicates that the target pixel may be included in the low frequency component block. Conversely, when the value of the horizontal direction determination result 108 is “1”, it indicates that the pixel of interest is not included in the low-frequency component block (that is, cannot be block noise).

  FIG. 6 is a flowchart illustrating a procedure of horizontal direction determination processing performed by the image processing apparatus 20. The image processing unit 20 calculates a primary differential value between pixels adjacent in the horizontal direction of the specific region 100 (step S31). The image processing unit 20 binarizes the primary differential value by comparing the absolute value of the calculated primary differential value with a threshold (step S32), and performs horizontal OR on the binarized primary differential value. Calculation is performed (step S33). Further, the image processing unit 20 calculates a secondary differential value in the horizontal direction of the specific region 100 by further calculating a horizontal differential value with respect to the calculation result in step S31 (step S34). The image processing unit 20 binarizes the secondary differential value by comparing the absolute value of the calculated secondary differential value with a threshold (step S35), and performs horizontal OR on the binarized secondary differential value. Calculation is performed (step S36).

  Next, the image processing unit 20 further ORs the calculation result of step S33 and the calculation result of step S36 at the corresponding position (step S37). Thus, a plurality of OR operation values are obtained, and the image processing unit 20 performs an OR operation on the upper half value (step S38) and an OR operation on the lower half value (step S39). Further, the image processing unit 20 performs an AND operation on the calculation result of step S38 and the calculation result of step S39 (step S40), and ends the horizontal direction determination process.

  FIG. 7 is a schematic diagram for explaining the vertical direction determination processing performed in step S22. Note that the vertical direction determination process is substantially the same as the horizontal direction determination process except that the calculation direction is different. In the vertical direction determination processing, the image processing unit 20 first calculates a primary differential value of pixel values between pixels adjacent in the vertical direction in the specific region 100. As a result, the image processing unit 20 obtains a vertical primary differential value matrix 111 composed of 5 × 4 primary differential values. The image processing unit 20 binarizes the vertical primary differential value matrix 111 by comparing the absolute value of each primary differential value of the vertical primary differential value matrix 111 with a predetermined threshold value. Next, the image processing unit 20 performs an OR operation on the four binarized primary differential values arranged in the vertical direction, and obtains a vertical primary differential value OR row 112 including five calculation results.

  Further, the image processing unit 20 calculates a differential value (that is, a secondary differential value) of primary differential values adjacent in the vertical direction in the vertical primary differential value matrix 111. As a result, the image processing unit 20 obtains a vertical secondary differential value matrix 113 composed of 5 × 3 secondary differential values. The image processing unit 20 binarizes the vertical secondary differential value matrix 113 by comparing the absolute value of each secondary differential value of the vertical secondary differential value matrix 113 with a predetermined threshold value. Next, the image processing unit 20 performs an OR operation on the three binarized secondary differential values arranged in the vertical direction, and obtains a vertical secondary differential value OR row 114 including five calculation results.

  Next, the image processing unit 20 performs an OR operation on the two values at the corresponding positions of the vertical primary differential value OR row 112 and the vertical secondary differential value OR row 114, and a vertical OR row 115 constituted by five calculation results. Get. The image processing unit 20 performs an OR operation on the upper three values of the vertical OR row 115 to obtain the vertical upper OR value 116 and obtains a vertical lower OR value 117 by performing an OR operation on the lower three values. Further, the image processing unit 20 performs an AND operation on the upper vertical OR value 116 and the lower vertical OR value 117 to obtain a vertical direction determination result 118.

  The vertical direction determination result 118 obtained in this way is 1-bit information whose value is “0” or “1”. The vertical direction determination result 118 indicates whether or not the target pixel in the specific area 100 is included in a block having a low frequency component (a change in the pixel value is constant and small in the vertical direction). When the value of the vertical direction determination result 118 is “0”, it indicates that the target pixel may be included in the low frequency component block. If the value of the horizontal direction determination result 108 is “0” and the value of the vertical direction determination result 118 is “0”, it can be determined that the pixel of interest is included in the low frequency component block. When the value of the vertical direction determination result 108 is “1”, it indicates that the pixel of interest is not included in the low frequency component block.

  The vertical direction determination process is substantially the same as the horizontal direction determination process, and thus the flowchart is not shown. The flowchart relating to the vertical direction determination process can be obtained by replacing the description of “horizontal direction” with “vertical direction” in the flowchart of the horizontal direction determination process shown in FIG.

  The image processing unit 20 smoothes the target pixel in the specific region 100 based on the horizontal direction determination result 108 obtained by the horizontal direction determination process and the vertical direction determination result 118 obtained by the vertical direction determination process. It is determined whether or not to be the target of. Specifically, when the horizontal direction determination result 108 is “0” and the value of the vertical direction determination result 118 is “0”, the image processing unit 20 sets the target pixel as the target of the smoothing process and thereafter Perform the process. In addition, when the value of either the horizontal direction determination result 108 or the vertical direction determination result 118 is “1”, the image processing unit 20 excludes the target pixel from the smoothing process target and ends the smoothing determination process. To do.

<1-3. Noise boundary direction determination processing>
When it is determined that the target pixel in the specific area 100 is the target of the smoothing process, the image processing unit 20 performs a process of determining the extending direction of the noise boundary included in the specific area 100. FIG. 8 is a schematic diagram for explaining the noise boundary direction determination processing. In the noise boundary direction determination processing, the image processing unit 20 determines whether the direction of the noise boundary included in the specific region 100 is any of the horizontal pattern in FIG. 8A, the vertical pattern in FIG. 8B, the internal pattern in FIG. 8C, or the other pattern in FIG. To determine whether The horizontal pattern in FIG. 8A is a pattern in which a noise boundary extends in the horizontal direction (lateral direction) of the specific region 100. The vertical pattern in FIG. 8B is a pattern in which a noise boundary extends in the vertical direction (longitudinal direction) of the specific region 100. The internal pattern in FIG. 8C is a pattern when the noise boundary is not included in the specific area 100 and the specific area 100 is an internal area of block noise. The other patterns in FIG. 8D are all patterns other than those in FIGS. 8A to 8C, and the illustrated pattern is an example.

  The image processing unit 20 performs noise boundary direction determination processing using data generated in the course of the above-described horizontal direction determination processing and vertical direction determination processing. Specifically, the image processing unit 20 uses the horizontal OR column 105 generated by the horizontal direction determination process and the vertical OR row 115 generated by the vertical direction determination process. The image processing unit 20 determines whether or not all of the five values included in the horizontal OR row 105 (however, when the specific area 100 is enlarged, more than five values are included) are “0”. Similarly, the image processing unit 20 determines whether or not all five values included in the vertical OR row 115 are “0”. When all the values of the horizontal OR row 105 are “0”, the pixel value of the specific area 100 changes smoothly in the horizontal direction. When all the values of the vertical OR row 115 are “0”, the pixel value of the specific area 100 smoothly changes in the vertical direction.

  Therefore, when the horizontal OR column 105 is not all “0” and the vertical OR row 115 is all “0”, the image processing unit 20 determines that the noise boundary of the specific region 100 is the horizontal pattern in FIG. 8A. When the horizontal OR column 105 is all “0” and the vertical OR row 115 is not all “0”, the image processing unit 20 determines that the noise boundary of the specific region 100 is the vertical pattern of FIG. 8B. When the horizontal OR column 105 is all “0” and the vertical OR row 115 is all “0”, the image processing unit 20 sets the specific region 100 as an internal pattern (that is, a noise boundary in the input image). (A region surrounded by, sandwiched or adjacent). Further, when the horizontal OR column 105 is not all “0” and the vertical OR row 115 is not all “0”, the image processing unit 20 determines that the noise boundary of the specific region 100 is the other pattern in FIG. 8D.

  Further, when the image processing unit 20 determines that the specific area 100 is the internal pattern of FIG. 8C in the noise boundary direction determination process, the image processing unit 20 enlarges the specific area 100 (for example, 5 × 5 pixels → 7 × 7 pixels). → 9 × 9 pixels → ...). The image processing unit 20 calculates the horizontal OR column 105 and the vertical OR row 115 for the enlarged specific region 100 by the same method as that shown in FIGS. Based on the calculated horizontal OR column 105 and vertical OR row 115, the image processing unit 20 determines which of the patterns shown in FIG. The image processing unit 20 determines that the noise boundary of the enlarged specific area 100 is the horizontal pattern in FIG. 8A, the vertical pattern in FIG. 8B, or the other pattern in FIG. 8D, or the size of the specific area 100 is a predetermined size. The process is repeated by enlarging the specific area 100 until it becomes 9 (for example, 9 × 9 pixels). When the specific area 100 reaches a predetermined size, the image processing unit 20 ends the smoothing determination process without further enlarging the specific area 100.

<1-4. Noise boundary position determination processing>
When the noise boundary of the specific area 100 is the horizontal pattern of FIG. 8A or the vertical pattern of FIG. 8B, the image processing unit 20 performs a process of determining the position of the noise boundary in the specific area 100. FIG. 9 is a schematic diagram for explaining the noise boundary position determination process, and shows an example in which the noise boundary of the specific region 100 is a vertical pattern. The position of the noise boundary of the vertical pattern existing in the specific area 100 of 5 × 5 pixels is one of the four positions shown in FIGS. 9A to 9D. Since the position determination process when the noise boundary is a horizontal pattern is substantially the same as the case of the vertical pattern, the description is omitted.

  The image processing unit 20 performs noise boundary direction determination processing using the data generated in the above-described horizontal direction determination processing. Specifically, the image processing unit 20 uses a binarized horizontal secondary differential value matrix 103a obtained by binarizing the horizontal secondary differential value matrix 103 generated in the horizontal direction determination process by comparison with a threshold value. For example, as shown in FIG. 9A, when the noise boundary is located at the left end of the specific region 100, the 3 × 5 binarized horizontal second-order differential value matrix 103a includes “1” in the first column, The third column is a pattern of “0”. When the noise boundary is located at the center left of the specific region 100 as shown in FIG. 9B, the 3 × 5 binarized horizontal second-order differential value matrix 103a includes “1” in the first and second columns, The third column is a pattern of “0”. When the noise boundary is located at the center right of the specific region 100 as shown in FIG. 9C, the 3 × 5 binarized horizontal second-order differential value matrix 103a includes “1” in the second and third columns, The first column has a pattern of “0”. When the noise boundary is located at the right end of the specific region 100 as shown in FIG. 9D, the 3 × 5 binarized horizontal second-order differential value matrix 103a includes “1” in the third column, and the first and second columns. The column has a pattern of “0”.

  Therefore, the image processing unit 20 examines the arrangement pattern of “0” and “1” included in the binarized horizontal second-order differential value matrix 103a generated by the horizontal direction determination process, thereby making the vertical included in the specific region 100. The position of the noise boundary of the pattern can be determined. The determination of the position of the noise boundary of the horizontal pattern is the same, and the image processing unit 20 binarizes the vertical second derivative matrix 113 generated by the vertical direction determination process by binarization by comparison with a threshold value. The position of the noise boundary of the horizontal pattern included in the specific area 100 is determined by examining the arrangement pattern of “0” and “1” included in the binarized vertical second-order differential value matrix using the second derivative matrix. can do.

<2. Smoothing process>
When it is determined that the smoothing process is performed in the above-described smoothing determination process, the image processing unit 20 performs a process of smoothing the pixel value of the specific region 100. The image processing unit 20 stores a plurality of smoothing filters for performing the smoothing process, and the one based on the direction determined by the noise boundary direction determination process, the position determined by the noise boundary position determination process, and the like. A smoothing filter is selected, and the pixel value of the pixel of interest in the specific area 100 is smoothed using the selected smoothing filter.

10 and 11 are schematic diagrams illustrating an example of the smoothing filter. The size of the smoothing filter for the specific region 100 of 5 × 5 pixels is 5 × 5. When each pixel value of the specific region is a _ij (i = 1 to 5, j = 1 to 5) and each value of the smoothing filter is f _ij (i = 1 to 5, j = 1 to 5), The image processing unit 20 calculates a value A obtained by performing a matrix operation of A = (a ₁₁ × f ₁₁ + a ₁₂ × f ₁₂ +... + A ₅₅ × f ₅₅ ) / (f ₁₁ + f ₁₂ +... + F ₅₅ ). The pixel value of the target pixel is a smoothed value. This calculation method is similarly used in filtering processing using other filters (edge-preserving smoothing filter, Sobel filter, Laplacian filter, etc.) described below.

  The example illustrated in FIG. 10A is a smoothing filter used when the noise boundary of the specific region 100 is a vertical pattern. In the smoothing filter of FIG. 10A, five values in the third row are set to 1 and other values are set to 0. By using the smoothing filter of FIG. 10A, the average value of the target pixel of the specific region 100 and the four pixels located on the left and right sides of the target pixel can be calculated. The average value of the pixels is output as a smoothing result of the target pixel. The smoothing filter in FIG. 10A performs smoothing by calculating an average value of pixel values in a direction that intersects the noise boundary of the vertical pattern.

  The example shown in FIG. 10B is a smoothing filter used when the noise boundary of the specific region 100 is a vertical pattern, as in FIG. 10A. However, the smoothing filter in FIG. 10B further considers the position of the noise boundary, and is used when the position of the noise boundary is the center left shown in FIG. 9B. In the smoothing filter of FIG. 10B, the center value is set to 3, the value on the left side is set to 2, and the other values are set to 0. By using the smoothing filter in FIG. 10B, it is possible to calculate a weighted average value of the target pixel of the specific region 100 and the pixel adjacent to the left side, and the image processing unit 20 calculates the average value of the target pixel. Output as smoothing result.

  The example illustrated in FIG. 10C is a smoothing filter used when the noise boundary of the specific region 100 is a horizontal pattern. In the smoothing filter of FIG. 10C, the five values in the third column are set to 1 and the other values are set to 0. By using the smoothing filter in FIG. 10C, the average value of the target pixel of the specific region 100 and the four pixels located on the upper and lower sides of the target pixel can be calculated. The average value of the pixels is output as a smoothing result of the target pixel. The smoothing filter in FIG. 10C performs smoothing by calculating an average value of pixel values in a direction intersecting the noise boundary of the horizontal pattern.

  The example illustrated in FIG. 11D is a smoothing filter used when the noise boundary of the specific region 100 is another pattern. In the smoothing filter of FIG. 11D, all the values of 5 × 5 = 25 are set to 1. By using the smoothing filter of FIG. 11D, the average value of the pixel values of all the pixels in the specific region 100 can be calculated, and the image processing unit 20 outputs the average value as a smoothing result of the target pixel.

  The example shown in FIG. 11E is a smoothing filter that is used for a specific region 100 in which no noise boundary exists in the specific region 100 of 5 × 5 pixels and there is a noise boundary when expanded to 7 × 7 pixels. . Note that the smoothing filter of FIG. 11E is an enlargement of the smoothing filter of FIG. 11D and is used when the noise boundary is another pattern. Similarly, the smoothing filters of FIGS. 10A-C can be expanded to a size of 7 × 7. By using the smoothing filter in FIG. 11E, the average value of the pixel values of all the pixels in the enlarged specific region 100 can be calculated, and the image processing unit 20 outputs the average value as a smoothing result of the target pixel. To do.

  As described above, the image processing unit 20 selects a smoothing filter according to the direction and position of the noise boundary, and smoothes and outputs the pixel value of the pixel of interest in the specific region 100. Note that the smoothing filter shown in FIGS. 10 and 11 is an example, and the present invention is not limited to this. In addition, the image processing unit 20 determines whether or not there is a noise boundary, but does not determine the direction and position thereof, and performs filtering processing using the smoothing filter illustrated in FIG. 11 for the specific region 100 determined to have a noise boundary. It is good also as a structure to perform.

<3. Edge-preserving smoothing process>
<3-1. Edge-preserving smoothing filter>
When it is determined that the smoothing process is not performed in the above-described smoothing determination process, the image processing unit 20 smoothes the pixel value of the target pixel in the specific region 100 using an edge-preserving smoothing filter. Process. The image processing unit 20 stores a plurality of edge-preserving smoothing filters, determines the direction of the edge component included in the specific region 100, selects one filter according to the determined direction of the edge component, The pixel value of the pixel of interest in the specific area 100 is smoothed using the selected filter.

  12 to 14 are schematic diagrams illustrating an example of an edge-preserving smoothing filter. In the present embodiment, in the image processing unit 20, the direction of the edge component of the specific region 100 is horizontal (0 °), diagonal 45 °, vertical (90 °), diagonal 135 °, diagonal 22.5 °, diagonal 67. It is determined whether there are eight directions of 5 °, oblique 112.5 °, oblique 157.5 °, or no direction. For this reason, the image processing unit 20 stores eight edge preserving smoothing filters corresponding to eight directions and an isotropic smoothing filter that is not an edge preserving type corresponding to the non-direction determination.

  The filter illustrated in FIG. 12A is a smoothing filter that stores edge components in the horizontal direction. When the image processing unit 20 determines that the direction of the edge component of the specific region 100 is the horizontal direction, the image processing unit 20 performs a smoothing process using the filter of FIG. 12A. When the image processing unit 20 performs the filtering process using the filter of FIG. 12A, the pixel value of the target pixel in the specific region 100 and the four pixels located on the left and right sides along the edge component direction with respect to the target pixel. A weighted average value with the pixel value of is calculated. The pixel values are weighted so that the pixels closer to the target pixel become heavier. The image processing unit 20 outputs the calculated average value as a smoothing result of the target pixel.

  The filter shown in FIG. 12B is a smoothing filter that stores an edge component in a 45 ° oblique direction. A weighted average value of the pixel of interest and the pixel values of four pixels located in the 45 ° oblique direction with respect to this pixel. Can be calculated. Similarly, the filter illustrated in FIG. 12C is a smoothing filter that stores edge components in the vertical (90 °) direction. The filter shown in FIG. 12D is a smoothing filter that stores edge components in the oblique 135 ° direction.

  The filter shown in FIG. 13E is a smoothing filter that stores edge components in the diagonal 22.5 ° direction, and weights the pixel of interest and the pixel values of six pixels positioned in the diagonal 22.5 ° direction. The average value obtained can be calculated. Similarly, the filter illustrated in FIG. 13F is a smoothing filter that stores edge components in a diagonal 67.5 ° direction. The filter illustrated in FIG. 13G is a smoothing filter that stores edge components in the oblique 112.5 ° direction. The filter shown in FIG. 13H is a smoothing filter that stores edge components in the oblique 157.5 ° direction.

  The filter shown in FIG. 14 is a smoothing filter used when it is determined that the direction of the edge component of the specific region 100 does not correspond to the above eight directions. The smoothing filter can calculate a weighted average value of the pixel values of all the pixels in the specific region 100 by performing weighting according to the distance from the target pixel.

  Further, based on the 8-direction edge preserving smoothing filter shown in FIGS. 12 and 13, a smoothing filter X for preserving edges in the direction of an arbitrary angle x (0 ° <x <180 °) is It can be generated using an expression. However, the edge preserving smoothing filter in the 0 ° direction shown in FIG. 12 is A, the edge preserving smoothing filter in the 45 ° direction is B, the edge preserving smoothing filter in the 90 ° direction is C, and 135 An edge-preserving smoothing filter in the direction of D is denoted by D. Further, the edge preserving smoothing filter in the 22.5 ° direction shown in FIG. 13 is E, the edge preserving smoothing filter in the 67.5 ° direction is F, and the edge preserving smoothing in the 112.5 ° direction is shown in FIG. The filter is G, and the edge-preserving smoothing filter in the 157.5 ° direction is H.

When 0 ° <x <22.5 °: X = αA + (1−α) E
When 22.5 ° <x <45 °: X = αE + (1-α) B
When 45 ° <x <67.5 °: X = αB + (1-α) F
When 67.5 ° <x <90 °: X = αF + (1−α) C
When 90 ° <x <112.5 °: X = αC + (1−α) G
When 112.5 ° <x <135 °: X = αG + (1−α) D
When 135 ° <x <157.5 °: X = αD + (1−α) H
When 157.5 ° <x <180 °: X = αH + (1−α) A

  In the above equation, α is a coefficient determined according to the angle x, and 0 <α <1. For example, when x = 10 °, α = (10−0) / (22.5−0) = 0.44. For example, when x = 75 °, α = (75-67.5) / (90-67.5) = 0.33. That is, when m <x <n, it can be determined by α = (x−m) / (n−m).

<3-2. Edge direction judgment>
(A) Edge Strength Calculation Processing The image processing unit 20 needs to determine the direction of the edge component of the specific region 100 in order to select one filter from the plurality of edge-preserving smoothing filters. For example, the image processing unit 20 can determine the direction of the edge component by performing a filtering process using a Sobel filter. FIG. 15 is a schematic diagram illustrating an example of a Sobel filter for direction determination. The image processing unit 20 is a diagram corresponding to each direction in order to determine whether the direction of the edge component is four directions of horizontal (0 °), diagonal 45 °, vertical (90 °), and diagonal 135 °. Four Sobel filters shown in 15A to D are used.

  The illustrated Sobel filter is a 3 × 3 matrix for determining the strength of edge components in a direction orthogonal to the arrangement direction of elements whose values are set to zero. The image processing unit 20 calculates the strength of the edge component by performing a filtering process using the illustrated Sobel filter on the 3 × 3 region centered on the pixel of interest in the specific region 100 (which is still calculated). Indicates that the closer the absolute value is to 0, the stronger the edge component is).

  As described above, the edge-preserving smoothing filter includes 22.5 °, 67.5 °, 112.5 °, and 4 filters corresponding to 0 °, 45 °, 90 °, and 135 °. Four filters corresponding to 157.5 ° are used. For this reason, the image processing unit 20 needs to acquire the strengths of the edge components for 22.5 °, 67.5 °, 112.5 °, and 157.5 °. Therefore, the image processing unit 20 uses one of the two methods described below (or a combination of both) to generate edge components related to 22.5 °, 67.5 °, 112.5 °, and 157.5 °. Calculate or estimate intensity.

  In the first method for acquiring the intensity of the edge component, the image processing unit 20 adds 22.5 °, 67. In addition to the four basic Sobel filters corresponding to 0 °, 45 °, 90 °, and 135 °. Four extended Sobel filters corresponding to 5 °, 112.5 °, and 157.5 ° are stored, and the strength of the edge component is calculated using these eight Sobel filters.

  The extended Sobel filter X corresponding to an arbitrary angle x [rad] can be calculated by the following equations (1) to (4). Note that 0 <x <π. A basic Sobel filter with an angle 0 (0 °) is A, a basic Sobel filter with an angle π / 4 (45 °) is B, a basic Sobel filter with an angle π / 2 (90 °) is C, and an angle 3π A / 4 (135 °) basic Sobel filter is designated as D. In addition, a Sobel filter having an angle π (180 °) can be obtained by inverting the sign of a basic Sobel filter having an angle 0. Therefore, the Sobel filter with the angle π is set to E (= −A).

When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)

  In the above equation, α is a coefficient determined according to the angle x, and 0 <α <1. For example, when x = π / 8, α = 1/2. For example, when x = π / 6, α = 2/3. That is, when 0 <x <π / 4, the coefficient α is determined according to the ratio of the difference between the angle x and the angle 0 of the calculated Sobel filter X and the difference between the angle x and π / 4. Can do. The same applies to other angle ranges. In m <x <n, it can be determined by α = (x−m) / (n−m).

  Further, based on the above formulas (1) to (4), the 3 × 3 Sobel filter for calculating the intensity of the edge component for an arbitrary angle x [rad] is represented by the following (5) to (8 ) Expression. Note that 0 ≦ x <π.

  By using the above equations (5) to (8), extended Sobel filters of 22.5 °, 67.5 °, 112.5 °, and 157.5 ° can be generated. Moreover, said (5)-(8) Formula can also produce | generate the basic Sobel filter shown to FIG.

  For example, a Sobel filter having an angle of 22.5 ° (x = π / 8) is expressed by α = 1/2, β = 3/2, γ = 3 / in the matrix (a) in the above equation (5). 2, δ = 1/2. Here, when the 0 ° Sobel filter shown in FIG. 15A and the 45 ° Sobel filter shown in FIG. 15C are compared, each value included in the 45 ° Sobel filter is 0 ° Sobel filter. It can be seen that these values are arranged rotated by 45 °. Similarly, the value of the 22.5 ° Sobel filter calculated by the above equation corresponds to the value obtained by rotating each value of the 0 ° Sobel filter by 22.5 °. Each value of the 22.5 ° Sobel filter corresponds to an average value of corresponding values of the 0 ° and 45 ° Sobel filters.

  In the first method for acquiring the intensity of the edge component, the image processing unit 20 stores four basic Sobel filters and four extended Sobel filters generated in advance, and uses these eight Sobel filters. To calculate the edge strength. The image processing unit 20 can calculate the strength of the edge component in eight directions, and selects one of the edge-preserving smoothing filters shown in FIGS. 12 and 13 based on the calculated strength. be able to. As described above, the image processing unit 20 may store the same number of Sobel filters in one-to-one correspondence with the stored angles of the edge-preserving smoothing filter. (In this case, since the image processing unit 20 can determine which smoothing filter to use based on the strength of the edge component calculated using each Sobel filter, the following (b) edge strength estimation processing is performed. No need.)

  On the other hand, in the second method for acquiring the intensity of the edge component, the image processing unit 20 stores the four Sobel filters shown in FIGS. The 5 °, 112.5 °, and 157.5 ° Sobel filters need not be stored. The intensity of the edge component related to the angle that does not store the Sobel filter is calculated by the image processing unit 20 in the following (b) edge intensity estimation process.

(B) Edge Strength Estimation Processing When the number of stored Sobel filters is smaller than the number of stored edge-preserving smoothing filters, the image processing unit 20 calculates using the Sobel filter. A process of estimating the strength of the edge component in a direction that cannot be performed is performed. The image processing unit 20 estimates the strength of edge components in other directions based on the strengths of a plurality of edge components calculated in (a) edge strength calculation processing.

  FIG. 16 is a graph showing an example of a correspondence between a value (absolute value) calculated by a Sobel filter and an angle. In the graph shown in FIG. 16, the horizontal axis represents the angle [°], and the vertical axis represents the absolute value of the value indicating the intensity of the edge component calculated by the Sobel filter. As shown in FIG. 16, when the edge component has a direction dependency, the change in the absolute value of the value calculated using the Sobel filter is a change having a maximum value and a minimum value. The image processing unit 20 of the display device 1 according to the present embodiment uses the edge component strengths in four directions of 0 °, 45 °, 90 °, and 135 ° calculated by the four Sobel filters shown in FIG. By performing an interpolation process based on this, the strength of the edge component in the other direction is estimated.

  FIG. 17 is a schematic diagram for explaining an interpolation process for intensity estimation. A point indicated by a black circle in the figure is a point indicating an intensity calculated by a filtering process using a Sobel filter. Here, it is assumed that four points A1 (X1, Y1), A2 (X2, Y2), A3 (X3, Y3), and A4 (X4, Y4) are calculated. Moreover, the points indicated by hatched circles in the figure are points C (Xi, Yi) indicating the intensity estimated based on A1 to A4. Point C is a point between points A2 and A3, and the ratio of the distance between coordinates X2 to Xi and the distance between coordinates Xi to X3 is r: 1-r (where 0 <r <1). Shall.

  A point B13 indicated by a white circle in the drawing is a point of the coordinate X2 that is interpolated from the points A1 and A3 by linear interpolation (primary interpolation). Similarly, the point B24 is a point of the coordinate X3 that is interpolated from the points A2 and A4 by linear interpolation. Point B23 is a point of coordinates Xi interpolated from points A2 and A3 by linear interpolation. Further, the distance between the points A2 and B13 (difference between the Y coordinates of both points) is Δα, and the distance between the points A3 and B24 is Δβ. Similarly, the distance between the points B23 and C is Δ.

From these, the coordinates Yi of the point C to be estimated can be expressed by the following equation (11).
Yi = (1−r) × Y2 + r × Y3 + Δ (11)
In the above equation (11), Δ can be expressed by the following equation (12).
Δ = (1−r) × Δα + r × Δβ (12)
Furthermore, Δα and Δβ can be expressed by the following equations (13) and (14).
Δα = Y2− (Y1 + Y3) / 2 (13)
Δβ = Y3− (Y2 + Y4) / 2 (14)

  That is, the point C to be calculated is linearly interpolated using two points (points A1 and A4) that are distant from the point (point B23) that is interpolated by linear interpolation using the two neighboring points (points A2 and A3). Thus, the average value is calculated by weighting the errors (Δα and Δβ) of the points (points B13 and B24) obtained by interpolating the two neighboring points, and the average value (Δ) is added.

  The above equations (11) to (14) are applied to the strength of the edge component calculated by the Sobel filter. The edge components calculated by the 0 °, 45 °, 90 °, and 135 ° Sobel filters (absolute values) shown in FIG. 15 are a, b, c, and d, respectively, and estimated 22.5 ° edges. Let e be the intensity (absolute value) of the component. In this example, since the absolute value of the intensity is handled, the intensity of the edge component at 135 ° can be regarded as the intensity at −45 °.

Therefore, the points A1 to A4 and C shown in FIG.
A1 (135 °, d)
A2 (0 °, a)
A3 (45 °, b)
A4 (90 °, c)
C (22.5 °, e)
In this case, r = 1/2.

By substituting these values into the equations (11) to (14), the following equation (21) is obtained.
e = {3 × (a + b) − (c + d)} / 4 (21)
Similarly, when the strength of the edge component at 67.5 ° is f, the strength of the edge component at 112.5 ° is g, and the strength of the edge component at 157.5 ° is h, the following equation (22) Equation (24) is obtained.
f = {3 × (b + c) − (d + a)} / 4 (22)
g = {3 × (c + d) − (a + b)} / 4 (23)
h = {3 × (d + a) − (b + c)} / 4 (24)

  Therefore, the image processing unit 20 cannot calculate with the Sobel filter based on the edge component intensities a to d calculated with the four Sobel filters shown in FIG. 15 and the equations (21) to (24). It is possible to estimate the strengths e to h of the edge components in other directions.

(C) Edge Direction Determination Processing The image processing unit 20 calculates the intensity of edge components in eight directions using four basic Sobel filters and four extended Sobel filters (first method). Alternatively, the image processing unit 20 calculates the strength of the edge component in each direction using the four Sobel filters of FIGS. 15A to 15D and estimates the strength of the edge component in the other directions (second second). Method). The image processing unit 20 compares the strengths of a plurality of edge components obtained by calculation or estimation, and determines the direction with the strongest strength (the absolute value of the calculated value is the smallest) as the direction of the edge component of the specific region 100. judge. The image processing unit 20 reads an edge-preserving smoothing filter (see FIGS. 12 and 13) corresponding to the determined direction, performs a filtering process on the specific region 100, and smoothes the pixel value of the target pixel.

  However, in the edge component intensity calculation using the Sobel filter, it is difficult to determine the direction of the edge component when, for example, the specific region 100 is an image including a 1-dot texture or a 1-dot wide thin line. FIG. 18 is a graph showing an example of a correspondence between a value (absolute value) calculated by a Sobel filter and an angle. In the graph shown in FIG. 18, the horizontal axis represents the angle [°], and the vertical axis represents the absolute value of the value indicating the intensity of the edge component calculated by the Sobel filter. Further, the example shown in FIG. 18 is a result of performing an operation using a Sobel filter on an image including a 1-dot texture or a 1-dot width thin line.

  As shown in FIG. 18, in the case of an image including a 1-dot texture or a 1-dot width thin line, the intensity of the edge component calculated by the Sobel filter has little change due to the angle. Therefore, the image processing unit 20 selects the maximum value and the minimum value from the absolute value of the calculated edge component intensity and the absolute value of the edge component intensity estimated therefrom, and calculates the difference. When the calculated difference does not exceed the threshold value, the image processing unit 20 regards the specific region 100 as an image having no directionality, and performs a smoothing process using the isotropic smoothing filter illustrated in FIG.

  FIG. 19 is a flowchart showing the procedure of the edge preserving type smoothing process, which is the process performed in step S7 of FIG. In the edge preserving type smoothing process, the image processing unit 20 first reads out a plurality of Sobel filters stored in advance in a memory or the like (step S51). The image processing unit 20 performs a filtering process on the specific region 100 using each Sobel filter, and calculates the strength of the edge component in each direction (Step S52).

  Next, the image processing unit 20 estimates the strengths of the edge components in the other directions based on the calculated strengths of the plurality of edge components (step S53). The image processing unit 20 calculates the difference between the absolute value of the intensity calculated in step S52 and the maximum value and the minimum value of the absolute value of intensity estimated in step S53 (step S54), and the calculated difference exceeds the threshold value. It is determined whether or not (step S55).

  When the difference exceeds the threshold value (S55: YES), the image processing unit 20 uses the intensity calculated in step S52 and the intensity estimated in step S53 to maximize the intensity (the absolute value is minimum). By determining, the direction of the edge component included in the specific region 100 is determined (step S56). The image processing unit 20 reads an edge-preserving smoothing filter corresponding to the determined direction of the edge component (step S57).

  If the difference does not exceed the threshold (S55: NO), the image processing unit 20 determines that the edge component of the specific region 100 is not directional, and is not an edge-preserving smoothing filter but isotropic smoothing. Read out the filter (step S58).

  The image processing unit 20 performs a filtering process on the specific region 100 using the smoothing filter read in step S57 or S58, smoothes the target pixel (step S59), and ends the process.

<4. Application judgment>
As described above, when it is determined in the smoothing determination process that the smoothing process is not performed, the image processing unit 20 performs a smoothing process using an edge-preserving smoothing filter. An edge-preserving smoothing filter is applied to a region containing a certain amount of high-frequency components. For example, in the case of an image having a clear edge or pattern, the image quality can be reduced by performing a filtering process. There is a risk of deterioration. Therefore, the image processing unit 20 determines whether or not to apply the smoothing result by the edge preserving smoothing filter based on one or a plurality of conditions. Hereinafter, three examples of determination conditions will be described.

(1) Edge amount determination For example, the image processing unit 20 calculates the amount (intensity) of an edge component in the specific region 100, and when the calculated edge amount is smaller than a threshold value, smoothing by an edge preserving type smoothing filter It is determined that the processing result is applied. Thereby, since smoothing is not performed when the amount of edges is large, it is possible to prevent an image having a clear edge or pattern from being smoothed to deteriorate the image quality. The edge amount of the specific region 100 can be calculated using, for example, a Laplacian filter.

  FIG. 20 is a schematic diagram for explaining edge amount determination using a Laplacian filter. The image processing unit 20 performs processing using a Laplacian filter 121 stored in advance. In the illustrated example, the Laplacian filter 121 is a 3 × 3 matrix, the central value is set to 8, and all other values around it are set to −1.

  The image processing unit 20 extracts 3 × 3 partial regions 100a to 100i from the specific region 100 that is a 5 × 5 matrix. From the 5 × 5 matrix, 9 types of 3 × 3 partial regions 100a to 100i can be extracted. The image processing unit 20 performs a filtering process using the Laplacian filter 121 on each of the extracted partial regions 100a to 100i, and calculates nine values A to I as processing results. The image processing unit 20 calculates the sum of absolute values of the values A to I as the evaluation value S (that is, S = | A | + | B | + ... + | I |).

  The calculated evaluation value S represents the amount of edge components in the specific area 100. The image processing unit 20 determines whether or not the calculated evaluation value S exceeds a threshold value. When the evaluation value S does not exceed the threshold value, the image processing unit 20 determines to apply the result of the smoothing process using the edge preserving type smoothing filter. Further, when the evaluation value S exceeds the threshold value, the image processing unit 20 determines that the result of the smoothing process using the edge preserving type smoothing filter is not applied.

(2) Texture determination In addition, in the case of a fine texture image in units of one pixel, if the smoothing process is performed, the texture may be crushed and the image quality may be deteriorated. Therefore, for example, the image processing unit 20 determines whether or not the image is a texture image based on the change characteristics of the pixel values in the specific region 100, and determines that the smoothing processing result is applied when the texture image is not a texture image. In the case of an image, it can be configured to determine that the smoothing processing result is not applied.

  For example, the image processing unit 20 calculates the frequency of the pixels in the specific region 100 (the number of increase / decrease of the pixel value), and applies a smoothing processing result using an edge-preserving smoothing filter based on the frequency. It is determined whether or not. 21 and 22 are schematic diagrams for explaining application determination based on the frequency. As illustrated in FIGS. 21A to 22D, the image processing unit 20 calculates the vibration frequency in the four directions of the specific region 100 in the horizontal direction, the vertical direction, the oblique 45 ° direction, and the oblique 135 ° direction.

  As illustrated in FIG. 21A, the image processing unit 20 calculates a difference between pixel values of pixels adjacent in the horizontal direction in the 5 × 5 specific area 100 and generates a 4 × 5 matrix using the calculated difference values. To do. At this time, the image processing unit 20 sets the value of the corresponding matrix to 0 when the calculated difference value is equal to or less than the threshold value.

  Next, the image processing unit 20 examines the sign (plus, minus, or value 0) of adjacent values in the horizontal direction of the 4 × 5 matrix, and points each set of adjacent values. When the signs of adjacent values are different, or when one of the adjacent values is 0 and the other is not 0, the image processing unit 20 gives one point to this set. The image processing unit 20 obtains 3 × 5 = 15 points by performing such pointing for all sets of values adjacent in the horizontal direction in a 4 × 5 matrix. The image processing unit 20 calculates the sum of these 15 points, and sets a value that is three times the sum as the frequency A in the horizontal direction. Note that the frequency A is obtained by multiplying the sum by three because the number of points obtained by the calculation in the oblique 45 ° and 135 ° directions described later is as small as 9, so that the number of points is less than the calculated frequency. This is because normalization is performed with weights corresponding to the numbers.

  Similarly, as illustrated in FIG. 21B, the image processing unit 20 generates a 5 × 4 matrix based on the difference between pixel values of pixels adjacent in the vertical direction in the specific region 100 (the difference value is equal to or less than a threshold value). The value of the matrix is 0). The image processing unit 20 assigns points based on codes of adjacent values in the vertical direction of the 5 × 4 matrix to obtain 5 × 3 = 15 points. The image processing unit 20 calculates the sum of these 15 points, and sets the value of three times the sum as the frequency B in the vertical direction.

  As illustrated in FIG. 22C, the image processing unit 20 generates a 4 × 4 matrix based on a difference between pixel values of pixels adjacent in the specific region 100 in an oblique 45 ° direction. (If the difference value is less than or equal to the threshold value, the matrix value is 0). The image processing unit 20 assigns points based on the signs of adjacent values in the 45 × oblique direction of the 4 × 4 matrix to obtain 3 × 3 = 9 points. The image processing unit 20 calculates the sum of these nine points, and sets a value five times the sum as the frequency C in the oblique 45 ° direction.

  As illustrated in FIG. 22D, the image processing unit 20 generates a 4 × 4 matrix based on a difference between pixel values of pixels adjacent in the specific region 100 in an oblique 135 ° direction. (If the difference value is less than or equal to the threshold value, the matrix value is 0). The image processing unit 20 assigns points based on the signs of adjacent values in the oblique 135 ° direction of the 4 × 4 matrix to obtain 3 × 3 = 9 points. The image processing unit 20 calculates the sum of these nine points, and sets a value five times the sum as the frequency C in the oblique 135 ° direction.

  The image processing unit 20 selects a minimum value from the calculated four vibration frequencies A to D, and determines whether or not the minimum value exceeds a threshold value. When the minimum value of the frequencies A to D does not exceed the threshold value, the image processing unit 20 determines to apply the result of the smoothing process using the edge-preserving smoothing filter. Further, when the minimum value of the frequencies A to D exceeds the threshold value, the image processing unit 20 determines that the result of the smoothing process using the edge preserving type smoothing filter is not applied.

(3) Pixel value change amount determination Further, for example, whether the image processing unit 20 applies the smoothing process result according to the change amount of the pixel value of the target pixel by the smoothing process using the edge preserving type smoothing filter. It can be set as the structure which determines whether or not. The image processing unit 20 calculates a difference between the pixel value of the target pixel before the smoothing process (that is, the pixel value of the input image) and the pixel value of the target pixel that has been subjected to the smoothing process, and the difference is a threshold value. If the difference does not exceed the threshold value, it is determined that the smoothing process result is applied, and if the difference exceeds the threshold value, it is determined that the smoothing process result is not applied. As a result, it is possible to prevent the pixel value of the target pixel from being largely changed by the smoothing process.

  The image processing unit 20 determines whether to apply the result of the smoothing process using the edge-preserving smoothing filter based on the determinations (1) to (3). Specifically, the image processing unit 20 applies the result of the smoothing process when it is determined that the result of the edge preserving type smoothing process is applied in all the determinations (1) to (3). If it is determined in any of the determinations (1) to (3) that the result of the edge-preserving smoothing process is not applied, the image processing unit 20 does not apply the result of the smoothing process. The image processing unit 20 may be configured to perform one or two determinations of the three conditions instead of performing all the determinations of the three conditions (1) to (3). The image processing unit 20 may be configured to perform application determination based on conditions different from the above (1) to (3).

  Further, in the present embodiment, for example, as shown in the flowchart of FIG. 3, the image processing unit 20 performs smoothing processing using an edge preserving smoothing filter and then determines whether or not to apply the smoothing result. However, the present invention is not limited to this. When the application determination with reference to the smoothing result as in the application determination condition (3) is not performed, the image processing unit 20 performs the application determination first and performs smoothing by the edge preserving smoothing filter according to the determination result. It is good also as a structure which performs a process.

  23 and 24 are flowcharts showing the procedure of the application determination process, which is the process performed in step S8 of FIG. First, the image processing unit 20 performs edge amount determination processing. The image processing unit 20 extracts one 3 × 3 partial regions 100a to 100i from the 5 × 5 specific region 100 (step S71), and performs filtering processing by the Laplacian filter 121 on the extracted partial regions (step S71). S72). The image processing unit 20 determines whether or not the extraction of all the specific areas 100a to 100i and the filtering process by the Laplacian filter 121 are finished. When these processes are not performed on all the specific areas 100a to 100i (S73: NO), the image processing unit 20 returns the process to step S71, extracts other partial areas 100a to 100i, and the Laplacian filter 121. Perform the filtering process.

  When the extraction and filtering processing of the specific areas 100a to 100i are finished for all the specific areas 100a to 100i (S73: YES), the image processing unit 20 calculates the sum of absolute values of the plurality of values calculated in the filtering process. Calculation is made (step S74), and this sum is taken as an evaluation value S. The image processing unit 20 determines whether or not the evaluation value S is less than the threshold value (step S75). When the evaluation value S is equal to or greater than the threshold value (S75: NO), the image processing unit 20 determines not to apply the result of the edge preserving smoothing process (step S85), and ends the application determination process.

  When the evaluation value S is less than the threshold (S75: YES), the image processing unit 20 performs texture determination processing. The image processing unit 20 calculates a pixel value difference between adjacent pixels in one direction of the specific region 100 (step S76). The image processing unit 20 assigns points based on the sign of the difference value adjacent in one direction to the matrix constituted by the calculated differences (step S77). The image processing unit 20 calculates the sum of a plurality of points, performs normalization by multiplying the calculated sum by a predetermined amount (step S78), and sets the normalized value as the frequency in this direction. Thereafter, the image processing unit 20 determines whether or not the processing in steps S76 to S78 has been completed for all directions to be processed (step S79). If the process has not been completed for all directions (S79: NO), the image processing unit 20 returns the process to step S76, and performs the same process for the other directions.

  When the process has been completed for all directions (S79: YES), the image processing unit 20 selects the minimum value from the frequencies calculated for each direction (step S80), and whether or not the minimum value of the frequencies is less than the threshold value. Is determined (step S81). When the minimum value of the frequency is equal to or greater than the threshold value (S81: NO), the image processing unit 20 determines not to apply the result of the edge preserving smoothing process (step S85), and ends the application determination process.

  When the minimum value of the frequency is less than the threshold value (S81: YES), the image processing unit 20 performs a pixel value change amount determination process. The image processing unit 20 calculates a change amount (difference) for the target pixel value before and after the smoothing process using the edge-preserving smoothing filter (step S82), and whether or not the change amount is less than a threshold value. Is determined (step S83). When the amount of change is less than the threshold (S83: YES), the image processing unit 20 determines to apply the result of the edge preserving smoothing process (step S84), and ends the application determination process. If the amount of change is equal to or greater than the threshold (S83: NO), the image processing unit 20 determines not to apply the result of the edge preserving smoothing process (step S85), and ends the application determination process.

  The display device 1 according to the present embodiment having the above configuration extracts the specific area 100 including the target pixel from the input image, and determines whether or not to perform the smoothing process on the target pixel in the specific area 100. When it determines with performing a smoothing process, the display apparatus 1 performs the filtering process using a smoothing filter, and smoothes an attention pixel. When it is determined that the smoothing process is not performed, the display device 1 performs a filtering process using an edge-preserving smoothing filter to smooth the target pixel. At this time, the display device 1 determines whether or not to apply the smoothing result by the edge preserving smoothing filter according to the pixel value in the specific region 100 or the smoothing result by the edge preserving smoothing filter. To do. As a result, the display device 1 uses the pixel value smoothed by the smoothing filter, the pixel value smoothed by the edge-preserving smoothing filter, or the pixel value that has not been smoothed for the target pixel of the input image. Can be output. The display device 1 can generate and display an output image subjected to smoothing suitable for each pixel by performing these processes for all the pixels of the input image.

  Further, the display device 1 calculates a differential value of the pixel value between the pixels in the specific region 100, and determines whether or not to perform the smoothing process on the target pixel in the specific region 100 based on the calculated differential value. Further, when it is determined that the smoothing process is to be performed, the display device 1 determines the direction and position of the noise boundary in the specific region 100. As a result, smoothing suitable for the direction and position of the noise boundary existing in the specific region 100 can be performed, so that noise can be accurately removed or reduced from the input image.

  Further, the display device 1 enlarges the size of the specific region 100 when the direction of the noise boundary in the specific region 100 is not a horizontal pattern, a vertical pattern, or other patterns (that is, when no noise boundary exists in the specific region 100). The determination is repeated until the noise boundary of the pattern is detected or the specific area 100 is enlarged to a predetermined size. Thereby, the display apparatus 1 can detect and remove or reduce block noise of various sizes with high accuracy.

  Further, the display device 1 calculates a primary differential value and a secondary differential value of pixel values between adjacent pixels in the specific region 100, and determines whether or not to perform a smoothing process on the target pixel in the specific region 100. . At this time, the display device 1 generates a matrix in which the primary differential values are binarized by comparison with the threshold value, calculates the logical sum of each row or each column of the matrix, and also compares the secondary differential value with the threshold value. A matrix in which the values are binarized is generated, and a logical sum of each row or each column of the matrix is calculated. The display device 1 further calculates the logical sum of the logical sum related to the calculated primary differential value and the logical sum related to the secondary differential value, and determines whether smoothing processing can be performed based on the calculation result. Accordingly, the display device 1 can accurately determine whether smoothing processing can be performed on the target pixel in the specific region 100.

  The display device 1 calculates the differential value and the logical sum in the vertical direction and the horizontal direction of the specific area 100, and the noise boundary extends in the vertical direction or the horizontal direction based on the logical sum calculated in each direction. Determine whether or not. The display device 1 determines the position of the noise boundary in the specific region 100 based on the arrangement pattern of 0/1 in the matrix of secondary differential values. Accordingly, the display device 1 determines whether the extending direction of the noise boundary included in the specific area 100 is the vertical direction or the horizontal direction (vertical direction or horizontal direction), and the position of the noise boundary in the specific area 100. It can be easily judged.

  The display device 1 stores a plurality of smoothing filters for removing or reducing noise, selects a smoothing filter according to the determined direction and / or position of the noise boundary, and uses the selected smoothing filter. The pixel value of the target pixel in the specific area 100 is smoothed. Thereby, the display apparatus 1 can perform smoothing suitable for the direction and position of the noise boundary in the specific region 100.

  Further, the display device 1 calculates the strength of the edge component included in the specific region 100 in a plurality of directions using a plurality of Sobel filters, and estimates the strength of the edge component in other directions based on the calculated strength. The display device 1 determines the extending direction of the edge component in the specific region 100 based on the calculated and estimated intensity of the edge component, and performs smoothing using an edge preserving type smoothing filter corresponding to the determined direction. Do. Thereby, the display device 1 can estimate the strength of the edge component in a direction that cannot be calculated by the stored Sobel filter, and can determine the direction of the edge component in more directions. Therefore, the display device 1 can select an edge-preserving smoothing filter that is more suitable for the edge component of the specific region 100, and can perform smoothing with high accuracy.

  Further, the display device 1 performs the edge component intensity by the Sobel filter in at least four directions (0 °, 45 °, 90 °, and 135 °), and based on the calculated four intensities, the other direction (22.5 ° , 67.5 °, 112.5 ° and 157.5 °). The display device 1 stores formulas (21) to (24) for estimating the strength of the edge component by linear interpolation, and substitutes the four strengths calculated using the Sobel filter into this formula. Thus, it is possible to calculate an estimated value of the intensity in other directions.

  In addition to the four basic Sobel filters corresponding to 0 °, 45 °, 90 °, and 135 °, the display device 1 has 22.5 °, 67.5 °, 112.5 °, and 157.5 °. Four corresponding extended Sobel filters may be stored, and the strengths in the eight directions may be calculated using the eight Sobel filters. The four extended Sobel filters can be calculated in advance using equations (1) to (4) or equations (5) to (8). Accordingly, the display device 1 can select an edge-preserving smoothing filter that is more suitable for the edge component of the specific region 100, and can perform smoothing with high accuracy.

  In addition, the display device 1 selects the maximum value and the minimum value from the plurality of calculated and estimated edge strengths, and determines whether or not these differences exceed a threshold value. When the difference does not exceed the threshold value, the display device 1 regards the specific region 100 as a substantially flat image that does not include an edge component, and performs smoothing using an isotropic smoothing filter that is not an edge-preserving type. This makes it possible to appropriately smooth an image that does not include an edge component.

  Further, the display device 1 calculates the strength of the edge component for the specific region 100 using a Laplacian filter, and determines whether the calculated strength exceeds a threshold value. When the calculated intensity does not exceed the threshold, the display device 1 does not apply the result of the smoothing process using the edge preserving type smoothing filter. Thereby, for example, when the input image is an image having a clear edge or pattern, it is possible to prevent the image quality from being deteriorated by the smoothing process.

  Further, the display device 1 calculates the increase / decrease frequency of the pixel values of adjacent pixels in the specific region 100, that is, the vibration frequency, and determines whether the calculated vibration frequency exceeds a threshold value. When the calculated frequency exceeds the threshold value, the display device 1 does not apply the result of the smoothing process using the edge preserving type smoothing filter. Thereby, for example, when the input image is an image including a fine texture such as one pixel unit, it is possible to prevent image quality deterioration such as the texture becoming unclear due to the smoothing process.

  The display device 1 calculates the amount of change in the pixel value of the pixel of interest before and after the smoothing process using the edge-preserving smoothing filter, and determines whether the calculated amount of change exceeds the threshold value. When the change amount exceeds the threshold value, the display device 1 does not apply the result of the smoothing process using the edge preserving type smoothing filter. Thereby, for example, when it is difficult to determine whether the pixel of interest is texture or noise, for example, only the pixel of interest within the specific region 100 is greatly different from other pixels, the influence of smoothing Can be reduced.

  In the present embodiment, the display device 1 is configured to perform smoothing using an edge-preserving smoothing filter when it is determined that smoothing processing is not performed on the target pixel in the specific region 100. However, the present invention is not limited to this, and the display device 1 may be configured not to perform smoothing using an edge-preserving smoothing filter. In addition, the display device 1 is configured to extract a 5 × 5 pixel area as the specific area 100, but is not limited to this. For example, the display device 1 has other sizes such as 3 × 3 pixels or 7 × 7 pixels. An area may be extracted as the specific area 100. Moreover, the smoothing filter shown in FIG.10 and FIG.11 is an example, Comprising: It does not restrict to this. Similarly, the edge-preserving smoothing filter shown in FIGS. 12 and 13 is an example, and the present invention is not limited to this. The isotropic smoothing filter shown in FIG. 14 is an example, and the present invention is not limited to this. The Sobel filter shown in FIG. 15 and the Laplacian filter shown in FIG. 20 are examples, and the present invention is not limited to this.

  The display device 1 calculates the intensity of edge components of 0 °, 45 °, 90 °, and 135 ° using a Sobel filter, and 22.5 °, 67.5 °, 112.5 °, and 157. The intensity of the 5 ° edge component is calculated or estimated, but these directions (angles) are merely examples, and the present invention is not limited to this. The display device 1 stores eight sobel filters and stores the first method for calculating the intensity of edge components in eight directions, or stores four sobel filters, and other four directions. Any of the second methods for estimating the intensity of the edge component may be used. Furthermore, the first and second methods may be combined. For example, it is possible to calculate the strength of the edge component using a six-direction Sobel filter and estimate the strength of the remaining two-direction edge component. Further, for example, the configuration is such that the strength of the edge component in the eight directions is calculated by the eight-direction Sobel filter, the strength of the edge component in the other eight directions is estimated, and the strength of the edge component in the sixteen directions is obtained can do.

  Further, the display device 1 is configured to determine whether or not to apply the result of the smoothing process using the edge-preserving smoothing filter under the three conditions of edge amount determination, texture determination, and pixel value change amount determination. However, it is not limited to this. The display device 1 may perform application determination based on, for example, one or two of the above three conditions, or may perform application determination based on conditions other than the above three conditions, for example.

  The display device 1 is configured to perform two types of smoothing processing on the input image, the first smoothing processing using the smoothing filter or the second smoothing processing using the edge preserving type smoothing filter. However, it is not limited to this. The display device 1 may have a configuration in which the first smoothing process is performed and the second smoothing process is not performed. In this case, for example, in the flowchart of FIG. 3, when it is determined in step S4 that smoothing is not performed, the display device 1 may be configured to perform the process of outputting the pixel value of the target pixel in step S11.

  The display device 1 also performs noise boundary direction determination processing (step S24) and position determination processing (step S29) on the specific region 100 extracted from the input image, and smoothes based on the determined direction and position. It may be configured to perform image processing other than processing.

  The display device 1 may be configured to perform the second smoothing process without performing the first smoothing process. In this case, for example, in the flowchart of FIG. 3, after the specific region 100 is extracted in step S <b> 2, the process proceeds to step S <b> 7 to perform the smoothing process using the edge-preserving smoothing filter. Furthermore, the display device 1 may be configured not to perform application determination as to whether or not to apply the result of the edge preserving smoothing filter process. In this case, the configuration may be such that the result of the edge preserving smoothing filter process is always applied.

  The display device 1 also performs edge component direction determination processing (step S52), edge strength estimation processing (step S53), and the like on the specific region 100 extracted from the input image. Step S56) may be performed, and image processing other than smoothing processing may be performed based on the determined direction.

  In the present embodiment, the display device 1 has been described as an example of a device that removes or reduces noise included in an input image. However, the present invention is not limited to this, and the same applies to other various image processing devices. Configuration can be applied. For example, the same configuration can be applied to a display device that displays not an input image from the PC 5 but an image related to a television broadcast received by a tuner or the like. A similar configuration can be applied to an image processing apparatus that prints an input image, such as a printer or a facsimile.

(Modification)
Further, for example, the PC may perform processing such as image expansion and noise removal described above, and an image from which noise is removed or reduced may be input to the display device. In this case, the above-described image processing function may be provided as a computer program, and the above-described image processing may be performed by a PC CPU executing the computer program. The computer program may be provided by being recorded on a recording medium such as a disk or a memory card, or may be provided via a network such as the Internet.

  FIG. 25 is a block diagram illustrating a configuration of the PC 155 according to the modification. The PC 155 according to the modified example decompresses an input image compressed by a method such as JPEG or MPEG, performs image processing for removing or reducing noise, and outputs the processed image to the display device 151. The display device 151 displays an input image from the PC 155 on the liquid crystal panel. The PC 155 includes a CPU 156, an operation unit 157, a primary storage unit 158, a secondary storage unit 159, a recording medium mounting unit 160, a communication unit 161, an image output unit 162, and the like.

The CPU 156 reads and executes the image processing program 181 stored in the secondary storage unit 159, whereby the above-described image expansion unit 17, the image processing unit 20, and the like are realized as software function blocks. The operation unit 157 is an input device such as a keyboard and a mouse. The operation unit 157 receives a user operation and notifies the CPU 156 of the operation. The primary storage unit 158 includes a memory element such as an SRAM (Static Random Access Memory), and temporarily stores various data used for processing of the CPU 156. The secondary storage unit 159 is configured by a storage device such as a hard disk, and stores various computer programs such as the image processing program 181 and various data required for executing the computer program. The recording medium mounting unit 160 is a device such as a disk drive or a memory card slot, and is loaded with a recording medium 180 such as a DVD (Digital Versatile Disk) or a memory card, and reads computer programs and data recorded on the recording medium. . The communication unit 161 communicates with other devices via a network such as the Internet by wireless or wired. The image output unit 162 outputs data subjected to image processing by the CPU 156 to the display device 151.

  In the illustrated example, the image processing program 181 is provided by being recorded on the recording medium 180. When the recording medium 180 is loaded in the recording medium loading unit 160, the CPU 156 of the PC 155 reads the image processing program 181 and the like recorded on the recording medium 180 and stores them in the secondary storage unit 159. The CPU 156 operates as the image expansion unit 17 and the image processing unit 20 by reading out and executing the image processing program 181 from the secondary storage unit 159. For example, when the operation unit 157 receives an instruction to display an image compressed by a method such as JPEG or MPEG, the CPU 156 obtains an image to be displayed by, for example, reading it from the secondary storage unit 159, and decompresses the image. The unit 17 decompresses the image.

  Next, the CPU 156 performs image processing for removing or reducing noise included in the decompressed image in the image processing unit 20. The image processing unit 20 extracts the specific area 100 including the target pixel from the expanded image, determines whether to perform the smoothing process on the target pixel in the extracted specific area 100, and determines to perform the smoothing process. In this case, a smoothing process using a smoothing filter is performed. Further, when it is determined that the smoothing process is not performed, the image processing unit 20 performs a smoothing process using an edge-preserving smoothing filter and determines whether to apply this process. The image processing unit 20 repeats these processes for all the pixels of the image, and the pixel value of each pixel is smoothed by the smoothing filter, the pixel value smoothed by the edge-preserving smoothing filter, or Then, an image having one of the original pixel values is generated and output. Accordingly, the image processing unit 20 can generate an output image from which stepped noise such as block noise has been removed or reduced.

1 Display device (image processing device)
5 PC
DESCRIPTION OF SYMBOLS 11 Control part 12 Operation part 13 Liquid crystal panel 14 Backlight 15 Light drive part 16 Image input part 17 Image expansion part 18 Panel drive part 20 Image processing part (specific area extraction means, Sobel filter storage means, edge strength calculation means, edge Strength estimation means, edge direction determination means, edge preservation type smoothing filter storage means, edge strength difference determination means, second edge strength calculation means, edge strength determination means, increase / decrease count calculation means, increase / decrease count determination means, smooth difference calculation Means, smooth difference judging means)
21 Specific area extraction unit (specific area extraction means)
22 smoothing determination unit 23 first smoothing unit 24 second smoothing unit (smoothing means)
25 Application determination unit (application determination means)
26 pixel value selection unit 100 specific area 101 horizontal primary differential value matrix 102 horizontal primary differential value OR sequence 103 horizontal secondary differential value matrix 103a binarized horizontal secondary differential value matrix 104 horizontal secondary differential value OR sequence 105 horizontal OR column 106 Horizontal upper OR value 107 Horizontal lower OR value 108 Horizontal direction determination result 111 Vertical primary differential value matrix 112 Vertical primary differential value OR row 113 Vertical secondary differential value matrix 114 Vertical secondary differential value OR row 115 Vertical OR Row 116 Vertical upper OR value 117 Vertical lower OR value 118 Vertical direction determination result 151 Display device 155 PC (image processing device)
156 CPU
157 Operation unit 158 Primary storage unit 159 Secondary storage unit 160 Recording medium mounting unit 161 Communication unit 162 Image output unit 180 Recording medium 181 Image processing program (computer program)

Claims (24)

A specific area extracting means for extracting a specific area including a target pixel from an input image composed of a plurality of pixels arranged in a matrix;
A Sobel filter storage means for storing a plurality of Sobel filters each detecting the intensity of an edge component with respect to a specific direction in different directions;
Edges for performing filtering using a plurality of Sobel filters stored in the Sobel filter storage unit on the specific region extracted by the specific region extraction unit, and calculating the strength of edge components included in the specific region in a plurality of directions Intensity calculation means;
Edge strength estimating means for estimating the strength of an edge component in one or a plurality of directions different from the plurality of directions based on the plurality of strengths calculated by the edge strength calculating means;
Edge direction determining means for determining a direction in which an edge component included in the specific region extends based on the strength calculated by the edge strength calculating means and the strength estimated by the edge strength estimating means. An image processing apparatus. The edge strength calculating means is configured to calculate the strength of the edge component in at least four directions,
The edge strength estimating means includes
For the strength interpolated by linear interpolation based on the two strengths calculated by the edge strength calculation means for two directions in the vicinity of the direction to be estimated,
Based on the intensity in the four directions calculated by the edge intensity calculation means, the intensity interpolated by linear interpolation for the two directions in the vicinity, and the intensity in the two directions in the vicinity calculated by the edge intensity calculation means A value obtained by adding an average value weighted according to the difference between the direction in which each error should be estimated and the two directions in the vicinity,
The image processing apparatus according to claim 1, wherein the image processing apparatus is calculated as an estimated value of the intensity of an edge component in the direction to be estimated. The Sobel filter storage means includes
When the angle in the horizontal direction of the input image is 0 [rad], basic Sobel filters for four directions corresponding to angles 0, π / 4, π / 2, and 3π / 4,
The basic Sobel filter corresponding to angle 0 is A, the basic Sobel filter corresponding to angle π / 4 is B, the basic Sobel filter corresponding to angle π / 2 is C, and the angle corresponds to 3π / 4. When the basic Sobel filter is D and the Sobel filter corresponding to the angle π is E (= −A), the angle x given by the following equations (1) to (4) (where 0 <x < one or a plurality of extended Sobel filters X with respect to the direction of π) When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)
The image processing apparatus according to claim 1, wherein: A specific area extracting means for extracting a specific area including a target pixel from an input image composed of a plurality of pixels arranged in a matrix;
A Sobel filter storage means for storing a plurality of Sobel filters each detecting the intensity of an edge component with respect to a specific direction in different directions;
Edges for performing filtering using a plurality of Sobel filters stored in the Sobel filter storage unit on the specific region extracted by the specific region extraction unit, and calculating the strength of edge components included in the specific region in a plurality of directions Intensity calculation means;
Edge direction determining means for determining a direction in which an edge component included in the specific region extends based on the strength calculated by the edge strength calculating means,
The Sobel filter storage means includes
When the angle in the horizontal direction of the input image is 0 [rad], basic Sobel filters for four directions corresponding to angles 0, π / 4, π / 2, and 3π / 4,
The basic Sobel filter corresponding to angle 0 is A, the basic Sobel filter corresponding to angle π / 4 is B, the basic Sobel filter corresponding to angle π / 2 is C, and the angle corresponds to 3π / 4. When the basic Sobel filter is D and the Sobel filter corresponding to the angle π is E (= −A), the angle x given by the following equations (1) to (4) (where 0 <x < one or a plurality of extended Sobel filters X with respect to the direction of π) When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)
An image processing apparatus. The plurality of Sobel filters stored in the Sobel filter storage means is a 3 × 3 matrix, and when the horizontal direction of the input image is 0 [rad], the Sobel filter relating to the direction of the angle x is as follows ( 5) to be given by equations (8)

The image processing apparatus according to claim 3, wherein the image processing apparatus is an image processing apparatus. Edge-preserving smoothing filter storage means for storing a plurality of edge-preserving smoothing filters each storing edge components in a specific direction and smoothing pixel values;
Smoothing that smoothes the pixel value of the pixel of interest in the specific region by selecting one edge preserving type smoothing filter from the edge preserving type smoothing filter storage unit according to the direction determined by the edge direction determining unit The image processing apparatus according to claim 1, further comprising: an image forming unit. Edge strength difference determination means for determining whether or not the difference between the maximum value and the minimum value of the plurality of edge strengths calculated by the edge strength calculation means exceeds a threshold;
When the edge strength difference determination unit determines that the difference does not exceed the threshold, the smoothing unit smoothes the pixel value of the target pixel in the specific region using a smoothing filter that is not an edge-preserving type. The image processing apparatus according to claim 6, wherein: Further comprising application determining means for determining whether or not to apply the smoothed result by the smoothing means;
An output image in which a pixel value of each pixel of the input image is either a pixel value smoothed by the smoothing unit or an original pixel value that has not been smoothed is generated. The image processing apparatus according to claim 6 or 7. Second edge strength calculating means for performing filter processing using a Laplacian filter on the specific area extracted by the specific area extracting means, and calculating the strength of an edge component included in the specific area;
Edge strength determination means for determining whether or not the strength calculated by the second edge strength calculation means exceeds a threshold;
The application determining means includes
When the edge strength determination means determines that the intensity exceeds the threshold, it is determined to apply the smoothing result by the smoothing means,
The image processing according to claim 8, wherein when the edge strength determination unit determines that the intensity does not exceed the threshold value, it is determined that the smoothing result by the smoothing unit is not applied. apparatus. Increase / decrease number calculating means for calculating the increase / decrease number of pixel values between pixels adjacent in a specific direction in the specific area extracted by the specific area extracting means;
An increase / decrease number determination means for determining whether the increase / decrease number calculated by the increase / decrease number calculation means exceeds a threshold,
The application determining means includes
If the increase / decrease count determination means determines that the increase / decrease count does not exceed the threshold, it is determined to apply the smoothing result by the smoothing means;
10. The method according to claim 8 or 9, wherein when the increase / decrease count exceeds the threshold, the increase / decrease count determination means determines that the smoothing result by the smoothing means is not applied. The image processing apparatus described. Smoothing difference calculating means for calculating a difference between a pixel value of a target pixel included in the specific area extracted by the specific area extracting means and a pixel value smoothed by the smoothing means;
Smooth difference determination means for determining whether or not the difference calculated by the smooth difference calculation means exceeds a threshold, and
The application determining means includes
When the smoothing difference determining means determines that the difference does not exceed the threshold, it is determined to apply the smoothing result by the smoothing means;
11. The method according to claim 8, wherein when the smoothing difference determining unit determines that the difference exceeds the threshold value, the smoothing result by the smoothing unit is determined not to be applied. The image processing apparatus according to claim 1. A specific area extracting step for extracting a specific area including a target pixel from an input image composed of a plurality of pixels arranged in a matrix;
A plurality of Sobel filters for detecting the intensity of the edge component in each specific direction are stored in different directions, and the specific region extracted in the specific region extraction step is subjected to filter processing by the stored plurality of Sobel filters. Performing an edge strength calculation step for calculating the strength of edge components included in the specific region in a plurality of directions;
Based on a plurality of intensities calculated in the edge intensity calculating step, an edge intensity estimating step for estimating the intensity of an edge component in one or a plurality of directions different from the plurality of directions;
An edge direction determining step for determining a direction in which an edge component included in the specific region extends based on the strength calculated in the edge strength calculating step and the strength estimated in the edge strength estimating step. A featured image processing method. In the edge strength calculating step, the strength of the edge component is calculated in at least four directions,
In the edge strength estimation step,
For the strength interpolated by linear interpolation based on the two strengths calculated by the edge strength calculation means for two directions in the vicinity of the direction to be estimated,
Based on the intensity in the four directions calculated by the edge intensity calculation means, the intensity interpolated by linear interpolation for the two directions in the vicinity, and the intensity in the two directions in the vicinity calculated by the edge intensity calculation means A value obtained by adding an average value weighted according to the difference between the direction in which each error should be estimated and the two directions in the vicinity,
The image processing method according to claim 12, wherein the image processing method is calculated as an estimated value of the intensity of the edge component in the direction to be estimated. As a Sobel filter used for filter processing in the edge strength calculation step,
When the angle in the horizontal direction of the input image is 0 [rad], basic Sobel filters for four directions corresponding to angles 0, π / 4, π / 2, and 3π / 4,
The basic Sobel filter corresponding to angle 0 is A, the basic Sobel filter corresponding to angle π / 4 is B, the basic Sobel filter corresponding to angle π / 2 is C, and the angle corresponds to 3π / 4. When the basic Sobel filter is D and the Sobel filter corresponding to the angle π is E (= −A), the angle x given by the following equations (1) to (4) (where 0 <x < one or a plurality of extended Sobel filters X with respect to the direction of π) When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)
The image processing method according to claim 12 or 13, wherein: A specific area extracting step for extracting a specific area including a target pixel from an input image composed of a plurality of pixels arranged in a matrix;
A plurality of Sobel filters for detecting the intensity of the edge component in each specific direction are stored in different directions, and the specific region extracted in the specific region extraction step is subjected to filter processing by the stored plurality of Sobel filters. Performing an edge strength calculation step for calculating the strength of edge components included in the specific region in a plurality of directions;
An edge direction determining step for determining a direction in which an edge component included in the specific region extends based on the strength calculated in the edge strength calculating step;
As a Sobel filter used for filter processing in the edge strength calculation step,
When the angle in the horizontal direction of the input image is 0 [rad], basic Sobel filters for four directions corresponding to angles 0, π / 4, π / 2, and 3π / 4,
The basic Sobel filter corresponding to angle 0 is A, the basic Sobel filter corresponding to angle π / 4 is B, the basic Sobel filter corresponding to angle π / 2 is C, and the angle corresponds to 3π / 4. When the basic Sobel filter is D and the Sobel filter corresponding to the angle π is E (= −A), the angle x given by the following equations (1) to (4) (where 0 <x < one or a plurality of extended Sobel filters X with respect to the direction of π) When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)
An image processing method characterized by the above. The plurality of Sobel filters to be stored are 3 × 3 matrices, and when the horizontal direction of the input image is 0 [rad], the Sobel filters for the direction of the angle x are the following (5) to (8 ) Given by the formula

The image processing method according to claim 14 or 15, wherein: A plurality of edge-preserving smoothing filters each storing edge components in a specific direction and smoothing pixel values are stored for different directions, and stored according to the direction determined in the edge direction determination step. The method further comprises a smoothing step of selecting one edge preserving type smoothing filter from a plurality of edge preserving type smoothing filters and smoothing a pixel value of a pixel of interest in the specific region. The image processing method according to claim 16. An edge strength difference determination step for determining whether or not the difference between the maximum value and the minimum value of the plurality of edge strengths calculated in the edge strength calculation step exceeds a threshold;
In the smoothing step, when the edge strength difference determining step determines that the difference does not exceed the threshold value, the pixel value of the target pixel in the specific region is smoothed using a smoothing filter that is not an edge-preserving type. The image processing method according to claim 17, wherein: An application determining step of determining whether or not to apply the smoothed result of the smoothing step;
The output image in which the pixel value of each pixel of the input image is either the pixel value smoothed in the smoothing step or the original pixel value that has not been smoothed is generated. The image processing method according to claim 18. A second edge intensity calculating step of performing filter processing by a Laplacian filter on the specific area extracted in the specific area extracting step, and calculating the intensity of an edge component included in the specific area;
An edge strength determination step for determining whether or not the strength calculated in the second edge strength calculation step exceeds a threshold value,
In the application determining step,
If it is determined in the edge strength determination step that the intensity exceeds the threshold, it is determined to apply the smoothing result from the smoothing step,
20. The image processing method according to claim 19, wherein, when it is determined in the edge strength determination step that the intensity does not exceed the threshold value, it is determined that the smoothing result of the smoothing step is not applied. An increase / decrease number calculation step for calculating the increase / decrease number of pixel values between pixels adjacent in a specific direction within the specific region extracted in the specific region extraction step;
An increase / decrease count determination step for determining whether or not the increase / decrease count calculated in the increase / decrease count calculation step exceeds a threshold,
In the application determining step,
If it is determined in the increase / decrease count determination step that the increase / decrease count does not exceed the threshold, it is determined that the smoothing result by the smoothing means is applied,
21. The image according to claim 19 or 20, wherein when the increase / decrease count exceeds the threshold value, it is determined that the smoothing result by the smoothing unit is not applied when it is determined in the increase / decrease count determination step. Processing method. A smoothing difference calculating step of calculating a difference between a pixel value of a target pixel included in the specific region extracted in the specific region extracting step and a pixel value smoothed by the smoothing step;
A smoothing difference determining step for determining whether or not the difference calculated in the smoothing difference calculating step exceeds a threshold value,
In the application determining step,
If it is determined in the smoothing difference determination step that the difference does not exceed the threshold, it is determined that the smoothing result by the smoothing means is applied,
22. The method according to claim 19, wherein, when the difference exceeds the threshold value, it is determined that the smoothing result by the smoothing unit is not applied when the smoothing difference determination step determines that the difference exceeds the threshold value. An image processing method described in 1. Computer
A specific area extracting means for extracting a specific area including a target pixel from an input image composed of a plurality of pixels arranged in a matrix;
A Sobel filter storage means for storing a plurality of Sobel filters each detecting the intensity of an edge component with respect to a specific direction in different directions;
Edges for performing filtering using a plurality of Sobel filters stored in the Sobel filter storage unit on the specific region extracted by the specific region extraction unit, and calculating the strength of edge components included in the specific region in a plurality of directions Intensity calculation means;
Edge strength estimating means for estimating the strength of an edge component in one or a plurality of directions different from the plurality of directions based on the plurality of strengths calculated by the edge strength calculating means;
Based on the strength calculated by the edge strength calculation means and the strength estimated by the edge strength estimation means, the device is operated as an edge direction determination means for determining a direction in which an edge component included in the specific region extends. Computer program. Computer
A specific area extracting means for extracting a specific area including a target pixel from an input image composed of a plurality of pixels arranged in a matrix;
A Sobel filter storage means for storing a plurality of Sobel filters each detecting the intensity of an edge component with respect to a specific direction in different directions;
Edges for performing filtering using a plurality of Sobel filters stored in the Sobel filter storage unit on the specific region extracted by the specific region extraction unit, and calculating the strength of edge components included in the specific region in a plurality of directions Intensity calculation means;
Based on the strength calculated by the edge strength calculation means, it operates as an edge direction determination means for determining the direction in which the edge component included in the specific region extends,
The Sobel filter storage means includes
When the angle in the horizontal direction of the input image is 0 [rad], basic Sobel filters for four directions corresponding to angles 0, π / 4, π / 2, and 3π / 4,
The basic Sobel filter corresponding to angle 0 is A, the basic Sobel filter corresponding to angle π / 4 is B, the basic Sobel filter corresponding to angle π / 2 is C, and the angle corresponds to 3π / 4. When the basic Sobel filter is D and the Sobel filter corresponding to the angle π is E (= −A), the angle x given by the following equations (1) to (4) (where 0 <x < one or a plurality of extended Sobel filters X with respect to the direction of π) When 0 <x <π / 4: X = αA + (1−α) B (1)
When π / 4 <x <π / 2: X = αB + (1-α) C (2)
When π / 2 <x <3π / 4: X = αC + (1-α) D (3)
In the case of 3π / 4 <x <π: X = αD + (1−α) E (4)
(However, α is a coefficient determined according to the angle x, and 0 <α <1.)
A computer program characterized by the above.

Images