US8238683B2 - Image processing method - Google Patents

Image processing method Download PDF

Info

Publication number
US8238683B2
US8238683B2 US12/230,161 US23016108A US8238683B2 US 8238683 B2 US8238683 B2 US 8238683B2 US 23016108 A US23016108 A US 23016108A US 8238683 B2 US8238683 B2 US 8238683B2
Authority
US
United States
Prior art keywords
noise
image
frequency
images
low
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US12/230,161
Other languages
English (en)
Other versions
US20090046943A1 (en
Inventor
Kenichi Ishiga
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nikon Corp
Original Assignee
Nikon Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nikon Corp filed Critical Nikon Corp
Assigned to NIKON CORPORATION reassignment NIKON CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ISHIGA, KENICHI
Publication of US20090046943A1 publication Critical patent/US20090046943A1/en
Application granted granted Critical
Publication of US8238683B2 publication Critical patent/US8238683B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/40—Picture signal circuits
    • H04N1/409—Edge or detail enhancement; Noise or error suppression
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/10—Image enhancement or restoration using non-spatial domain filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing

Definitions

  • the present invention relates to an image processing method that may be adopted when removing noise from an image or emphasizing edges in the image.
  • Patent reference 1 discloses a method for acquiring a noise-free image by spatially filtering for noise removal, the high-frequency subband coefficients in the multiple-resolution images resulting from the transformation and then inverse-transforming the image.
  • Patent reference 2 discloses a contrasting method through which noise is sequentially removed from reduced images on the low-frequency subband side, which are generated on a temporary basis during the process of the multiple resolution transformation.
  • Patent reference 3 discloses a method through which, instead of removing noise directly on the transformation coefficient planes of the multiple-resolution images resulting from the transformation as described above, noise is removed by extracting noise components contained in the high-frequency subband transformation coefficients corresponding to LH, HL and HH resulting from the multiple resolution transformation via orthogonal wavelets, combining only the noise components through inverse wavelet transformation and subtracting the synthesized noise component from the original image, so as to assure better ease in the handling of the noise components.
  • This method allows the extent of noise removal to be adjusted easily by applying a constant multiple factor dependent upon the image structure in the real space to the synthesized noise component prior to the final subtraction processing.
  • Patent reference 3 also presents an application example in which a similar technology is adopted in edge emphasis, whereby edge components, as well as the noise components, are extracted from the subband coefficients resulting from the multiple resolution transformation, the extracted edge components are combined, a constant multiple factor dependent upon the image structure is applied to the synthesized edge component in the real spatial plane and then the product is added to the original image.
  • Patent reference 1 U.S. Pat. No. 5,526,446
  • Patent reference 2 U.S. Pat. No. 6,937,772
  • Patent reference 3 U.S. Pat. No. 6,754,398
  • an image processing method adopted to remove noise present in an image comprises: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which a plurality of band-limited images with resolutions decreasing in sequence are generated by filtering the input original image; a first noise removal step in which virtual noise removal processing is executed individually for each of the band-limited images; a second noise removal step in which actual noise removal processing is executed for the individual band-limited images based upon the band-limited images from which noise has been virtually removed through the first noise removal step; and an image acquisition step in which a noise-free image of the original image is obtained based upon the individual band-limited images from which noise has been actually removed through the second noise removal step, and the virtual noise removal processing executed in the first noise removal step and the actual noise removal processing executed in the second noise removal step are differentiated in correspondence to a frequency band of a band-limited image.
  • noise is removed to a greater extent in the virtual noise removal processing executed in the first noise removal step relative to the actual noise removal processing executed in the second noise removal step.
  • an image processing method adopted to remove noise present in an image comprises: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which a plurality of band-limited images with resolutions decreasing in sequence are generated by filtering the input original image; a first noise extraction step in which noise components contained in the band-limited images are individually extracted; a second noise extraction step in which noise components to be reflected in the original image are re-extracted from the individual band-limited images based upon the noise components having been extracted from the band-limited images in the first noise extraction step; a noise synthesis step in which the noise components having been re-extracted from the band-limited images in the second noise extraction step are synthesized; and an image acquisition step in which a noise-free image of the original image is acquired based upon the synthesized noise component.
  • the noise components are intrinsically re-extracted from the individual band-limited images in correspondence to frequency bands of the band-limited images in the second noise extraction step.
  • the image processing method in the image processing method according to the 4th aspect, it is preferred that: low-frequency images and high-frequency images with resolutions decreasing in sequence are generated in the multiple-resolution images generation step; and the noise components to be reflected in the original image are re-extracted by weighting the noise components having been extracted in the first noise extraction step differently between the low-frequency images and the high-frequency images in the second noise extraction step.
  • the noise components to be reflected in the original image are re-extracted by weighting the noise components having been extracted in the first noise extraction step differently between band-limited images with varying levels of resolution in the second noise extraction step.
  • an image processing method adopted to remove noise present in an image comprises: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which a plurality of band-limited images with resolutions decreasing in sequence are generated by filtering the input original image; a noise extraction step in which a noise component contained in one of the band-limited images is extracted and then a noise component in another band-limited image is extracted in sequence based upon the noise component having been extracted; and a noise removal step in which noise in the original image is removed based upon noise components in the individual band-limited images having been extracted, and a signal intensity level with which the noise component extracted from one band-limited image in the noise extraction step is reflected when extracting the noise component from another band-limited image during the noise extraction step is set differently from a signal intensity level with which the noise component extracted from one band-limited image in the noise extraction step is reflected in the original image during the noise removal step.
  • the noise component contained in the one band-limited image is extracted and a noise component contained in another band-limited image assuming a different resolution is extracted in sequence based upon the noise component having been extracted in the noise extraction step.
  • the 9th aspect of the present invention in the image processing method according to the 7th aspect, it is preferred that low-frequency images and high-frequency images with resolutions decreasing in sequence are generated in the multiple-resolution images generation step.
  • an image processing method adopted to remove noise present in an image comprises: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which sets of band-limited images each constituted with at least two types of band-limited images are generated in sequence over a plurality of resolutions by filtering the input original image, generating a set of band-limited images constituted with at least two different types of band-limited images and then repeatedly filtering at least one type of band-limited images among the band-limited images; a noise extraction step in which processing whereby noise components contained in the two types of band-limited images are extracted and a noise component in at least one type of band-limited image at another resolution is extracted based upon the extracted noise components, is executed in sequence so as to extract noise components in the individual band-limited images at various resolutions; and a noise removal step in which noise in the original image is removed from based upon noise components in the individual band-limited images having been extracted, and an inter-band signal intensity with
  • the image processing method in the image processing method according to the 7th aspect, it is preferred that two different types of images that are low-frequency images and high-frequency images are generated to make up the sets of band-limited images constituted with at least two different types of band-limited images in the multiple resolution image generation step.
  • an image processing method adopted to remove noise present in an image comprises: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which one or more low-frequency images with resolutions decreasing in sequence and one or more high-frequency images with the resolutions decreasing in sequence are generated by decomposing the input original image; a noise modulation step in which a noise component contained in each low-frequency image and a noise component contained in each high-frequency image are individually extracted, a low-frequency noise image and a high-frequency noise image are generated in correspondence to the low-frequency image and the high-frequency image and weights applied to the noise components over different frequency bands are modulated by applying a weighting coefficient to at least either the low-frequency noise image or the high-frequency noise image having been generated; a noise synthesis step in which the low-frequency noise image and the high-frequency noise image having undergone the noise modulation step are combined to generate a single synthesized noise image with higher resolution by one stage and noise images
  • an image processing method adopted to remove noise present in an image comprises: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which one or more low-frequency images having resolutions decreasing in sequence and one or more high-frequency images with the resolutions decreasing in sequence are generated by decomposing the input original image; a noise extraction step in which a noise component contained in each low-frequency image and a noise component contained in each high-frequency image are individually extracted and a low-frequency noise image and a high-frequency noise image are generated in correspondence to the low-frequency image and the high-frequency image; a noise modulation step in which weights applied to the noise components over different frequency bands are modulated by applying a weighting coefficient to at least either the low-frequency noise image or the high-frequency noise image having been generated; a noise synthesis step in which the low-frequency noise image and the high-frequency noise image having undergone the noise modulation step are combined to generate a single synthesized noise image with higher
  • an image processing method adopted to remove noise present in an image comprises: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which one or more low-frequency images with resolutions decreasing in sequence and one or more high-frequency images with the resolutions decreasing in sequence are generated by decomposing the input original image; a noise extraction step in which a noise component contained in each low-frequency image and a noise component contained in each high-frequency image are individually extracted and a low-frequency noise image and a high-frequency noise image are generated in correspondence to the low-frequency image and the high-frequency image; a noise modulation step in which weights applied to the noise components over different frequency bands are modulated by applying a weighting coefficient to at least either the low-frequency noise image or the high-frequency noise image having been generated; a noise synthesis step in which the low-frequency noise image and the high-frequency noise image having undergone the noise modulation step are combined to generate a single synthesized noise image with higher
  • an image processing method adopted to remove noise present in an image comprises: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which one or more low-frequency images with resolutions decreasing in sequence and one or more high-frequency images with the resolutions decreasing in sequence are generated by decomposing the input original image; a noise extraction step in which a noise component contained in each low-frequency image and a noise component contained in each high-frequency image are individually extracted and a low-frequency noise image and a high-frequency noise image are generated in correspondence to the low-frequency image and the high-frequency image; a noise modulation step in which weights applied to the noise components over different frequency bands are modulated by applying a weighting coefficient to at least either the low-frequency noise image or the high-frequency noise image having been generated; a noise synthesis step in which the low-frequency noise image and the high-frequency noise image having undergone the noise modulation step are combined to generate a single synthesized noise image with higher
  • the low-frequency noise image and the high-frequency noise image corresponding to the low-frequency image and the high-frequency image are generated based upon observation of local signal values in the low-frequency image and the high-frequency image in the noise extraction step.
  • an image processing method adopted to emphasize edges in an image comprises: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which one or more low-frequency images with resolutions decreasing in sequence and one or more high-frequency images with the resolutions decreasing in sequence are generated by decomposing the input original image; an edge component generation step in which an edge component is extracted by applying a band pass filter to each low-frequency image and an edge component is extracted by applying a band pass filter to each high-frequency image and a low-frequency edge component image and a high-frequency edge component image are generated in correspondence to the low-frequency image and the high-frequency image; an edge component modulation step in which weights applied to the edge components over different frequency bands are modulated by applying a weighting coefficient to at least either the low-frequency edge component image or the high-frequency edge component image having been generated; an edge component synthesis step in which the low-frequency edge component image and the high-frequency edge component image having undergone the edge component
  • an image processing method adopted to emphasize edges in an image comprises: an image input step in which an original image constituted of a plurality of pixels is input; a multiple-resolution images generation step in which one or more low-frequency images having resolutions decreasing in sequence and one or more high-frequency images with the resolutions decreasing in sequence are generated by decomposing the input original image; an edge component generation step in which an edge component is extracted by applying a band pass filter to each low-frequency image and an edge component is extracted by applying a band pass filter to each high-frequency image and a low-frequency edge component image and a high-frequency edge component image are generated in correspondence to the low-frequency image and the high-frequency image; an edge component modulation step in which weights applied to the edge components over different frequency bands are modulated by applying a weighting coefficient to at least either the low-frequency edge component image or the high-frequency edge component image having been generated; an edge component synthesis step in which the low-frequency edge component image and the high-frequency edge component image having undergone the edge component
  • the low-frequency image and the high-frequency image correspond to 1) a low-frequency component and a high-frequency component generated through orthogonal wavelet transformation, 2) a Gaussian component and a Laplacian component in a Laplacian pyramid representation, or 3) a low-frequency component and high-frequency components each corresponding to a specific direction in directional wavelet transformation.
  • the low-frequency image corresponds to an LL subband and the high-frequency image corresponds to an LH subband, an HL subband or an HH subband.
  • an image processing method adopted to remove noise present in an image comprises: inputting an original image constituted of a plurality of pixels; sequentially generating images with varying resolutions by that the input original image undergoes multiple resolution transformation; extracting a noise component by using an image generated at a given resolution; using the noise component extracted at the given resolution for purposes of extracting a noise component at another resolution and synthesizing a noise component to be removed from the original image through inverse multiple resolution transformation; and applying different weights to the noise component extracted at the given resolution to be used for purposes of extracting the noise component at the other resolution and to the noise component extracted at the given resolution to be used to synthesize the noise component to be removed from the original image through inverse multiple resolution transformation.
  • an image processing program enables a computer or an image processing apparatus to execute an image processing method according to any one of the 1st through the 21st aspects.
  • an image processing apparatus comprises an image processing program according to the 22nd aspect installed therein.
  • the present invention assuming the structure described above, allows noise to be removed for noise extraction with a high level of freedom by an extent matching the required intensity without being restricted by the conditions for retaining the image structure intact, so as to assure accurate noise extraction, and also allows the original image structure to be retained intact, by assuring preservation of the image structure.
  • FIG. 1 The image processing apparatus achieved in an embodiment of the present invention
  • FIG. 2 A flowchart of the color space conversion processing executed by the personal computer 1
  • FIG. 3 A flowchart of the luminance component processing executed in a first embodiment
  • FIG. 4 A flowchart of the chrominance component processing executed in the first embodiment
  • FIG. 5 The subband division achieved through five-stage wavelet transformation
  • FIG. 6 The simplest Laplacian filter commonly in use
  • FIG. 7 The weighting coefficients for the low-frequency subband (LL) and the high-frequency subbands (LH, HL, HH) of the noise component in the luminance component
  • FIG. 8 The weighting coefficients for the low-frequency subband (LL) and the high-frequency subbands (LH, HL, HH) of the noise component in a chrominance component
  • FIGS. 9A-9C A setting screen that may be brought up when setting intensity parameters (intensity) ⁇ th, rth, a frequency characteristics adjustment parameter (graininess) k0 and a parameter (sharpness) ⁇ related to the noise removal intensity
  • FIG. 10 A flowchart of the luminance component processing executed in a second embodiment
  • FIG. 11 A flowchart of the chrominance component processing executed in the second embodiment
  • FIG. 12 A flowchart of the edge emphasis processing using multiple resolution transformation
  • FIG. 13 The weighting coefficients for the low-frequency subband (LL) and the high-frequency subbands (LH, HL, HH) of the edge component in the luminance component
  • FIG. 14 A schematic diagram illustrating the relationship between the low-frequency subband and the high-frequency subbands in various types of multiple-resolution representations
  • FIG. 15 A schematic illustration of frequency bands covered in the high-frequency subbands and the low-frequency subband in a multiple-resolution representations
  • FIG. 16 A flowchart of luminance component processing similar to that shown in FIG. 3 , except that specific noise removal processing on the image signals in the real space is skipped
  • FIG. 17 A flowchart of luminance component processing similar to that shown in FIG. 10 , except that specific noise removal processing on the image signals in the real space is skipped
  • FIG. 18 A flowchart of edge emphasis processing using multiple resolution transformation, similar to that shown in FIG. 12 except that specific edge component extraction processing on the image signals in the real space is skipped
  • FIG. 19 The structure adopted in a personal computer
  • FIG. 20 A flowchart of the noise removal processing executed to remove noise in the luminance component (luminance signal) in a variation of the first embodiment
  • noise is removed from high-frequency-side subbands (LH, HL, HH) resulting from the orthogonal wavelet transformation.
  • noise is removed sequentially from the low-frequency subband (LL) resulting from the orthogonal wavelet transformation.
  • Noise is normally removed from a color image by dividing the color image into a luminance (brightness) plane and chrominance (color difference) planes and individually removing noise from the luminance plane and the chrominance planes. Grainy noise is reduced by removing noise from the luminance plane, whereas color mottling noise is reduced by removing noise from the chrominance planes.
  • noise can be removed from the luminance component more effectively through noise removal from the high-frequency subbands rather than through the method whereby noise is removed sequentially from the low-frequency component.
  • the sequential noise removal from the low-frequency subband in the luminance component is problematic in that the gradation can be easily lost and a flat image that looks “binary image” may be created.
  • the noise removal from the high-frequency subbands assures gradation retention and preserves the image structure such as the texture of the image in a desirable manner.
  • This characteristic difference between the luminance component and the chrominance components is assumed to be attributable to the difference between the frequency characteristics of the image structure on the luminance plane and the frequency characteristics of the image structure on the chrominance planes, which necessitates the noise components in the luminance plane and the chrominance planes to be extracted in different frequency spaces.
  • the noise removal for the luminance component was executed by removing noise from the high-frequency subbands as in the related art and the noise removal for the chrominance components was executed by sequentially removing noise from the low-frequency subband as in the related art.
  • the results of this experiment indicated that no matter how superior an edge-preserving smoothing filter may be used as each noise removal filter, noise components manifesting as streaks or checkered patterns will not be removed over flat areas in the luminance component and color noise manifesting as points is likely to remain to a significant extent, particularly around color boundary areas in the chrominance components.
  • FIG. 15 schematically illustrates the frequency bands covered by the high-frequency subbands and the low-frequency subband in a multiple-resolution representation.
  • the issues discussed above are further examined, first with regard to the luminance component. Since the original image can be completely reconstructed simply by expressing it with a low-frequency subband corresponding to the lowest resolution and high-frequency subbands corresponding to the individual resolutions, the noise component over the full frequency band would appear to be covered at a superficial level simply through the noise removal from the high-frequency subbands alone. However, as noise is sequentially removed from the high-frequency components at varying levels of resolution, the noise component may not be extracted to the full extent over areas where the intensity of the frequency bands overlapping each other in different resolutions is low.
  • the noise component over the full frequency band would appear to be covered at a superficial level with regard to the chrominance component processing, simply as the noise in the low-frequency subband alone is removed.
  • point-like noise are primarily detected as high-frequency component-side signals as the original image is broken down to the low-frequency component and the high-frequency components, allowing the noise component entering the high-frequency component side, to remain.
  • the opposite characteristics discussed above can be assumed to cause the difference in the optimal frequency space for the luminance component noise extraction and the chrominance component noise extraction. Namely, the findings obtained through the conducted testing substantiate the fact that the smoothing filtering processing executed on the real space plane handled via a single channel or on low-frequency side subbands in a multiple-resolution representation handled via multiple channels is likely to lead to gradation loss whereby the gradations are adjusted to a uniform level matching the average within the filtering range.
  • the chrominance components are generally regarded to readily reflect an image component representing overall color information with a gentle behavioral pattern over a wide range.
  • the chrominance components are thus normally assumed not to contain significant color texture that changes drastically. Accordingly, the noise component can be separated from the chrominance components more easily on the low-frequency side, unlike the noise component in the luminance component, which can be extracted more easily on the high-frequency side.
  • the noise component fluctuation information readily flows into the high-frequency subbands as well, as is widely known, and a target original image may contain significant color texture, noise component separation on the high-frequency subband side is also likely to be necessary.
  • the conjugate subband is the low-frequency subband when removing the noise in the luminance component but is a high-frequency subband when removing the noise in a chrominance component.
  • the noise component extraction and the actual noise removal are regarded as separate processes and the role of the conjugate subband in the actual noise removal is basically regarded as a complementary role, so as to prevent destruction of the image structure.
  • noise is removed from the luminance component by assigning the high-frequency side subbands as primary bands and assigning the low-frequency side subband as a complementary band
  • noise is removed from the chrominance component by assigning the low-frequency side subband as a primary band and assigning the high-frequency side subbands as complementary bands.
  • the roles of the primary band and the complementary bands do not need to be as distinguishable as those of the primary bands and the complementary band used in the luminance component noise removal and the levels of their efficacy may be substantially equal to each other, according to the findings obtained through testing.
  • the noise extraction and the noise removal are conceptualized separately, allowing for flexible thinking in that as long as a subband image is used solely for purposes of noise extraction, the subband image can be virtually destroyed as much as needed down to the level at which accurate noise extraction is enabled.
  • there are two different types of noise removal i.e., virtual noise removal executed for noise extraction and noise removal executed as the actual noise removal processing.
  • an environment in which the residual streaking noise in the luminance component can be extracted easily as data clearly distinguishable from the image structure data in the low-frequency image, is created and an environment in which the residual point-like noise in the chrominance components can be easily extracted as data clearly distinguishable from the image structure data in a high-frequency image is created.
  • noise is fully removed from an upper layer subband image or a lower layer subband image at different resolutions to achieve a noise-free state even if it means the image structure is virtually destroyed, the results of the noise removal are reflected on the subband at the currently targeted resolution and noise is extracted sequentially as the resolution is switched.
  • the sequential noise removal is advantageous in that when it is adopted in noise removal from both the high-frequency band and the low-frequency band, the noise extraction performance in the complementary band is significantly enhanced. Namely, noise manifesting as vertical/horizontal streaks or a checkered pattern present in the low-frequency side complementary band can be completely extracted through the sequential noise removal adopted in conjunction with the luminance component, whereas point-like noise present in the high-frequency side complementary band can be extracted fully through the sequential noise removal adopted in conjunction with the chrominance components.
  • the use of orthogonal wavelet transformation with a lesser overcompleteness of processing as a secondary two-dimensional separation filter also indirectly contributes to the specific characteristics of the residual noise component in the luminance component, manifesting as the vertical/horizontal streaks or checkered patterns.
  • Steerable wavelets may be utilized so as to generate high-frequency bands in correspondence to multiple directions in the multiple resolution transformation.
  • the advantages of the present invention are not limited to noise removal through the orthogonal wavelet transformation and they will be also effective in noise removal in an image in multiple-resolution representations such as the Laplacian-pyramid representation and the Steerable wavelet transformation since the technology according to the present invention successfully complements the weaknesses in the multiple resolution transformation filter characteristics of the Laplacian-pyramid representation and the Steerable wavelet transformation.
  • Virtual noise removal may be sequentially reflected among various resolutions by disintegrating the resolution toward the lower side or by integrating the resolution toward the higher side.
  • the former method is referred to as “sequential analysis” and the latter is referred to as “sequential synthesis” in the description of embodiments.
  • analysis in this context implies breaking down image data to multiple resolution data at lower resolutions, whereas the term “synthesis” in this context is equivalent to integrating the broken down multiple resolution data into the initial high-resolution data (synthesizing the initial high-resolution data).
  • synthesis in this context is equivalent to integrating the broken down multiple resolution data into the initial high-resolution data (synthesizing the initial high-resolution data).
  • analysis refers to the wavelet transformation
  • synthesis refers to inverse wavelet transformation.
  • FIG. 1 shows the image processing apparatus achieved in an embodiment of the present invention.
  • the image processing apparatus is constituted with a personal computer 1 .
  • the personal computer 1 which is connected with a digital camera 2 , a recording medium 3 such as a CD-ROM, another computer 4 and the like, is able to receive various types of image data.
  • the personal computer 1 executes the image processing to be explained below on the image data provided thereto.
  • the personal computer 1 is connected to the computer 4 via an electronic communication network 5 , which may be the Internet.
  • the program that enables the computer 1 to execute the image processing is provided in a recording medium such as a CD-ROM or by another computer via the Internet or another electronic communication network connected to the personal computer in a manner similar to that shown in FIG. 1 , and the program thus provided is installed within the personal computer 1 .
  • FIG. 19 shows the structure adopted in the personal computer 1 .
  • the personal computer 1 comprises a CPU 11 and its peripheral circuits 13 .
  • the installed program is executed by the CPU.
  • the program to be provided via the Internet or another electronic communication network is converted to and transmitted as a signal on a carrier wave which is carried through the electronic communication network, i.e., a transmission medium.
  • the program can be distributed as a computer-readable computer program product adopting any of various modes including a recording medium and a carrier wave.
  • FIG. 2 presents a flowchart of the image processing executed by the personal computer 1 in the first embodiment.
  • step S 1 linear RGB image data are input.
  • step S 2 the input image data are converted to data in a uniform color and uniform noise space.
  • step S 3 the data undergo noise removal processing.
  • step S 4 the color space data undergo reverse conversion.
  • step S 5 the image data having undergone the processing are output. The following is a detailed explanation of the processing executed in each step.
  • step S 1 RGB color image data with linear gradation with respect to the light intensity are input.
  • step S 2 the input image data are converted to data in a uniform noise space in which the noise is set to the uniform level relative to the gradations, so as to create an environment in which noise removal is facilitated.
  • the image data are converted to data in a uniform color and uniform noise space so as to further assure color uniformity as well as noise uniformity, thereby providing both the noise removal effect and the color reproducibility retention effect.
  • Such a uniform color and uniform noise space used as the image processing space is described in Japanese Patent Application No. 2004-365881 submitted by the inventor of the present invention and accordingly, Japanese Patent Application No. 2004-365881 should be referred to for details of the uniform color and uniform noise space.
  • the following explanation is provided by assuming that the target input data are sRGB image data. It is to be noted, however, that gamma correction, having been executed on the sRGB image should first be undone, to revert to the initial image with linear gradation before starting the processing.
  • the linear gradation RGB values are converted to XYZ values.
  • the image data are converted to the XYZ calorimetric system space.
  • This processing is executed through 3 ⁇ 3 matrix conversion, which is defined in correspondence to the RGB original stimulus spectral characteristics.
  • the data constituting the input sRGB image may undergo the following standardized conversion.
  • X 0.4124 *R+ 0.3576 *G+ 0.1805 *B (1)
  • Y 0.2126 *R+ 0.7152 *G+ 0.0722 *B
  • Z 0.0193 *R+ 0.1192 *G+ 0.9505 *B (3)
  • L ⁇ a ⁇ bb ⁇ space is used to refer to a variation of the uniform color space L*a*b* in the related art achieved by modifying the L*a*b* space so as to assume noise uniformity.
  • X0, Y0 and Z0 in the expressions above each represent a value determined in correspondence to the illuminating light.
  • X0, Y0 and Z0 may assume values 95.045, 100.00 and 108.892 respectively in a 2° visual field under standard light D65.
  • the nonlinear gradation conversion function ⁇ (t) is defined below. Noise uniformity is achieved based upon the characteristics of the function ⁇ (t).
  • ⁇ ( t ) t+ ⁇ square root over (t+ ⁇ ) ⁇ (7)
  • origin point and the saturation point may be normalized by using the following expression.
  • ⁇ in the expression above represents an offset signal applied to the linear gradation signal. While ⁇ assumes a value depending on the specific sensor in use, a value very close to zero, for instance, will be assumed when a low sensitivity setting is selected and a value approximately equal to 0.05 will be assumed when a high sensitivity setting is selected.
  • FIG. 3 presents a flowchart of the processing executed for the luminance (brightness) component (luminance signal)
  • FIG. 4 presents a flowchart of the processing executed for a chrominance (color difference) component (color different signal).
  • FIG. 4 simply presents the part of the chrominance component processing that differs from the luminance component processing in FIG. 3 , as detailed later.
  • FIGS. 3 and 4 illustrate multiple resolution (multiresolution) transformation achieved through five-stage wavelet transformation
  • the number of stages over which wavelet transformation is executed may be adjusted in correspondence to the size of the input original image. As long as the wavelet transformation is executed over approximately five stages, the frequency band of the target noise component can be substantially covered under normal circumstances.
  • the frequency component in the image is divided into a high-pass component and a low-pass component.
  • a 5/3 filter is utilized to execute the five-stage wavelet transformation mentioned above.
  • the 5/3 filter generates the low-pass component with a filter having five taps (one-dimension/5 pixels) and generates the high-pass component with a filter having three taps (one-dimension/3 pixels).
  • the high-pass component and the low-pass component are generated as expressed below.
  • n indicates the pixel position and x[ ] indicates the pixel value assumed at the target image undergoing the wavelet transformation. For instance, n assumes a value in the range of 0 ⁇ 49 if 100 pixels are set along the horizontal direction.
  • Low-pass component: s[n] x[ 2 n ]+( d[n]+d[n ⁇ 1])/4 (10)
  • the one-dimensional wavelet transformation defined as described above is executed along the horizontal direction and the vertical direction independently through two-dimensional separation filtering processing so as to achieve wavelet decomposition.
  • the coefficient s is directed onto the L plane, whereas the coefficient d is directed onto the H plane.
  • the real space plane, identical to the input image, is designated as an LL 0 plane.
  • the LL 0 plane is handled as the low-frequency subband with the highest resolution as well as the wavelet transformation coefficient low-frequency subband LL 1 , LL 2 , LL 3 , LL 4 and LL 5 .
  • the five-stage wavelet transformation is executed sequentially as indicated below by using the expressions presented earlier.
  • the wavelet transformation is executed as noise signals are extracted sequentially by using the LL component data and the LH, HL and HH component data generated at the individual stages, as described later.
  • LL component data are low-frequency subband data
  • LH, HL and HH data are high-frequency subband data.
  • a low-frequency subband may be referred to as a low-frequency image
  • a high-frequency subband may be referred to as a high-frequency image.
  • each subband may be referred to as a frequency band-limited image.
  • a low-frequency subband is an image with band limits imposed upon the frequency bands of the original image toward the low-frequency side
  • a high-frequency subband is an image with band limits imposed upon the frequency band of the original image toward the high-frequency side.
  • Second-stage wavelet transformation LL 1 ⁇ LL 2 , LH 2 , HL 2 , HH 2
  • FIG. 5 shows the subband partition achieved through the five-stage wavelet transformation.
  • high-pass component data and low-pass component data are extracted from the image data in all the rows extending along the horizontal direction in the real space.
  • high-pass component data and low-pass component data corresponding to half the entire number of pixels are extracted along the horizontal direction.
  • the extracted high-pass component data and low-pass component data may then be stored into, for instance, memory areas on the right side and on the left side of the memory area where the image data in the real space have been present.
  • high-pass component data and low-pass component data are extracted as expressed earlier, individually from the high-pass component data having been stored on the right side in the memory area and from the low-pass component data having been stored on the left side in the memory area, along all the columns extending in the vertical direction.
  • high-pass component data and low-pass component data are extracted both from the high-pass component data stored on the right side in the memory area and from the low-pass component data stored on the left side in the memory area.
  • the high-pass component data and the low-pass component data thus extracted are stored on the bottom side and the top side respectively in the memory area where the corresponding source data have been present.
  • HH indicates data extracted as high-pass component data along the vertical direction from data having been extracted as high-pass component data along the horizontal direction
  • HL indicates data extracted as low-pass component data along the vertical direction from the data having been extracted as high-pass component data along the horizontal direction
  • LH indicates data extracted as high-pass component data along the vertical direction from data having been extracted as low-pass component data along the horizontal direction
  • LL indicates data extracted as low-pass component data along the vertical direction from the data having been extracted as low-pass component data along the horizontal direction.
  • high-pass component data and low-pass component data are extracted in a similar manner from the data LL extracted through the first-stage wavelet transformation as low-pass component data along the vertical direction from the data having been extracted as the low-pass component data along the horizontal direction.
  • the partition shown in FIG. 5 is achieved by repeatedly executing the processing described above over five stages.
  • a signal expressing the image is input to be used as the value x for the wavelet transformation, the noise components contained in the generated wavelet transformation coefficients s and dare extracted and a noise image x is generated by substituting the extracted noise components for s and d used in the inverse wavelet transformation.
  • Any noise removal filter may be used in the noise removal processing executed on the individual subband planes.
  • Typical examples of edge-preserving smoothing filters include a filters such as that described in reference “Jong—Sen Lee, “Digital image smoothing and the Sigma filter”, Computer Vision, Graphics and Image Processing 24 (1983) pp 255 ⁇ 269” and bilateral filters such as that described in reference “C. Tomasi et al., “Bilateral Filtering for Gray and Color Images,” Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay, India”.
  • V(vector r) represents an original signal in the input subband image plane whereas V′(vector r) or V′′(vector r) represents a signal on an image plane having undergone noise removal.
  • r with an arrow referred to as vector r
  • r′ with an arrow referred to as vector r′
  • each representing a vector indicate two dimensional positional coordinates.
  • V ′ ⁇ ( r ⁇ ) ⁇ ⁇ r ⁇ ′ ⁇ ⁇ 2 ⁇ r th ⁇ V ⁇ ( r ⁇ ′ ) ⁇ exp ⁇ ⁇ - ⁇ V ⁇ ( r ⁇ ′ ) - V ⁇ ( r ⁇ ) ⁇ 2 ⁇ th 2 ⁇ ⁇ r ⁇ ′ - r ⁇ ⁇ 2 r th 2 ⁇ ⁇ ⁇ d r ⁇ ′ ⁇ ⁇ r ⁇ ′ ⁇ ⁇ 2 ⁇ r th ⁇ exp ⁇ ⁇ - ⁇ V ⁇ ( r ⁇ ′ ) - V ⁇ ( r ⁇ ) ⁇ 2 ⁇ th 2 ⁇ ⁇ r ⁇ ′ - r ⁇ ⁇ 2 r th 2 ⁇ ⁇ d r ⁇ ′ ( 13 )
  • the threshold value rth taken along the spatial direction should assume a value within a range of approximately 0.5 ⁇ 3.0 pixels so as to create an overlap of hierarchical layers with different resolutions, since the noise removal filter assumes a range approximately twice the threshold value.
  • the threshold value may also be adjusted in correspondence to the image-capturing sensitivity.
  • the threshold value ⁇ th taken along the gradation direction should assume a greater value as a higher image-capturing sensitivity level is selected and the optimal value should be adjusted in correspondence to each subband as well.
  • the filter weighting coefficient in the bilateral filter in the related art is represented by the product of the weighting coefficient w_photo [V′ ⁇ V] of the photometric term that takes the difference (V′ ⁇ V) between pixel values alone as an argument and the weighting coefficient w_geometric[r′ ⁇ r] of the geometric term which takes the spatial distance (r′ ⁇ r) alone as an argument.
  • the bilateral filter in the related art may be referred to as a separately weighted bilateral filter, the weighting coefficient of which can be separated into the photometric term and the geometric term.
  • the weighting coefficient of the modified bilateral filter cannot be separated into the photometric term and the geometric term. In other words, it is an integrated weighted bilateral filter, the weighting coefficient of which is expressed with a single exponential function, taking on an exponent assuming the value matching the product of the two arguments.
  • V ′( ⁇ right arrow over (r) ⁇ ) V ( ⁇ right arrow over (r) ⁇ ) ⁇ 2 V ( ⁇ right arrow over (r) ⁇ ) ⁇ ( ⁇ 2 V ( ⁇ right arrow over (r) ⁇ )) (14)
  • V ′( ⁇ right arrow over (r) ⁇ ) V ( ⁇ right arrow over (r) ⁇ ) ⁇ 2 V ( ⁇ right arrow over (r) ⁇ ) ⁇ ( ⁇ 2 V ( ⁇ right arrow over (r) ⁇ )) (15)
  • V ′′( ⁇ right arrow over (r) ⁇ ) V ′( ⁇ right arrow over (r) ⁇ )+ ⁇ 2 V ′( ⁇ right arrow over (r) ⁇ ) ⁇ ( ⁇ 2 V ′( ⁇ right arrow over (r) ⁇ )) (16)
  • ⁇ (x) is a function expressed below.
  • ⁇ 2 represents the Laplacian filter (high-pass filter).
  • FIG. 6 shows the simplest Laplacian filter in common use.
  • the threshold value ⁇ th taken along the gradation direction should be set based upon a concept similar to that applied to the modified bilateral filter detailed above. It will be obvious that individually selected optimal values should be set as the threshold value for the luminance component data and the chrominance component data.
  • the modified bilateral filter and the Laplacian filter described above are each a function of the values of signals contained in localized areas. Namely, noise is extracted by checking the values indicated in the signals in the individual localized areas in the low-frequency subband and the high-frequency subbands.
  • noise is removed from an image signal S 0 (LL 0 ) in the real space through the noise removal filter described above, so as to generate a noise-free image signal S 0 ′(LL 0 ).
  • processing ( 0 - 3 ) the noise signal n 0 (LL 0 ) retaining the initial intensity (or multiplied by a factor of ⁇ (0)) is subtracted from the image signal S 0 (LL 0 ), thereby removing the noise from S 0 (LL 0 ).
  • noise is removed from the individual image signals S 1 (LL 1 , LH 1 , HL 1 , HH 1 ) via the noise removal filter described above and, as a result, noise-free image signals S 1 ′(LL 1 , LH 1 , HL 1 , HH 1 ) are generated.
  • n 1 (LL 1 ) S 1 (LL 1 ) ⁇ S 1 ′(LL 1 )
  • n 1 (LH 1 ) S 1 (LH 1 ) ⁇ S 1 ′(LH 1 )
  • n 1 (HL 1 ) S 1 (HL 1 ) ⁇ S 1 ′(HL 1 )
  • n 1 (HH 1 ) S 1 (HH 1 ) ⁇ S 1 ′(HH 1 ).
  • the noise signal n 1 (LL 1 ) retaining the initial intensity (or multiplied by a factor of ⁇ (1)) is subtracted from the image signal S 1 (LL 1 ), thereby removing the noise from S 1 (LL 1 ). It is to be noted that 0 ⁇ (1) ⁇ 1 and that ⁇ (1) normally takes on the value of 1.
  • the image signal in the LL 1 plane, having undergone noise removal through the processing ( 1 - 3 ) undergoes wavelet transformation, thereby generating image signals S 2 (LL 2 , LH 2 , HL 2 , HH 2 ) at 1 ⁇ 4 resolution.
  • This processing is executed much the same way as the processing executed at the 1 ⁇ 2 resolution described in (2-3-1-2) above.
  • This processing is executed much the same way as the processing executed at the 1 ⁇ 2 resolution described in (2-3-1-2) above.
  • noise is removed from the individual image signals S 4 (LL 4 , LH 4 , HL 4 , HH 4 ) via the noise removal filter described above and, as a result, noise-free image signals S 4 ′(LL 4 , LH 4 , HL 4 , HH 4 ) are generated.
  • n 4 (LL 4 ) S 4 (LL 4 ) ⁇ S 4 ′(LL 4 )
  • n 4 (LH 4 ) S 4 (LH 4 ) ⁇ S 4 ′(LH 4 )
  • n 4 (HL 4 ) S 4 (HL 4 ) ⁇ S 4 ′(HL 4 )
  • n 4 (HH 4 ) S 4 (HH 4 ) ⁇ S 4 ′(HH 4 ).
  • the noise signal n 4 (LL 4 ) retaining the initial intensity (or multiplied by a factor of ⁇ (4)) is subtracted from the image signal S 4 (LL 4 ), thereby removing the noise from S 4 (LL 4 ). It is to be noted that 0 ⁇ (4) ⁇ 1 and that ⁇ (4) normally takes on the value of 1.
  • the image signal in the LL 4 plane, having undergone noise removal through the processing ( 4 - 3 ) undergoes wavelet transformation, thereby generating image signals S 5 (LL 5 , LH 5 , HL 5 , HH 5 ) at 1/32 resolution.
  • noise is removed from the individual image signals S 5 (LL 5 , LH 5 , HL 5 , HH 5 ) via the noise removal filter described above and, as a result, noise-free image signals S 5 ′(LL 5 , LH 5 , HL 5 , HH 5 ) are generated.
  • n 5 (LL 5 ) S 5 (LL 5 ) ⁇ S 5 ′(LL 5 )
  • n 5 (LH 5 ) S 5 (LH 5 ) ⁇ S 5 ′(LH 5 )
  • n 5 (HL 5 ) S 5 (HL 5 ) ⁇ S 5 ′(HL 5 )
  • n 5 (HH 4 ) S 5 (HH 5 ) ⁇ S 5 ′(HH 5 ).
  • the processing described above differs from the related art in that the noise components in the high-frequency subbands LH, HL and HH on the low-resolution side generated from a low-frequency subband LL having undergone the sequential noise removal, too, are extracted with a high level of accuracy after the subbands data first undergo the high-resolution-side noise removal.
  • the results of the noise removal executed on an upper layer low-frequency subband affect noise extraction from the lower layer high-frequency subbands as well as the lower layer low-frequency subband.
  • the noise components can be extracted both from the low-frequency subband and the high-frequency subbands, containing very little residual noise to begin with, in the multiple-resolution representations.
  • the extracted noise components are modified into noise components to be used for the actual noise removal.
  • This modification is achieved by re-extracting noise components for the actual noise removal from the extracted noise components.
  • This process executed to assure the indestructibly of the luminance component image structure, fulfills the role of a variable parameter with which the visual effect achieved through noise removal can be adjusted with ease.
  • the noise component frequency characteristics are adjusted by altering the weight applied to the low-frequency subband (LL) and the high-frequency subbands (LH, HL, HH) relative to each other.
  • the parameter may be provided as a graininess-modifying parameter for the noise removal in a graphic user interface in software processing or the like.
  • different weighting coefficients are applied to the noise component in a low-frequency subband and the noise components in the high-frequency subbands (e.g., k0 applied to the LL subband and 1 applied to the other subbands in the example presented below, so as to modulate the weights applied to different noise component frequency bands.
  • This process is executed as expressed below and the individual processing procedures correspond to the processing ( 0 - 5 ), the processing ( 1 - 5 ), the processing ( 2 - 5 ), the processing ( 3 - 5 ), the processing ( 4 - 5 ) and the processing ( 5 - 5 ) in FIG. 3 .
  • n 1 ′(LL 1 ) and n 1 (LH 1 , HL 1 , HH 1 ) are directly bundled together into n 1 ′(LL 1 , LH 1 , HL 1 , HH 1 ).
  • n 2 ′(LL 2 ) and n 2 (LH 2 , HL 2 , HH 2 ) are directly bundled together into n 2 ′(LL 2 , LH 2 , HL 2 , HH 2 ).
  • n 3 ′(LL 3 ) and n 3 (LH 3 , HL 3 , HH 3 ) are directly bundled together into n 3 ′(LL 3 , LH 3 , HL 3 , HH 3 ).
  • n 4 ′(LL 4 ) and n 4 (LH 4 , HL 4 , HH 4 ) are directly bundled together into n 4 ′(LL 4 , LH 4 , HL 4 , HH 4 ).
  • n 5 ′(LL 5 ) and n 5 (LH 5 , HL 5 , HH 5 ) are directly bundled together into n 5 ′(LL 5 , LH 5 , HL 5 , HH 5 ).
  • k0(0), k0(1), k0(2), k0(3), k0(4) and k0(5) are all set equal to one another at k0, which is variable within a range of 0 ⁇ k0 ⁇ 1.
  • k0 should assume a value close to the median value, e.g., 0.5, in order to prevent manifestation of significant residual noise components and also preserve textural structure in the image by retaining the optimal level of graininess, whereas a lower value such as 0.2 may be assumed for k0 if it is more important to preserve the image structure by sustaining a higher level of graininess. If, on the other hand, the wide range high-frequency background noise over the entire image plane needs to be eliminated, a higher value, e.g., 0.8, should be assumed for k0.
  • the noise signal in a high-frequency subband sustaining the initial intensity should be directly output under normal circumstances.
  • the weight applied to the high-frequency subbands should be set to a greater value than the weight applied to the low-frequency subband.
  • the high-frequency subbands noise signal may be multiplied by a weighting coefficient.
  • FIG. 7 shows the weighting coefficients applied to the low-frequency subband (LL) and the high-frequency subbands (LH, HL, HH).
  • the present invention adopts the noise removal concept that assumes two different types of noise removal, i.e., the noise removal executed for noise component extraction and the noise removal executed for actual noise removal during which the image structure must remain intact.
  • Noise removal for noise extraction can be executed with as much intensity, as needed, unrestricted by the conditions imposed to assure image structure preservation. Namely, the intensity with which noise is removed for noise component extraction can be increased freely, unlike the intensity with which the noise removal for actual noise removal is executed. Consequently, noise can be extracted accurately from each subband while assuring a satisfactory level of image structure preservation.
  • the frequency characteristics of the synthesized noise component can be adjusted with ease simply by applying a weighting coefficient to a complementary subband among the high-frequency subbands and the low-frequency subband.
  • a weighting coefficient to a complementary subband among the high-frequency subbands and the low-frequency subband.
  • the modified noise components are then synthesized through inverse wavelet transformation executed in sequence from the lowest resolution side.
  • n 4 ′′(LL 4 ) and n 4 ′(LH 4 , HL 4 , HH 4 ) are directly bundled together into n 4 ′′(LL 4 , LH 4 , HL 4 , HH 4 ).
  • the noise component corresponding to the LL 4 plane is generated by combining the noise components over the two layers, as indicated in FIG. 3 .
  • the noise components for LH 4 , HL 4 and HH 4 correspond to a single layer.
  • the noise signals n 4 ′′(LL 4 , LH 4 , HL 4 , HH 4 ) obtained by combining the noise components over the two layers undergo inverse wavelet transformation, so as to generate a noise signal N 4 (LL 3 ) corresponding to the LL 3 subband plane. (2-3-3-3) Processing Executed at 1 ⁇ 8 Resolution
  • This processing is executed in much the same way as the “processing executed at 1/16 resolution” described in (2-3-3-2) above.
  • This processing is executed in much the same way as the “processing executed at 1/16 resolution” described in (2-3-3-2) above.
  • n 1′′( LL 1) n 1′( LL 1)+ N 2( LL 1) (25)
  • n 1 ′′(LL 1 ) and n 1 ′(LH 1 , HL 1 , HH 1 ) are directly bundled together into n 1 ′′(LL 1 , LH 1 , HL 1 , HH 1 ).
  • n 0′′( LL 0) n 0′( LL 0)+ N 1( LL 0) (26).
  • the processing according to the present invention described above differs from the related art in that the noise component for the low-frequency subband is generated through noise synthesis by combining the noise components over two layers, i.e., the noise component obtained by synthesizing the noise components in the low-frequency subband and the high-frequency subbands on the low resolution side and the noise component extracted from the low-frequency subband at the target resolution are combined.
  • the correct noise component minus any residual noise component can be synthesized with ease, while assuring a high level of image structure preservation and noise characteristics that allow the visual effect to be adjusted easily.
  • the actual noise removal is executed on the single synthesized noise component with the resolution matching that in the real space, to which a noise removal factor, i.e., a weighting coefficient parameter ⁇ , has been applied so as to adjust the extent of noise removal for the overall image. Namely, the processing is executed as expressed below.
  • S 0 NR ( LL 0) S 0( LL 0) ⁇ *n0′′( LL 0) (32) with parameter ⁇ assuming value within the range of 0 ⁇ 1.
  • the chrominance component data (a ⁇ ) undergoes noise extraction through the “sequential analysis”.
  • the noise removal processing executed for the chrominance component differs from the noise removal from the luminance component in that the weighting coefficients are applied to different target subbands in order to modify the frequency characteristics from the target subbands in the processing executed to adjust the noise component frequency characteristics described in (2-3-2) above, i.e., different weighting processing is executed and in that the noise removal factor parameter is selected differently from the noise removal factor parameter set in the “Actual noise removal processing” described in (2-3-4) above.
  • FIG. 4 only shows the part of the “noise component frequency characteristics adjustment” processing that is different from the processing shown in FIG. 3 .
  • a weighting coefficient parameter used to assure both a desirable point-like noise removal effect and color retention when actually removing noise from the chrominance component data is applied to the noise components in the high-frequency subbands (LH, HL, HH), since the low-frequency subband is assigned as a primary band and the high-frequency subbands are assigned as complementary bands in the case of the chrominance component data.
  • n 1 (LL 1 ) and n 1 ′(LH 1 , HL 1 , HH 1 ) are directly bundled together into n 1 (LL 1 , LH 1 , HL 1 , HH 1 ).
  • n 2 (LL 2 ) and n 2 ′(LH 2 , HL 2 , HH 2 ) are directly bundled together into n 2 ′(LL 2 , LH 2 , HL 2 , HH 2 ).
  • n 3 (LL 3 ) and n 3 ′(LH 3 , HL 3 , HH 3 ) are directly bundled together into n 3 ′(LL 3 , LH 3 , HL 3 , HH 3 ).
  • n 4 (LL 4 ) and n 4 ′(LH 4 , HL 4 , HH 4 ) are directly bundled together into n 4 ′(LL 4 , LH 4 , HL 4 , HH 4 ).
  • n 5 (LL 5 ) and n 5 ′(LH 5 , HL 5 , HH 5 ) are directly bundled together into n 5 ′(LL 5 , LH 5 , HL 5 , HH 5 ).
  • k1(1), k1(2), k1(3), k1(4) and k1(5) are all set equal to one another at k1 and k2(1), k1(2), k2(3), k2(4) and k2(5) are all set equal to one another at k2.
  • k1 is commonly set for the LH subband and the HL subband, different values may be selected for the LH subband and the HL subband instead.
  • the weighting coefficient applied to the low-frequency subband (LL) is 1 and accordingly, the value initially assumed in the noise signal is directly used. In other words, the weight applied to the low-frequency subband should be set to a greater value than the weights applied to the high-frequency subbands. However, the weights applied to the low-frequency subband and the high-frequency subbands are very nearly the same, as k1 is set to 0.9 and k2 is set to 0.8, both close to 1.
  • the actual noise removal processing for the chrominance component data is executed much the same way as the actual noise removal processing executed for the luminance component data (L ⁇ ) described in (2-3-4).
  • the noise removal factor ⁇ set for the chrominance component may assume the value of 1.0 under normal circumstances.
  • the noise extraction processing is executed in the optimal frequency spaces where the noise components can be extracted with ease, based upon the difference of the image structures and the difference of the noise characteristics between separated luminance and chrominance planes.
  • fine noise removal from a color image is achieved with ease while assuring a minimum loss of image structures and minimum residual noise.
  • the noise removal for the chrominance component (b ⁇ ) is executed in much the same way as the noise removal for the chrominance component (a ⁇ ) described in (2-4).
  • An intensity parameter (intensity) used for noise component extraction ⁇ th (rth as well, in some filters)
  • FIGS. 9A-9C show a setting screen that may be brought up to allow the user to set the intensity parameters (intensity) ⁇ th and rth, the frequency characteristics adjustment parameter (graininess) k0 and the noise removal intensity parameter (sharpness) ⁇ .
  • a slide bar is displayed in correspondence to each parameter so as to allow the user to select a preferred value for a given parameter by setting the pointer in the slide bar to a specific position.
  • the setting screen in FIGS. 9A-9C is brought up on display at the monitor (not shown) of the personal computer 1 and the user sets the pointer in each slide bar to a specific position via a keyboard (not shown) or a mouse (not shown).
  • the user is thus able to set the parameters with ease.
  • the user may adjust the frequency characteristics adjustment parameter (graininess) k0 as explained earlier, so as to alter the visual effect to be achieved through noise removal with ease while preserving fine image characteristics.
  • the operation will be executed in quick response as soon as k0 and ⁇ are changed.
  • step S 4 in FIG. 2 the image data resulting from the noise removal processing executed in step S 3 described above undergo conversion processing which is the reverse of “(1) color space conversion” having been executed in step S 2 so as to convert the image data back to data constituting an RGB image.
  • step S 5 the image data constituting the RGB image resulting from the reverse conversion are output.
  • two types of processing equivalent to two different types of noise removal are executed each for noise extraction or noise removal, and the noise removal results obtained by removing noise from an upper layer low-frequency subband are made to affect the noise extraction from the lower layer high-frequency subbands as well as the noise extraction from the lower layer low-frequency subband.
  • noise is extracted sequentially from both the high-frequency subbands and the low-frequency subband constituting images obtained through multiple resolution transformation while allowing the processing of either type of subbands to be influenced by the characteristics of the other type of subbands. Consequently, the level of freedom with which different frequency bands can be combined in the processing increases and the noise extraction can be executed in the frequency space optimized for the noise extraction. This means that fine noise removal is enabled without loss of image structure while minimizing the residual noise that is not extracted.
  • fine noise removal processing edge-preserving smoothing assuring a high level of image structure preservation while effectively addressing the issue of residual noise in a regular image such as a digital picture, is realized through the embodiment.
  • a specific type of noise removal processing is executed on the image signal S 0 (LL 0 ) in the real space (see FIG. 3 ).
  • a load of the processing executed on the image signal S 0 (LL 0 ) in the real space is bound to be extremely heavy.
  • FIG. 3 a specific type of noise removal processing executed on the image signal S 0 (LL 0 ) in the real space
  • FIG. 16 presents a flowchart of the processing executed on the luminance component (luminance signal) by skipping the specific noise removal processing on the image signal S 0 (LL 0 ) in the real space in FIG. 3 .
  • the chrominance components may be processed in a similar manner. Through these measures, fine noise removal processing can be executed with a lighter processing load.
  • the image processing apparatus in the second embodiment assumes a structure similar to that of the image processing apparatus in the first embodiment illustrated in FIG. 1 , its explanation is omitted.
  • the overall flow of the image processing executed by the personal computer 1 in the second embodiment is similar to that in the flowchart presented in FIG. 2 and, accordingly, its explanation is omitted.
  • the following explanation focuses on the aspects of the second embodiment that differ from those of the processing executed in the first embodiment.
  • FIG. 10 presents a flowchart of the processing executed for luminance component data
  • FIG. 11 presents a flowchart of the processing executed for chrominance component data. It is to be noted that FIG. 11 simply shows the part of the chrominance component processing that is different from the luminance component processing in FIG. 10 . It is also to be noted that the individual processing phases (xx) and (xx-x) described below are also indicated as (xx) and (xx-x) in FIG. 10 .
  • the image signal S 0 (LL 0 ) on the real space plane undergoes wavelet transformation so as to generate image signals S 1 (LL 1 , LH 1 , HL 1 , HH 1 ) at 1 ⁇ 2 resolution.
  • the image signal S 1 (LL 1 ) on the LL 1 plane undergoes wavelet transformation so as to generate image signals S 1 (LL 2 , LH 2 , HL 2 , HH 2 ) at 1 ⁇ 4 resolution.
  • the image signal S 2 (LL 2 ) on the LL 2 plane undergoes wavelet transformation so as to generate image signals S 3 (LL 3 , LH 3 , HL 3 , HH 3 ) at 1 ⁇ 8 resolution.
  • the image signal S 3 (LL 3 ) on the LL 3 plane undergoes wavelet transformation so as to generate image signals S 4 (LL 4 , LH 4 , HL 4 , HH 4 ) at 1/16 resolution.
  • noise is removed from the individual image signals S 5 (LL 5 , LH 5 , HL 5 , HH 5 ) and, as a result, noise-free image signals S 5 ′(LL 5 , LH 5 , HL 5 , HH 5 ) aregenerated.
  • n 5 (LL 5 ) S 5 (LL 5 ) ⁇ S 5 ′(LL 5 )
  • n 5 (LH 5 ) S 5 (LH 5 ) ⁇ S 5 ′(LH 5 )
  • n 5 (HL 5 ) S 5 (HL 5 ) ⁇ S 5 ′(HL 5 )
  • n 5 (HH 5 ) S 5 (HH 5 ) ⁇ S 5 ′(HH 5 ).
  • the noise signal N 5 (LL 4 ) retaining the initial intensity (or multiplied by a factor of ⁇ (5)) is subtracted from the image signal S 4 (LL 4 ), so as to obtain an image signal S 4 ′(LL 4 ).
  • 0 ⁇ (5) ⁇ 1 and that ⁇ (5) normally takes on the value of 1.
  • S 4 ′(LL 4 ) and S 4 (LH 4 , HL 4 , HH 4 ) are directly bundled together into S 4 ′(LL 4 , LH 4 , HL 4 , HH 4 ).
  • noise is removed from the individual image signals S 4 ′(LL 4 , LH 4 , HL 4 , HH 4 ) and, as a result, noise-free image signals S 4 ′′(LL 4 , LH 4 , HL 4 , HH 4 ) are generated while these signals are indicated as S 4 ′′(LL 4 ′′, LH 4 ′, HL 4 ′, HH 4 ′) in FIG. 10 , they are identical to S 4 ′′(LL 4 , LH 4 , HL 4 , HH 4 ).
  • n 4 (LL 4 ) S 4 ′(LL 4 ) ⁇ S 4 ′′(LL 4 )
  • n 4 (LH 4 ) S 4 ′(LH 4 ) ⁇ S 4 ′′(LH 4 )
  • n 4 (HL 4 ) S 4 ′(HL 4 ) ⁇ S 4 ′′(HL 4 )
  • S 4 ′ n 4 (HH 4 ) S 4 ′(HH 4 ) ⁇ S 4 ′′(HH 4 ).
  • n 4′( LL 4) n 4( LL 4)+ N 5( LL 4) (48)
  • n 4 ′(LL 4 ) and n 4 (LH 4 , HL 4 , HH 4 ) are directly bundled together into n 4 ′(LL 4 , LH 4 , HL 4 , HH 4 ).
  • the noise signals n 4 ′(LL 4 , LH 4 , HL 4 , HH 4 ) undergo inverse wavelet transformation to generate a noise signal N 4 (LL 3 ) corresponding to the LL 3 subband plane.
  • This processing is executed much the same way as the processing executed at the 1/16 resolution, as described in (2-3-2-2) above.
  • This processing is executed much the same way as the processing executed at the 1/16 resolution, as described in (2-3-2-2) above.
  • the noise signal N 2 (LL 1 ) retaining the initial intensity (or multiplied by a factor of ⁇ (2)) is subtracted from the image signal S 1 (LL 1 ), so as to obtain an image signal S 1 ′(LL 1 ).
  • 0 ⁇ (2) ⁇ 1 and that ⁇ (2) normally takes on the value of 1.
  • S 1 ′(LL 1 ) and S 1 (LH 1 , HL 1 , HH 1 ) are directly bundled together into S 1 ′(LL 1 , LH 1 , HL 1 , HH 1 ).
  • noise is removed from the individual image signals S 1 ′(LL 1 , LH 1 , HL 1 , HH 1 ) and, as a result, noise-free image signals S 1 ′′(LL 1 , LH 1 , HL 1 , HH 1 ) are generated while these signals are indicated as S 1 ′′(LL 1 ′′, LH 1 ′, HL 1 ′, HH 1 ′) in FIG. 10 , they are identical to S 1 ′′(LL 1 , LH 1 , HL 1 , HH 1 ).
  • n 1 (LL 1 ) S 1 ′(LL 1 ) ⁇ S 1 ′′(LL 1 )
  • n 1 (LH 1 ) S 1 ′(LH 1 ) ⁇ S 1 ′′(LH 1 )
  • n 1 (HL 1 ) S 1 ′(HL 1 ) ⁇ S 1 ′′(HL 1 )
  • n 1 (HH 1 ) S 1 ′(HH 1 ) ⁇ S 1 ′′(HH 1 ).
  • n 1′( LL 1) n 1( LL 1)+ N 2( LL 1) (49)
  • n 1 ′(LL 1 ) and n 1 (LH 1 , HL 1 , HH 1 ) are directly bundled together into n 1 ′(LL 1 , LH 1 , HL 1 , HH 1 ).
  • the noise signals n 1 ′(LL 1 , LH 1 , HL 1 , HH 1 ) undergo inverse wavelet transformation to generate a noise signal N 1 (LL 0 ) corresponding to the LL 0 subband plane.
  • the noise signal N 1 (LL 0 ) retaining the initial intensity (or multiplied by a factor of ⁇ (1)) is subtracted from the image signal S 0 (LL 0 ), so as to obtain an image signal S 0 ′(LL 0 ). It is to be noted that 0 ⁇ (1) ⁇ 1 and that ⁇ (1) normally takes on the value of 1.
  • processing ( 10 - 1 ) noise is removed from the individual image signal S 0 ′(LL 0 ) and, as a result, a noise-free image signal S 0 ′′(LL 0 ) is generated.
  • the feature of the embodiment that is particularly noteworthy is that the effect of the noise removal from the low resolution-side high-frequency subbands, as well as the effect of the noise removal from the low resolution side low-frequency subband is reflected in the execution of noise extraction from the higher resolution-side low-frequency subband. Namely, the results of noise removal from the lower layer low-frequency subband and the lower layer high-frequency subbands together affect the noise extraction from the upper layer low-frequency subband.
  • the correct noise component that needs to be extracted from the low-frequency subband side in the multiple-resolution representations can be extracted, and thus, a noise component without significant residual noise is extracted.
  • the noise removal effect achieved on the high-frequency subband side can be used to successfully draw out the residual noise component manifesting as streaks or checkered patterns in the low-frequency side subband.
  • the extracted noise components are modified into noise components to be used for the actual noise removal.
  • the noise component frequency characteristics are adjusted by selecting different setting for the weight applied to the low-frequency subband (LL) and the weight applied to the high-frequency subbands (LH, HL, HH).
  • the noise component frequency characteristics are adjusted based upon a principal identical to that of the first embodiment and the parameter should be selected for the processing in the second embodiment in much the same way as in the first embodiment.
  • processing is executed as expressed below and the individual processing procedures correspond to processing ( 10 - 5 ), processing ( 11 - 5 ), processing ( 12 - 5 ), processing ( 13 - 5 ), processing ( 14 - 5 ) and processing ( 15 - 5 ) in FIG. 10 .
  • n 0′′( LL 0) k 0(0)* n 0( LL 0) (50)
  • n 1′′( LL 1) k 0(1)* n 1( LL 1) (51)
  • n 2′′( LL 2) k 0(2)* n 2( LL 2)
  • n 3′′( LL 3) k 0(3)* n 3( LL 3)
  • n 4′′( LL 4) k 0(4)* n 4( LL 4)
  • n 5′′( LL 5) k 0(5)* n 5( LL 5) (55)
  • n 1 ′′(LL 1 ) and n 1 (LH 1 , HL 1 , HH 1 ) are directly bundled together into n 1 ′′(LL 1 , LH 1 , HL 1 , HH 1 ).
  • n 2 ′′(LL 2 ) and n 2 (LH 2 , HL 2 , HH 2 ) are directly bundled together into n 2 ′′(LL 2 , LH 2 , HL 2 , HH 2 ).
  • n 3 ′′(LL 3 ) and n 3 (LH 3 , HL 3 , HH 3 ) are directly bundled together into n 3 ′′(LL 3 , LH 3 , HL 3 , HH 3 ).
  • n 4 ′′(LL 4 ) and n 4 (LH 4 , HL 4 , HH 4 ) are directly bundled together into n 4 ′′(LL 4 , LH 4 , HL 4 , HH 4 ).
  • n 5 ′′(LL 5 ) and n 5 (LH 5 , HL 5 , HH 5 ) are directly bundled together into n 5 ′′(LL 5 , LH 5 , HL 5 , HH 5 ).
  • the modified noise components are then synthesized through inverse wavelet transformation executed in sequence from the lowest resolution side.
  • the synthesized noise component is to be used for the actual noise removal.
  • n 4′′′( LL 4) n 4′′( LL 4)+ N 5′( LL 4) (56) n 4 ′′′(LL 4 ) and n 4 ′′(LH 4 , HL 4 , HH 4 ) are directly bundled together into n 4 ′′′(LL 4 , LH 4 , HL 4 , HH 4 ).
  • the noise component corresponding to the LL 4 plane is generated by combining the noise components over the two layers, as indicated in FIG. 10 . It is to be noted, however, that the noise components for LH 4 , HL 4 and HH 4 correspond to a single layer.
  • This processing is executed in much the same way as the “processing executed at 1/16 resolution” described in (2-3-4-2) above.
  • This processing is executed in much the same way as the “processing executed at 1/16 resolution” described in (2-3-4-2) above.
  • n 1′′′( LL 1) n 1′′( LL 1)+ N 2′( LL 1) (57) n 1 ′′′(LL 1 ) and n 1 ′′(LH 1 , HL 1 , HH 1 ) are directly bundled together into n 1 ′′′(LL 1 , LH 1 , HL 1 , HH 1 ).
  • n 0′′′( LL 0) n 0′′( LL 0)+ N 1′( LL 0) (58)
  • a noteworthy feature of this processing is that two separate systems of noise synthesizing means are provided to generate two different types of noise components, one for noise extraction and the other for actual noise removal, through synthesis. As a result, the optimization processing executed to adjust the noise component intensity characteristics or adjust the noise component frequency characteristics in correspondence to a given application is facilitated.
  • the noise component for the low-frequency subband is integrated through the noise synthesis processing by using noise components obtained from two different layers, i.e., the noise component made up with the noise components from both the low-frequency subband and the high-frequency subbands on the low resolution side and the noise component obtained from the low-frequency subband at the target resolution itself, which clearly differentiates the present invention from the related art. Consequently, the noise frequency characteristics adjustment is facilitated and noise components optimized for the individual purposes of use of the two separate systems can be combined.
  • the processing is executed in much the same way as “(2-3-4) Actual noise removal processing” in the first embodiment.
  • the processing is executed in much the same way as “(2-4) noise removal from the chrominance component (a ⁇ )” in the first embodiment.
  • the expressions used in the processing are defined slightly differently, and accordingly, they need to be modified as indicated below.
  • n 1 (LL 1 ) and n 1 ′′(LH 1 , HL 1 , HH 1 ) are directly bundled together into n 1 ′′(LL 1 , LH 1 , HL 1 , HH 1 ).
  • n 2 (LL 2 ) and n 2 ′′(LH 2 , HL 2 , HH 2 ) are directly bundled together into n 2 ′′(LL 2 , LH 2 , HL 2 , HH 2 ).
  • n 3 (LL 3 ) and n 3 ′′(LH 3 , HL 3 , HH 3 ) are directly bundled together into n 3 ′′(LL 3 , LH 3 , HL 3 , HH 3 ).
  • n 4 (LL 4 ) and n 4 ′′(LH 4 , HL 4 , HH 4 ) are directly bundled together into n 4 ′′(LL 4 , LH 4 , HL 4 , HH 4 ).
  • n 5 (LL 5 ) and n 5 ′′(LH 5 , HL 5 , HH 5 ) are directly bundled together into n 5 ′′(LL 5 , LH 5 , HL 5 , HH 5 ).
  • the processing is executed in much the same way as “(2-4) noise removal from the chrominance component (a ⁇ )”.
  • two types of processing equivalent to two different types of noise removal, are executed each for purposes of noise extraction or noise removal, and the noise removal results of the noise removal from the lower layer high-frequency subbands as well as the results of the noise removal from the lower layer low-frequency subband are allowed to affect the execution of the noise extraction from the low-frequency subband in the upper layer.
  • noise is extracted sequentially from both the high-frequency subbands and the low-frequency subband constituting images obtained through multiple resolution transformation while allowing the processing of either type of subband to be influenced by the characteristics of the other type of subband, as in the first embodiment. Consequently, the level of freedom with which different frequency bands can be combined in the processing increases and the noise removal can be executed in the frequency spaces optimized for the noise extraction. This means that fine noise removal is enabled without loss of image structure while minimizing the residual noise that is not extracted.
  • fine noise removal processing edge-preserving smoothing assuring a high level of image structure preservation while effectively addressing the issue of residual noise in a regular image such as a digital picture, is realized through the embodiment.
  • the difference between the first embodiment and the second embodiment is briefly explained. It has been confirmed through testing that by adjusting the parameter settings, substantially equal levels of noise removal effect and countermeasures against residual noise can be achieved through the sequential analysis method and the sequential synthesis method. A noteworthy difference between them may be the different orders with which the individual processing phases are executed. Namely, the “sequential analysis” method, in which the low-resolution side is executed at a later stage, is more effective in preventing leakage of the noise of a long cycle component extracted on the low resolution side that is bound to affect the processing executed at another resolution.
  • the “sequential synthesis” method in which the high-resolution side is processed at a later stage, is more effective in preventing leakage of the noise extracted on the high resolution side and is thus more effective when adopted to extract persistent noise with a Nyquist frequency, such as a checkered pattern.
  • a specific type of noise removal processing is executed on the image signal S 0 (LL 0 ) in the real space (see FIG. 10 ).
  • a load of the processing executed on the image signal S 0 (LL 0 ) in the real space is bound to be extremely heavy.
  • FIG. 10 a specific type of noise removal processing executed on the image signal S 0 (LL 0 ) in the real space
  • FIG. 17 presents a flowchart of the processing executed on the luminance component (luminance signal) by skipping the specific noise removal processing on the image signal S 0 (LL 0 ) in the real space in FIG. 10 .
  • the chrominance components may be processed in a similar manner. Through these measures, fine noise removal processing can be executed with a lighter processing load.
  • noise removal processing examples have been described in reference to the first and second embodiments.
  • the present invention is adopted in edge emphasis processing, instead of in noise removal processing, so as to facilitate adjustment of the frequency characteristics at multiple resolutions.
  • FIG. 12 presents a flowchart of the edge emphasis processing executed in conjunction with multiple resolution transformation.
  • the processing differs from that in the preceding embodiments only in that it does not include the sequential processing feedback routine executed in the noise removal and that edge component extraction processing, instead of the noise component extraction processing, is executed.
  • the edge component extraction processing may be executed through, for instance, unsharp mask processing or band pass filter processing executed on the individual subband planes.
  • the edge component extraction processing may be executed by using the multiple-resolution images obtained through transformation for noise component extraction as has been explained in reference to the first embodiment and the second embodiments, concurrently while the noise component extraction processing is executed, or the edge component extraction processing may be executed on an image having undergone the noise removal processing in the first embodiment or the second embodiment.
  • the edge component extraction processing alone may be executed as long as the target image simply requires edge emphasis. It is to be noted that the edge component extraction processing should, in principle, be executed on the luminance plane alone.
  • the edge emphasis processing is executed by itself, it is more desirable from the viewpoint of image quality, to execute the noise removal and the edge emphasis at the same time. More specifically, it is desirable to synthesize edge components extracted from subband planes having undergone the virtual intense noise removal to achieve a noise-free state, as has been explained in reference to the first and second embodiments, so as to ensure that the extracted edge components do not contain any noise component and then to incorporate through addition processing the synthesized edge component to the image having undergone the actual noise removal. Accordingly, if the edge emphasis is executed in conjunction with the processing in, for instance, the second embodiment, the synthesis processing on the right hand side in FIG. 10 will include three processing phases, 1) noise component synthesis for virtual noise removal, 2) noise component synthesis for actual noise removal and 3) edge component synthesis for actual edge emphasis.
  • While the present invention adopted in noise removal processing executed in conjunction with multiple resolution transformation, facilitates adjustment of the noise component frequency characteristics and noise component removal intensity so that the user is able to easily check how the target image is visually altered through the noise removal. It also provides a system that allows the edge component frequency characteristics and intensity to be freely adjusted so that the user is able to easily check how the target image is visually affected through the edge emphasis when it is adopted in edge emphasis processing executed in conjunction with multiple resolution transformation.
  • the frequency characteristics of the extracted edge components are altered by adjusting the weights applied to the low-frequency subband (LL) and the high-frequency subbands (LH, HL, HH).
  • FIG. 13 shows the weighting coefficients applied to the low-frequency subband (LL) and the high-frequency subbands (LH, HL, HH).
  • k1 may assume different values in correspondence to LH and HL.
  • the term “low-frequency subband” is used in this context to refer to a low-frequency edge component image, whereas the term “high-frequency subbands” is used in this context to refer to a high-frequency edge component image.
  • the low-frequency edge component image and the high-frequency edge component images with the weights applied thereto modulated in correspondence to the different edge component frequency bands are utilized in inverse wavelet transformation.
  • the inverse wavelet transformation (synthesis) is executed repeatedly in sequence by using the low-frequency edge component image and the high-frequency edge component images with the weights applied thereto modulated at each resolution until a single edge component image, the resolution of which matches that of the original image, is obtained. Finally, based upon the synthesized edge component, the edges in the original image are emphasized.
  • edge components are extracted from both the high-frequency subbands and the low-frequency subband at each specific level of resolution among the multiple-resolutions and the edge components having been extracted are synthesized by applying specific weighting coefficients to the different subbands.
  • the entire edge component frequency band is covered and an environment in which the frequency characteristics can be adjusted with ease and the visual effect of the edge emphasis can be adjusted is provided.
  • a specific type of edge component extraction processing is executed on the image signal S 0 (LL 0 ) in the real space (see FIG. 10 ).
  • a load of the processing executed on the image signal S 0 (LL 0 ) in the real space is bound to be extremely heavy.
  • Laplacian pyramids may be utilized in the multiple resolution transformation. Individual Gaussian pyramids generated during the process of Laplacian pyramid generation correspond to the low-frequency subband (LL) generated through the wavelet transformation, whereas individual Laplacian pyramids correspond to the high-frequency subbands (LH, HL, HH) generated through the wavelet transformation.
  • LL low-frequency subband
  • LH, HL, HH high-frequency subbands
  • the resolution of the Laplacian bands i.e., the high-frequency subbands, corresponding to a specific Gaussian band, i.e., the low-frequency subband, is higher by one step relative to the resolution of the Gaussian band.
  • a Steerable pyramid (Steerable wavelet transformation, directional wavelet transformation) representation may be adopted in the multiple resolution transformation.
  • the Gaussian band generated through Laplacian pyramid generation directly corresponds to the low-frequency subband.
  • a single type of isotropic high pass component is generated as a Laplacian band corresponding to a high-frequency subband in the Laplacian pyramid representation
  • a plurality of Laplacian bands with anisotropic high pass components taken along a plurality of directions correspond to high-frequency subbands in the Steerable pyramid representation.
  • FIG. 14 presents a schematic diagram indicating the correspondence between the low-frequency subband and the high-frequency subbands in each type of multiple-resolution representations among the orthogonal wavelet transformation, the Laplacian pyramid and the Steerable pyramid.
  • noise is removed from both the luminance component and the chrominance components through the “sequential analysis” method
  • noise is removed from both the luminance component and the chrominance components through the “sequential synthesis” method.
  • noise may be removed from the luminance component through the sequential analysis and from the chrominance components through the sequential synthesis.
  • noise may be removed from the luminance component through the sequential synthesis and from the chrominance components through the sequential analysis.
  • the band-limited images in each band resulting from the wavelet transformation individually undergo virtual noise removable processing
  • noise components are extracted from the band-limited images having undergone the virtual noise removable processing
  • the extracted noise components are synthesized through inverse wavelet transformation
  • the actual noise-free image is generated by subtracting the synthesized noise components from the image in the real space.
  • virtual noise removal processing may be executed based upon the individual band-limited images in each band, noise components may be extracted from the band-limited images having undergone the virtual noise removal processing, actual noise-free images may be generated by using the extracted noise components, the noise-free images thus generated may be synthesized through inverse wavelet transformation and the synthesized noise-free image may be output as the actual noise-free image.
  • FIG. 20 presents a flowchart of such noise removal processing executed on the luminance component (luminance signal).
  • FIG. 20 corresponds to FIG. 3 in reference to which the first embodiment has been described.
  • the processing executed at, for instance, the 1 ⁇ 2 resolution is identical to that executed in the first embodiment up to the point at which the noise components are extracted in processing ( 1 - 5 ).
  • actual noise-free image signals S 1 ′′(LL 1 ′′, LH 1 ′′, HL 1 ′′, HH 1 ′′) are generated by subtracting noise components n 1 ′ obtained by applying specific weights k0:1:1:1 to the extracted noise components n 1 from the image signals S 1 (LL 1 , LH 1 , HL 1 , HH 1 ).
  • noise removal is executed after applying a weighting coefficient parameter ⁇ , i.e., the noise removal factor, to the extracted noise components n 1 ′ so as to enable adjustment of the extent of noise removal for the entire image.
  • the weighting coefficient parameter ⁇ is similar to the weighting coefficient parameter ⁇ used in the final noise removal processing executed in the real space in the first embodiment.
  • the present invention is not limited to this example.
  • the present invention may be adopted in processing executed in an image-capturing device such as a camera, or it may be adopted in another type apparatus.
  • the present invention can be adopted in all types of apparatus that handle image data.
  • the present invention is not limited to this example.
  • two different types of noise-free subbands images may be generated to be individually synthesized through two separate systems as disclosed in patent references 5 (U.S. Pat. No. 5,526,446) and 9 (U.S. Pat. No. 6,937,772) in the related art.
  • the noise removal processing may be executed by using another type of noise removal filter.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Picture Signal Circuits (AREA)
  • Facsimile Image Signal Circuits (AREA)
US12/230,161 2006-03-31 2008-08-25 Image processing method Active 2030-01-04 US8238683B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2006-096986 2006-03-31
JP2006096986 2006-03-31
PCT/JP2007/057186 WO2007114363A1 (ja) 2006-03-31 2007-03-30 画像処理方法

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2007/057186 Continuation WO2007114363A1 (ja) 2006-03-31 2007-03-30 画像処理方法

Publications (2)

Publication Number Publication Date
US20090046943A1 US20090046943A1 (en) 2009-02-19
US8238683B2 true US8238683B2 (en) 2012-08-07

Family

ID=38563631

Family Applications (1)

Application Number Title Priority Date Filing Date
US12/230,161 Active 2030-01-04 US8238683B2 (en) 2006-03-31 2008-08-25 Image processing method

Country Status (4)

Country Link
US (1) US8238683B2 (ja)
EP (1) EP2003877B1 (ja)
JP (2) JP5163489B2 (ja)
WO (1) WO2007114363A1 (ja)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110317045A1 (en) * 2007-05-25 2011-12-29 Zoran Corporation Advanced noise reduction in digital cameras
US11113796B2 (en) * 2018-02-09 2021-09-07 Delta Electronics, Inc. Image enhancement circuit and method thereof
US11288773B1 (en) 2019-04-23 2022-03-29 Alarm.Com Incorporated Multi-resolution image or video

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE602006021728D1 (de) * 2005-03-31 2011-06-16 Nippon Kogaku Kk Bildverarbeitungsverfahren
JP4862897B2 (ja) 2007-02-02 2012-01-25 株式会社ニコン 画像処理方法
JP4766118B2 (ja) 2007-02-02 2011-09-07 株式会社ニコン 画像処理方法
WO2008105268A1 (ja) * 2007-02-28 2008-09-04 Nikon Corporation 画像処理方法
WO2009093386A1 (ja) * 2008-01-21 2009-07-30 Olympus Corporation 画像処理装置、画像処理プログラム、画像処理プログラムを記憶したコンピュータ読取り可能な記憶媒体、及び画像処理方法
US8120679B2 (en) 2008-08-01 2012-02-21 Nikon Corporation Image processing method
JP5083138B2 (ja) * 2008-09-12 2012-11-28 株式会社ニコン 画像分類装置
US8374457B1 (en) 2008-12-08 2013-02-12 Adobe Systems Incorporated System and method for interactive image-noise separation
JP2011015277A (ja) * 2009-07-03 2011-01-20 Olympus Corp 画像処理装置、画像処理方法、画像処理プログラムおよび画像処理プログラムが記録された記録媒体
US9792012B2 (en) * 2009-10-01 2017-10-17 Mobile Imaging In Sweden Ab Method relating to digital images
SE534551C2 (sv) 2010-02-15 2011-10-04 Scalado Ab Digital bildmanipulation innefattande identifiering av ett målområde i en målbild och sömlös ersättning av bildinformation utifrån en källbild
JP4991907B2 (ja) * 2010-05-11 2012-08-08 キヤノン株式会社 画像処理装置、および、画像処理装置の制御方法
SE1150505A1 (sv) 2011-05-31 2012-12-01 Mobile Imaging In Sweden Ab Metod och anordning för tagning av bilder
EP2718896A4 (en) 2011-07-15 2015-07-01 Mobile Imaging In Sweden Ab METHOD FOR PROVIDING ADJUSTED DIGITAL GRAPHIC REPRESENTATION OF VIEW AND APPROPRIATE APPARATUS
CN103139448B (zh) * 2011-11-22 2016-02-03 华晶科技股份有限公司 影像处理装置及其处理方法
KR20140099319A (ko) * 2011-12-04 2014-08-11 디지털 메이크업 엘티디 디지털 메이크업
JP5385487B1 (ja) * 2012-02-29 2014-01-08 独立行政法人科学技術振興機構 スーパーハイブリッド画像生成装置、スーパーハイブリッド画像生成方法、印刷媒体製造方法、電子媒体製造方法、および、プログラム
WO2014088049A1 (en) * 2012-12-07 2014-06-12 Canon Kabushiki Kaisha Image generating apparatus and image generating method
US9542617B2 (en) 2013-02-28 2017-01-10 Nec Corporation Image processing device and image processing method for correcting a pixel using a corrected pixel statistical value
JP5668105B2 (ja) * 2013-06-25 2015-02-12 アキュートロジック株式会社 画像処理装置、画像処理方法及び画像処理プログラム
JP2015070451A (ja) * 2013-09-30 2015-04-13 株式会社 日立産業制御ソリューションズ 撮像装置
JP6365662B2 (ja) * 2014-04-01 2018-08-01 株式会社ニコン 超解像観察装置及び超解像観察方法
CN105794195B (zh) * 2014-10-27 2020-02-07 松下知识产权经营株式会社 图像形成系统、图像形成方法以及拍摄元件
KR102251440B1 (ko) * 2014-11-05 2021-05-14 삼성전자주식회사 로컬 톤 맵핑 회로와 이를 포함하는 모바일 컴퓨팅 장치
US10475215B2 (en) * 2016-09-23 2019-11-12 Carestream Health, Inc. CBCT image processing method
KR102282455B1 (ko) * 2017-07-11 2021-07-28 한화테크윈 주식회사 영상 처리 장치 및 영상 처리 방법
WO2019079398A1 (en) 2017-10-18 2019-04-25 Gopro, Inc. CHROMINANCE DENGING
US11132448B2 (en) * 2018-08-01 2021-09-28 Dell Products L.P. Encryption using wavelet transformation
CN114565540B (zh) * 2022-04-30 2022-07-22 深圳市巨力方视觉技术有限公司 基于多路对照图像去噪用机器视觉集成系统

Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5526446A (en) 1991-09-24 1996-06-11 Massachusetts Institute Of Technology Noise reduction system
US5708693A (en) * 1995-06-23 1998-01-13 U.S. Philips Corporation Image processing for noise reduction
US5963676A (en) 1997-02-07 1999-10-05 Siemens Corporate Research, Inc. Multiscale adaptive system for enhancement of an image in X-ray angiography
JP2000224421A (ja) 1999-02-03 2000-08-11 Fuji Photo Film Co Ltd 画像処理方法および装置並びに記録媒体
JP2001167264A (ja) 1999-09-30 2001-06-22 Fuji Photo Film Co Ltd 画像処理方法および装置並びに記録媒体
JP2001211327A (ja) 1999-12-10 2001-08-03 Ricoh Co Ltd データ処理方法及び装置、複写機、記録媒体
US20020071600A1 (en) * 2000-10-17 2002-06-13 Masahiko Yamada Apparatus for suppressing noise by adapting filter characteristics to input image signal based on characteristics of input image signal
US20020196907A1 (en) * 2001-06-19 2002-12-26 Hiroyuki Shinbata Image processing apparatus, image processing system, image processing method, program, and storage medium
US6754398B1 (en) 1999-06-10 2004-06-22 Fuji Photo Film Co., Ltd. Method of and system for image processing and recording medium for carrying out the method
US6766062B1 (en) 2000-02-16 2004-07-20 The Board Of Trustees Of The Leland Stanford Junior University - Office Of Technology Digital ridgelet transform via digital polar coordinate transform
US6781625B2 (en) 2002-03-15 2004-08-24 Fuji Photo Film Co., Ltd. Noise reducing apparatus
JP2004242285A (ja) 2003-01-14 2004-08-26 Fuji Photo Film Co Ltd ノイズ抑制処理方法および装置並びにプログラム
US20040252907A1 (en) * 2001-10-26 2004-12-16 Tsukasa Ito Image processing method, apparatus, and program
US6937772B2 (en) 2000-12-20 2005-08-30 Eastman Kodak Company Multiresolution based method for removing noise from digital images
US20060029287A1 (en) * 2004-08-03 2006-02-09 Fuji Photo Film Co., Ltd. Noise reduction apparatus and method
WO2006064913A1 (ja) 2004-12-17 2006-06-22 Nikon Corporation 画像処理方法
WO2006068025A1 (ja) 2004-12-20 2006-06-29 Nikon Corporation 画像処理方法
WO2006106919A1 (ja) 2005-03-31 2006-10-12 Nikon Corporation 画像処理方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3816151B2 (ja) * 1995-09-29 2006-08-30 富士写真フイルム株式会社 画像処理方法および装置
JP4141712B2 (ja) * 2002-03-20 2008-08-27 株式会社リコー 画像処理装置
JP2003281529A (ja) * 2002-03-20 2003-10-03 Ricoh Co Ltd 画像処理装置および画像処理方法
JP4064168B2 (ja) * 2002-06-25 2008-03-19 富士フイルム株式会社 画像処理装置
JP4337386B2 (ja) * 2003-04-23 2009-09-30 コニカミノルタフォトイメージング株式会社 画像処理方法、画像処理装置、画像処理プログラム及び画像記録装置
JP2005021456A (ja) * 2003-07-03 2005-01-27 Fuji Photo Film Co Ltd 放射線画像用画像処理装置、方法およびプログラム
JP2005052295A (ja) * 2003-08-01 2005-03-03 Fuji Photo Film Co Ltd 画像処理装置およびプログラム
US7018560B2 (en) 2003-08-05 2006-03-28 Rohm And Haas Electronic Materials Cmp Holdings, Inc. Composition for polishing semiconductor layers
JP2005151122A (ja) * 2003-11-14 2005-06-09 Canon Inc 画像処理装置及び画像処理方法
JP4632685B2 (ja) * 2004-04-12 2011-02-16 株式会社東芝 超音波診断装置及び画像データ処理装置
JP5015436B2 (ja) 2004-08-30 2012-08-29 東レ・ダウコーニング株式会社 熱伝導性シリコーンエラストマー、熱伝導媒体および熱伝導性シリコーンエラストマー組成物

Patent Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5526446A (en) 1991-09-24 1996-06-11 Massachusetts Institute Of Technology Noise reduction system
US5708693A (en) * 1995-06-23 1998-01-13 U.S. Philips Corporation Image processing for noise reduction
US5963676A (en) 1997-02-07 1999-10-05 Siemens Corporate Research, Inc. Multiscale adaptive system for enhancement of an image in X-ray angiography
JP2000224421A (ja) 1999-02-03 2000-08-11 Fuji Photo Film Co Ltd 画像処理方法および装置並びに記録媒体
US6754398B1 (en) 1999-06-10 2004-06-22 Fuji Photo Film Co., Ltd. Method of and system for image processing and recording medium for carrying out the method
JP2001167264A (ja) 1999-09-30 2001-06-22 Fuji Photo Film Co Ltd 画像処理方法および装置並びに記録媒体
JP2001211327A (ja) 1999-12-10 2001-08-03 Ricoh Co Ltd データ処理方法及び装置、複写機、記録媒体
US6766062B1 (en) 2000-02-16 2004-07-20 The Board Of Trustees Of The Leland Stanford Junior University - Office Of Technology Digital ridgelet transform via digital polar coordinate transform
US20020071600A1 (en) * 2000-10-17 2002-06-13 Masahiko Yamada Apparatus for suppressing noise by adapting filter characteristics to input image signal based on characteristics of input image signal
US6937772B2 (en) 2000-12-20 2005-08-30 Eastman Kodak Company Multiresolution based method for removing noise from digital images
US20020196907A1 (en) * 2001-06-19 2002-12-26 Hiroyuki Shinbata Image processing apparatus, image processing system, image processing method, program, and storage medium
US20040252907A1 (en) * 2001-10-26 2004-12-16 Tsukasa Ito Image processing method, apparatus, and program
US6781625B2 (en) 2002-03-15 2004-08-24 Fuji Photo Film Co., Ltd. Noise reducing apparatus
JP2004242285A (ja) 2003-01-14 2004-08-26 Fuji Photo Film Co Ltd ノイズ抑制処理方法および装置並びにプログラム
US20040258325A1 (en) 2003-01-14 2004-12-23 Fuji Photo Film Co., Ltd. Noise suppression processing method, apparatus and program
US20060029287A1 (en) * 2004-08-03 2006-02-09 Fuji Photo Film Co., Ltd. Noise reduction apparatus and method
WO2006064913A1 (ja) 2004-12-17 2006-06-22 Nikon Corporation 画像処理方法
WO2006068025A1 (ja) 2004-12-20 2006-06-29 Nikon Corporation 画像処理方法
WO2006106919A1 (ja) 2005-03-31 2006-10-12 Nikon Corporation 画像処理方法

Non-Patent Citations (9)

* Cited by examiner, † Cited by third party
Title
Aydin et al.; "Multidirectional and Multiscale Edge Detection via M-Band Wavelet Transform;" IEEE Transactions on Image Processing; Sep. 1996; pp. 1370-1377; vol. 5, No. 9.
Burt et al., "The Laplacian Pyramid as a Compact Image Code," IEEE Transactions on Communications, vol. 31, No. 4, Apr. 1983, pp. 532-540.
European Office Action dated May 2, 2011 in corresponding European Application No. 07 740 622.1.
Freeman et al., "The Design and Use of Steerable Filters," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, No. 9, Sep. 1991, pp. 891-906.
Laine et al.; "Wavelets for Contrast Enhancement of Digital Mammography;" IEEE Engineering in Medicine and Biology; Sep./Oct. 1995; pp. 536-550; vol. 14, No. 5.
Lee, "Digital Image Smoothing and the Sigma Filter," Computer Vision, Graphics and Image Processing, vol. 24, 1983, pp. 255-269.
Ranganath, "Image Filtering Using Multiresolution Representations," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, No. 5, May 1991, pp. 426-440.
Supplementary European Search Report dated Jul. 28, 2010 in European Patent Application No. 07 74 0622.
Tomasi et al., "Bilateral Filtering for Gray and Color Images," Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay, India.

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110317045A1 (en) * 2007-05-25 2011-12-29 Zoran Corporation Advanced noise reduction in digital cameras
US8824831B2 (en) * 2007-05-25 2014-09-02 Qualcomm Technologies, Inc. Advanced noise reduction in digital cameras
US9148593B2 (en) 2007-05-25 2015-09-29 Qualcomm Technologies, Inc. Advanced noise reduction in digital cameras
US11113796B2 (en) * 2018-02-09 2021-09-07 Delta Electronics, Inc. Image enhancement circuit and method thereof
US11288773B1 (en) 2019-04-23 2022-03-29 Alarm.Com Incorporated Multi-resolution image or video

Also Published As

Publication number Publication date
EP2003877A1 (en) 2008-12-17
WO2007114363A1 (ja) 2007-10-11
JP2013030191A (ja) 2013-02-07
EP2003877B1 (en) 2015-05-06
JPWO2007114363A1 (ja) 2009-08-20
US20090046943A1 (en) 2009-02-19
EP2003877A4 (en) 2010-09-01
JP5163489B2 (ja) 2013-03-13
JP5541339B2 (ja) 2014-07-09

Similar Documents

Publication Publication Date Title
US8238683B2 (en) Image processing method
US8244034B2 (en) Image processing method
EP1865460B1 (en) Image processing method
US8369640B2 (en) Image processing for removing a noise component contained in an image
JP5023702B2 (ja) 画像処理方法、画像処理プログラム、記録媒体及び画像処理装置
JP5352942B2 (ja) 画像処理方法、画像処理プログラム、および画像処理装置
US7305144B2 (en) System and method for compressing the dynamic range of an image
US7181086B2 (en) Multiresolution method of spatially filtering a digital image
US8120679B2 (en) Image processing method
US8860856B2 (en) Multiplexed imaging
CN102037491A (zh) 提高图像锐度的图像处理
EP2851866A1 (en) Image processing device, image processing method, program, print medium, and recording medium
JP5256582B2 (ja) 画像処理方法、画像処理プログラム、および画像処理装置
US9600713B2 (en) Identification and processing of facial wrinkles in a digital image
CN114638761A (zh) 一种高光谱图像全色锐化方法、设备及介质
Ghous et al. Efficient image enhancement using improved RIQMC based ROHIM model
JP2004086539A (ja) 画像分割方法および装置並びにプログラム
Zhang et al. Supplementary Information for: Luminosity rectified blind Richardson-Lucy deconvolution for single retinal image restoration
Velde et al. Multi-Resolution Enhancement of Photographic Images

Legal Events

Date Code Title Description
AS Assignment

Owner name: NIKON CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ISHIGA, KENICHI;REEL/FRAME:021672/0077

Effective date: 20080926

STCF Information on status: patent grant

Free format text: PATENTED CASE

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 4

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 12