JP2010034713A - Photographic image processing method, photographic image processing program and photographic image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a photographic image processing method, a photographic image processing program and a photographic image processing apparatus which effectively suppress granular noise by setting a smoothed area corresponding to the granular noise in photographic image data. <P>SOLUTION: The photographic image processing method for suppressing granular noise included in photographic image data includes: a pixel component extraction step of extracting a pixel component of a noticed pixel in the photographic image data; and a noise suppression processing step of generating an energy function which includes a data term indicating a difference between an original pixel component which is the pixel component of the noticed pixel and a pixel value of a noise suppressed pixel component which is a pixel component of an assumed pixel corresponding to a noticed pixel of noise suppressed image data and a smoothing term obtained by multiplying dispersion in pixel values of the noise suppressed pixel component by a spatially uneven weight coefficient, and which is obtained by multiplying the data term and the smoothing term by respective coefficients and adding the multiplied values together, and calculating a noise suppressed pixel component indicating a minimum energy function by a graph cut method. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a photographic image processing method, a photographic image processing program, and a photographic image processing apparatus that suppress granular noise included in photographic image data.

  When an image taken with a digital camera is observed, it is observed that fine granular noise is superimposed on the image. In particular, noise that is rough in the dark part of the image is conspicuous, and similar noise appears in the entire image in an image shot in a dark place without flash emission.

  These noises are caused by the characteristics of the elements such as the reset noise and dark current noise of the image sensor.The imaging environment is worse, that is, the lower the illuminance of the subject, the better the noise component. Noise is enhanced when the brightness is corrected.

  Similarly, a film image taken by an optical camera also has grainy irregularities based on the size of the pigment cloud, and fine grained noise is superimposed on the photographic image data converted into a digital image through a film scanner. ing. These granular noises are noticeable when a photographic image is enlarged.

  The present inventor has developed a three-dimensional space in which pixel values of an image shot by automatic exposure (AE) and an image shot in a state where the shutter speed is increased and the gain is increased are used with RGB color channels as axes. To know how pixel values of an image including granular noise are distributed in the RGB color space.

  According to the above knowledge, images taken with automatic exposure are linearly distributed in color, but in images taken with increased gain, the color is applied to each RGB channel due to the occurrence of granular noise. It has been confirmed that the width varies with a certain width and the width of the line increases. In addition, since the bulge of this line is isotropic, it is considered that granular noise is generated independently for each color.

  In Non-Patent Document 1, as a result of examining eigenvalues and eigenvectors of the RGB values in the divided areas, it is shown that the first eigenvalue is significantly larger than the other two eigenvalues. Furthermore, since the image obtained by projecting all pixel values to the first eigenvalue while ignoring the other two eigenvalues is almost restored to the original image, the RGB values in each region are distributed linearly. It has also been confirmed that there is a limit to the width of expansion. Furthermore, it has been confirmed that almost no granular noise is observed in parts such as windows and fluorescent lamps that emit strong light.

  From the above, it was confirmed that granular noise has the following properties. 1) Independent noise independent of RGB color channels (color independence). 2) The magnitude of the granular noise cannot be said to be independent of the luminance value (luminance value dependency). 3) In order to observe granular noise, it is necessary to analyze the image locally (locality).

  Conventional methods for suppressing such granular noise include “Edge Preserving Smoothing”, “Bilateral Filter”, “Graph Cuts”, and noise estimation by region segmentation. A smoothing method such as removal (Non-Patent Document 1) is used.

  “Edge preserving smoothing” is a technique for suppressing noise while preserving edge information using a set of pixels with small variance in a smoothing mask of a predetermined size. However, since smoothing processing in a spatially wide range is required to suppress granular noise, color irregularities remain in a normally used mask size, and a contour (hereinafter referred to as “edge”) portion is present. An excessively sharpened image results in an unnatural image.

  The “bilateral filter” is a technique of performing smoothing by weighted averaging while preserving edge information by increasing the load of pixels that are spatially close and also close to the pixel value. However, since granular noise is highly isolated both spatially and in terms of pixel values, the width of the filter must be widened by removing the noise with a “bilateral filter”. Due to the resulting excessive smoothing, even the texture of the subject is lost. Further, in an area where adjacent pixels are difficult to have many adjacent pixel values, fine irregularities appear in the area boundary part, and the edge part becomes conspicuous.

  On the other hand, “Graph Cuts”, which finds the optimal solution of problems formulated using Markov Random Field (MRF), has been attracting attention in recent years. Attempts have also been made to apply it to noise removal. This is a technique for obtaining an image satisfying the two conditions of “matching as much as a given pixel value as much as possible” and “dispersion of pixel values around each pixel as small as possible” by energy minimization.

This energy function is expressed as the sum of a “data term” that is an energy term representing a deviation from a given image and a “smoothing term” that is an energy term representing variation in pixel values. This method has an advantage that the degree of smoothing can be adjusted by the weight of each energy term, and fine granular noise can be suppressed while storing edge information. However, even when the weights are actually adjusted, color irregularities appear in flat areas, as in “Edge Preserving Smoothing” and “Bilateral Filter”, and each of them is subtly It turns out to have a different color.
C. Liu, R. Szeliski, SBKang, CLZitnick, and WTFreeman, “Automatic Estimation and Removal of Noise from a Single Image”, IEEE Transactions on PAMI, vol. 30, no. 2, pp. 299-314, 2008.

  What is common to the above-described conventional methods is “to apply spatially uniform processing to the entire image”. The fact that these methods do not give good results suggests that some spatial heterogeneity should be introduced. Note that, in Non-Patent Document 1, this succeeds in removing color noise by projecting pixel values in a region obtained by dividing an image into a predetermined size onto a linear subspace in the RGB space. It is justified from that.

  However, since the method shown in Non-Patent Document 1 uses a relatively large unit of “area”, it is procedural and complicated, and it is difficult to say that the method captures the nature of the problem.

  In view of the above-described conventional disadvantages, the present invention sets a smoothing area corresponding to granular noise to photographic image data, and can effectively suppress granular noise, a photographic image processing method, a photographic image processing program, And a photographic image processing apparatus.

  In order to achieve the above object, the first characteristic configuration of the photographic image processing method according to the present invention is to suppress the granular noise included in the photographic image data as described in claim 1 of the claims. A photographic image processing method for generating noise-suppressed image data, a pixel component extracting step for extracting a pixel component of a target pixel of the photographic image data, an original pixel component that is a pixel component of the target pixel, and the noise suppression A data term indicating a difference between pixel values of a noise suppression pixel component that is a pixel component of an assumed pixel corresponding to the target pixel of image data, and a spatially non-uniform weighting factor for variations in pixel values of the noise suppression pixel component A smoothing term obtained by multiplying each of the data term and the smoothing term by a coefficient, and generating an energy function, wherein the energy function exhibits a minimum value. In that it includes a noise suppression processing step of calculating a suppression pixel components by graph cut method, a.

  According to the above-described configuration, in the smoothing term, the variation in pixel value is multiplied by a spatially non-uniform weighting coefficient, so that a noise suppression pixel component in which the degree of smoothing substantially changes for each location is generated. Will be able to.

  In addition to the first feature configuration described above, the second feature configuration should be smoothed and not smoothed from the pixel component extracted in the pixel component extraction step, in addition to the first feature configuration described above. A smoothing processing step of identifying non-smoothed pixels and calculating smoothed mask image data based on the identification result, wherein the data term is a difference between pixel values of the noise suppression pixel component and the original pixel component; Is a product of a cumulative addition value for all the pixels of the noise suppression pixel component and a predetermined first coefficient, and the weighting factor is the smoothing corresponding to two adjacent pixels of the noise suppression pixel component The average value of the pixel values of the two pixels of the mask image data, and the smoothing term is a smoothing energy obtained by multiplying the difference between the pixel values of two adjacent pixels of the noise suppression pixel component by the weighting factor. And in that it is produced by the product of the cumulative added value and a predetermined second coefficient for every two adjacent pixels of the noise suppressing pixel components.

  According to the above configuration, the spatially non-uniform weighting factor included in the smoothing term is calculated from the smoothed mask image data that is a result of once identifying the smoothed pixel and the non-smoothed pixel. It becomes possible to suppress the conversion area from becoming too wide.

  In the third feature configuration, in addition to the second feature configuration described above, the pixel component extraction step extracts a luminance component and a color difference component from the photographic image data, and the noise The suppression processing unit calculates the noise suppression pixel component of the luminance component and the noise suppression pixel component of the color difference component, and from the noise suppression pixel component of the luminance component and the noise suppression pixel component of the color difference component, A pixel component conversion step of converting the noise suppression image data that is the noise suppression pixel component of the RGB component into the noise suppression image data is further included.

  According to the above-described configuration, smoothed mask image data is generated from the luminance component, so that the granular noise can be appropriately suppressed in consideration of the dependency of the luminance component that is a feature of the granular noise.

  In the fourth feature configuration, as described in claim 4, in addition to the third feature configuration described above, the smoothing processing step includes edge information obtained by filtering the luminance component using an edge extraction filter. The smoothing mask image data is calculated based on the above.

  According to the above configuration, non-smoothed pixels that should not be smoothed are identified based on the edge information obtained by the filtering process by the edge extraction filter, and smoothed mask image data is calculated based on the identification result. Therefore, the effect of storing the edge information is also reflected in the weighting coefficient calculated from the smoothed mask image data.

  The fifth feature configuration is that, in addition to the fourth feature configuration described above, the edge extraction filter is a Sobel filter.

  According to the above configuration, smooth edge extraction and noise suppression effects, which are characteristic effects of smoothing by the Sobel filter, are reflected in the smoothed mask image data, and thus calculated from the smoothed mask image data. The effect of smooth edge extraction and noise suppression is also reflected in the weighting factor.

  In the sixth feature configuration, as described in claim 6, in addition to the third feature configuration described above, the smoothing processing step uses a support vector machine to convert the RGB component of the photographic image data. From the covariance value matrix feature amount, the smoothed pixel and the non-smoothed pixel are identified for each pixel of the RGB component, the output result before threshold processing is normalized, and the smoothed mask image data is obtained. The point is to calculate.

  According to the above configuration, the local color distribution that is the characteristic of granular noise can be evaluated from the RGB component covariance value matrix feature amount, so it is possible to analyze how close to the granular noise is. It becomes. In addition, using a support vector machine (Support Vector Machine), which is one of the identification methods using supervised learning, by dealing with the problem of identifying the covariance value matrix feature of the RGB component, It is possible to calculate smoothed mask image data that is identified and granular noise is appropriately suppressed. Accordingly, the weighting coefficient calculated from the smoothed mask image data also reflects an appropriate suppression effect of granular noise.

  In the seventh feature configuration, in addition to any one of the third to sixth feature configurations described above, the noise suppression processing unit may include the noise suppression pixel component of the luminance component. The value of the second coefficient when calculating the color difference component is set to a value smaller than the value of the second coefficient when calculating the noise suppression pixel component of the color difference component.

  Human visual spatial resolution perception is more sensitive to luminance components than to color difference components. According to the above-described configuration, granular noise that is not visually affected is strongly suppressed by reducing the degree of smoothing for the luminance component, while increasing the degree of smoothing for the color difference component, and is visually affected. Noise-suppressed image data in which noise is moderately suppressed can be generated.

  The eighth characteristic configuration is a characteristic configuration of the photographic image processing program according to the present invention, and as described in claim 8, the noise-suppressed image data is generated by suppressing granular noise included in the photographic image data. A photographic image processing program, a pixel component extraction step for extracting a pixel component of a target pixel of the photographic image data, an original pixel component that is a pixel component of the target pixel, and the target pixel of the noise-suppressed image data A data term indicating a difference between pixel values of a noise suppression pixel component that is a pixel component of an assumed pixel to be calculated, and a smoothing term obtained by multiplying a variation in the pixel value of the noise suppression pixel component by a spatially nonuniform weighting factor. And generating an energy function obtained by multiplying each of the data term and the smoothing term by a coefficient, and adding the noise suppression pixel component indicating the minimum value of the energy function. There are a noise suppression processing step of calculating a rough cut method, to the point to be executed by a computer.

  The ninth characteristic configuration is a characteristic configuration of the photographic image processing apparatus according to the present invention. As described in the ninth aspect, the noise suppression image data is generated by suppressing the granular noise included in the photographic image data. A photographic image processing apparatus, a pixel component extraction unit that extracts a pixel component of a target pixel of the photographic image data, an original pixel component that is a pixel component of the target pixel, and the target pixel of the noise-suppressed image data A data term indicating a difference between pixel values of a noise suppression pixel component that is a pixel component of an assumed pixel to be calculated, and a smoothing term obtained by multiplying a variation in the pixel value of the noise suppression pixel component by a spatially nonuniform weighting factor. And generating an energy function obtained by multiplying each of the data term and the smoothing term by a coefficient, and adding the noise suppression pixel component whose energy function exhibits a minimum value by a graph cut method. In that it includes a noise suppression processing unit for calculating, a.

  As described above, according to the present invention, a photographic image processing method, a photographic image processing program capable of effectively suppressing granular noise by setting a smoothing region corresponding to granular noise to photographic image data, In addition, a photographic image processing apparatus can be provided.

  Embodiments of a photographic image processing method and a photographic image processing apparatus according to the present invention will be described below.

  As shown in FIG. 1, a photographic image processing apparatus 1 performs an exposure process based on output image data on a photographic paper P, develops the exposed photographic paper, and generates and outputs a photographic print. And an operation station 3 for setting and inputting print order information for the photographic image, performing various image correction processes, and outputting output image data edited from the original image to the photographic printer 2.

  The operation station 3 includes a film scanner 31 that reads an image from a developed photographic film F, and a media driver that reads image data from an image data storage medium M such as a memory card in which image data shot by a digital still camera or the like is stored. 32, a general-purpose computer as the controller 33, and the like.

  As shown in FIGS. 1 and 2, the photographic printer 2 has two systems of photographic paper magazines 21 containing roll-shaped photographic paper P, and the photographic paper P drawn from the photographic paper magazine 21 in a predetermined print size. A sheet cutter 22 for cutting, a back print unit 23 for printing print information such as a frame number on the back of the cut photographic paper P, an exposure unit 24 for exposing the photographic paper P based on the print data, and a post-exposure unit A development processing unit 25 including a plurality of processing tanks 25a, 25b, and 25c filled with processing solutions for developing, bleaching, and fixing the photographic paper P is disposed along the conveyance path of the photographic paper P, and development processing is performed. A lateral feed conveyor 26 that discharges the photographic paper P that has been dried later, and a sorter 27 that sorts a plurality of photographic papers (photo prints) P stacked on the lateral feed conveyor 26 in order units. To have.

  The exposure unit 24 outputs a three-color laser beam bundle of RGB that is modulated based on the print data in the main scanning direction orthogonal to the conveyance direction with respect to the photographic paper P conveyed in the sub-scanning direction by the conveyance mechanism 28. Then, an exposure head 24a for exposure is accommodated.

  A transport mechanism 28 composed of a plurality of roller pairs that transport the printing paper P at a predetermined process speed is disposed in the exposure unit 24 and the development processing unit 25 disposed along the transport path. A chucker-type transport mechanism 28a capable of transporting the paper P in multiple rows is provided.

  A controller 33 provided in the operation station 3 operates under the management of a general-purpose operating system, and is installed with a plurality of application programs for executing various image processing and input / output control of the photo processing apparatus 1. As an operation interface, a monitor 34, a keyboard 35, a mouse 36, and the like are connected. The application program includes an image processing program according to the present invention.

  The controller 33 is a block in which the hardware and software cooperate to execute a photo processing process, and will be described below in each functional block.

  As shown in FIG. 3, the controller 33 receives photographic image data as an original image read by the film scanner 31 or the media driver 32, performs a predetermined preprocessing, and transfers it to the memory 41. A print operation information and image editing information is displayed on the screen of the monitor 34, and a graphic operation screen for displaying operation icons for inputting necessary data is generated for the print operation information or a keyboard for the displayed graphic operation screen. 35, a graphic user interface unit 42 that generates various control commands based on input operations from the mouse 36, photographic image data transferred from the image input unit 40, photographic image data after correction processing by the image processing unit 47, Correction parameters at that time, and also set A memory 41 in which lint order information is partitioned and stored in a predetermined area, an order processing unit 43 that generates print order information, and each frame image or a predetermined number of frames for each photographic image data stored in the memory 41 An image processing unit 47 that performs density correction processing, contrast correction processing, and the like on the image is provided.

  Further, a display control unit 46 including a video RAM for displaying the image data developed in the memory 41 based on a display command from the graphic user interface unit 42 and various input / output graphic data on the monitor 34, and the like; A print data generation unit 44 that generates print data for outputting the final corrected image for which various correction processes have been completed to the photographic printer 2, and a storage medium such as a CD-R for the final corrected image according to the customer's order A formatter unit 45 for converting to a file format for writing to the file.

  The film scanner 31 is configured to operate in two modes: a pre-scan mode for reading an image recorded on the film F at a high speed although it is a low resolution and a main scan mode for reading at a high resolution although it is a low resolution. Various correction processes are performed on the low-resolution image read in the mode, and the final correction for the high-resolution image read in the main scan mode based on the correction parameters stored in the memory 41 at that time. The process is executed and output to the printer 2.

  Similarly, the image file read from the media driver 32 includes a high-resolution captured image and its thumbnail image, and various correction processes described later are performed on the thumbnail image and stored in the memory 41 at that time. Based on the correction parameters, the final correction processing for the high-resolution captured image is executed. When the thumbnail image is not included in the image file, the thumbnail image is generated from the high-resolution captured image by the image input unit 40 and transferred to the memory 41.

  As described above, various editing processes that are frequently trial and error are performed on low-resolution images, so that the calculation load of the controller 33 is reduced.

  The image processing unit 47 includes a distortion correction unit 50 that corrects distortion caused by the photographic lens with respect to photographic image data that is an original image stored in the memory 41, and a granular noise suppression processing unit 51 that suppresses granular noise. A sharpening processing unit 52 that enhances an edge of an image and suppresses noise, a color correction unit 53 that adjusts a color balance so as to reproduce a natural color, and an image size suitable for the size of a photographic print A plurality of image processing blocks such as the enlargement / reduction processing unit 54 are provided.

  The granular noise suppression processing unit 51 functions as an image processing apparatus according to the present invention, and the granular noise suppression processing unit 51 for film images suppresses granular noise included in image data read from the photographic film F in the main scan mode. In addition, a granular noise suppression processing unit 51 for a digital camera photographed image that suppresses granular noise included in high-resolution image data read from the media driver 32 is provided.

  Hereinafter, processing of photographic image data taken by a digital camera input from the media driver 32 will be described in detail. However, photographic image data read from the photographic film F in the main scan mode can be processed in the same manner. is there.

  When a plurality of thumbnail image data and high resolution image data read from the media driver 32 are stored in the memory 41, several frames of photographic images based on the thumbnail image data and a graphic operation screen are displayed on the screen of the monitor 34. . Normally, since the digital image stored in the medium is compressed by the JPEG method, image data obtained by inverse conversion and decompression is stored.

  For each frame image displayed on the screen of the monitor 34, the order processing unit 43 sets the print conditions, that is, the number of prints, the print size, and the like through the operator's operation on the graphic operation screen.

  Further, distortion correction, sharpening processing, and color correction are executed for each frame image by the image processing unit 47 through an operator's operation, and the image correction processing conditions set at this time are stored in the memory 41. .

  When the print condition setting and correction processing operations for each frame image displayed on the screen of the monitor 34 are completed, the screen is scrolled to the next screen, and the same processing is executed for all the frame images.

  When all the operation processes are completed and a print output operation is performed via the graphic operation screen, distortion correction, granular noise suppression process, and sharpening process are performed on the high-resolution photographic image data stored in the memory 41. Image processing such as color correction processing and enlargement / reduction processing is executed in order, and the image data after the image processing is converted and generated into print data by the print data generation unit 44 and output to the photographic printer 2.

  Distortion correction, sharpening processing, and color correction processing for high-resolution photographic image data are automatically corrected based on correction processing conditions set for thumbnail images, that is, correction processing conditions stored in the memory 41. Processing is executed.

  The granular noise suppression processing unit 51 for a digital camera photographed image according to the present invention will be described in detail below.

  As illustrated in FIG. 4, the granular noise suppression processing unit 51 includes a pixel component extraction unit 10 that extracts a pixel component of a pixel of interest from photographic image data, and a granular component included in the original pixel component extracted by the pixel component extraction unit 10. A noise suppression processing unit 11 that suppresses noise and outputs a noise suppression pixel component; and a pixel component conversion unit 12 that converts a pixel component indicated by the noise suppression pixel component into an RGB component to generate noise suppression image data. I have.

  The pixel component extraction unit 10 includes an RGB-YUV conversion processing unit 101 that converts each of the RGB components RGB of each pixel constituting the photographic image data into a YUV component, and the luminance component obtained by the RGB-YUV conversion processing unit 101 Each of Y and the color difference component UV and the RGB component RGB input to the pixel component extraction unit 10 can be output as they are.

  The pixel component conversion unit 12 is a YUV of each pixel obtained from the noise suppression pixel component of the luminance component Y and the noise suppression pixel component of the color difference component UV among the noise suppression pixel components obtained from the noise suppression processing unit 11 described later. A YUV-RGB conversion processing unit 121 that converts each of the components into RGB components RGB is provided.

  The noise suppression processing unit 11 identifies a smoothed pixel to be smoothed and a non-smoothed pixel that should not be smoothed from the luminance component Y or the RGB component RGB obtained from the pixel component extracting unit 10, and smoothes based on the identification result. The smoothing processing unit 112 for calculating the masking mask image data, and the smoothing mask image data obtained from the smoothing processing unit 112 and the original pixel component obtained from the pixel component extraction unit 10 are used for the granularity by a graph cut method described later. And a graph cut method calculation processing unit 111 that calculates a noise suppression pixel component in which noise is suppressed.

  The pixel component conversion unit 12 converts the noise-suppressed pixel component of the input luminance component Y and chrominance component UV back to the RGB component RGB via the YUV-RGB conversion processing unit 121, and noise-suppressed image data. Output as.

The RGB-YUV conversion processing unit 101, based on the following conversion formula, reads photographic image data that is a high-resolution original image read from the memory 41 (for example, about 8264 pixels, about 3264 pixels wide × 2448 pixels long). R (red), G (green), and B (blue) color components (hereinafter referred to as “RGB”) of each of the pixels constituting the pixel are converted into a luminance component Y and a color difference component UV. Output.
Y = 0.29900R + 0.58700G + 0.11400B
U = −0.16874R−0.33126G + 0.50000B
V = 0.50000R-0.41869G-0.0811B

  The smoothing processing unit 112 changes the calculation method of the smoothing mask image data according to the pixel component, and outputs the calculated smoothing mask image data to the graph cut method arithmetic processing unit 111. Which of the original pixel components of the luminance component Y and the RGB components RGB output from the pixel component extraction unit 10 is used to calculate the smoothed mask image data is determined when the photographic image processing apparatus 1 is initially set. Is determined based on the setting information stored in.

  The pixel component setting information used for the calculation in the memory 41 may be configured to be set by input from the user via the graphic user interface unit 42 every time image processing is performed.

  When the smoothing processing unit 112 calculates the smoothed mask image data using the luminance component Y obtained from the RGB-YUV conversion processing unit 101, the result of filtering the luminance component Y with a Sobel filter The smoothed mask image data is output.

  The Sobel filter is a filter that is often used when extracting edges while suppressing noise. For example, when extracting vertical edges, after differentiating with respect to the horizontal direction, the vertical direction perpendicular to the vertical direction is differentiated. Smoothing is performed to reduce noise while leaving a vertical edge.

When edge extraction is performed for each of the vertical direction and the horizontal direction by a Sobel filter, the vertical edge extraction result for each pixel is Sy, and the horizontal edge extraction result is Sx, the pixel value gradient S is

It is represented by

  That is, the smoothing processing unit 112 calculates the gradient S of each pixel value as a result of filtering the input luminance component Y by the Sobel filter, and smoothes the image data using the gradient S as the pixel value of each pixel. Output as masked mask image data.

  Note that the smoothing mask image data is not limited to the above-mentioned Sobel filter, and by extracting edge information, it is possible to identify smoothed pixels and non-smoothed pixels by “Edge Preserving Smoothing”. ”Or“ Bilateral Filter ”or the like.

  In addition, when the smoothing processing unit 112 calculates the smoothed mask image data using the RGB component RGB, the smoothing processing unit 112 uses the support vector machine to calculate the RGB component from the covariance value matrix feature amount of the RGB component RGB. A smoothed pixel and a non-smoothed pixel are identified for each pixel, and the output result before threshold processing is normalized to calculate smoothed mask image data.

First, from the RGB component RGB, the following three covariance value matrix feature quantities are obtained for a 3 × 3 neighborhood area centered on the target pixel.
1. Determinant of covariance matrix S of RGB components in region det (S)
2. Trace of covariance matrix S of RGB components in the region trace
3. Difference between the maximum luminance component value and the minimum luminance component value in the area maxmin

However, the determinant det (S) and the trace trace are normalized as follows.

  Using these covariance matrix feature quantities, a support vector machine (Support Vector Machine (SVM)), which is one of identification methods using supervised learning, is used to handle two classes of identification problems. The support vector machine identifies whether each pixel belongs to the class of smoothed pixels that should be smoothed or belongs to the class of non-smoothed pixels that are not smoothed, and before thresholding the two classes A value normalized so as to take a value from 0 to 255 is set as a pixel value of each pixel of the smoothed mask image data.

  The graph cut method calculation processing unit 111 uses the smoothed mask image data input from the smoothing processing unit 112, and each of the original pixel components of the luminance component Y and the color difference component UV extracted by the pixel component extraction unit 10. Therefore, the noise suppression pixel component that suppresses the granular noise is calculated by the graph cut method.

  The graph cut method has been widely used in recent years as a method for solving discrete optimization problems. It considers image processing and other problems as energy minimization problems, and uses a graph minimum cutting (graph cut) algorithm to find the optimal solution. This is a method for calculating.

  Hereinafter, the relationship between the energy function and the graph structure and the principle of optimization in the present invention will be described.

First, when each pixel v with respect to the image V is vεV, the energy allocated to each pixel v of the image V from the set L including a finite number of labels one by one is expressed by the following energy function E (L ).

Edata (L) is a penalty function for the difference between the observed image data and the label Lv, and Esmooth (L) is a penalty function for the discontinuity between pixels. Here, Edata (L) and Esmooth (L) are defined as follows.

  Here, N is an adjacency relationship, (u, v) εN indicates that the pixels u and v are adjacent, and Lu and Lv indicate labels assigned to the pixels u and v.

  Data (L) is a value that depends only on the label assigned to each pixel, and the label assigned to each pixel is directly influenced by the image data in the sense that it is directly influenced by the image data. be called.

  Esmooth (L) is a value that depends on the relationship between the labels assigned to adjacent pixels, and the values of labels that should be assigned between adjacent pixels are made close and smooth so that the values between adjacent pixels do not change significantly. In this sense, it is called “smoothing term”.

  In addition to nodes (vertices) and edges (edges) connecting them, a graph created by the graph cut method is composed of special nodes called terminals. All edges (sides) have a cost (weight).

  In the graph cut method, the above-described graph is created, an edge that minimizes the cost is cut by the minimum cut maximum flow algorithm, and a label is assigned to the node.

  For example, as shown in FIG. 5A, two terminals of a source s and a sink t and four nodes v1, v2, v3, and v4 are created, and an edge that connects each node called n-link is created. , An edge connecting each node and each terminal called t-link is created.

  Subsequently, as shown in FIG. 5B, the cost of the edge in the direction from the source s to the sink t is applied to the created graph using the minimum cut maximum flow algorithm (the bold line portion in FIG. 5B). The node set V is divided into two subsets S and T (cut) so that (see) is minimized. However, sεS and tεT.

  Here, assuming that a node is a pixel of each image and a terminal is a subset of pixel values assigned to each pixel, the cut in the above graph is regarded as assigning a subset of pixel values to each pixel. be able to.

  Further, since a subset of different pixel values is assigned to the pixels at both ends of the n-link to be cut, the cost of n-link is considered a penalty for discontinuity between pixels.

  Further, as shown in FIG. 5B, among the nodes at both ends of the cut t-link, a node indicating a subset of pixel values is assigned to a node that is not a terminal, and therefore the cost of t-link. Is considered a penalty for the difference between the originally assigned label and the label assigned after the cut.

  For example, if the node v2 is a node belonging to the subset S, a label belonging to the subset S is again assigned by the cut, and if the node v2 is a node belonging to the subset T, the subsets that differ depending on the cut A label belonging to S is assigned.

  That is, the cost of t-link having both ends of the node v2 and the sink t does not occur when the same label as the original is assigned, but occurs when the label different from the original is assigned.

  Therefore, n-link corresponds to Esmooth (L), and t-link is considered to correspond to Edata (L). E (L) can be considered as the total cost generated when the above graph is cut.

  Thus, the problem of cutting the above-mentioned graph so as to minimize the cost is that when the image is graphed, the energy function for allocating a label to each pixel as a node is minimized. By finding the cut that minimizes the cost by the correlation and graph cut method, it is possible to assign an optimal pixel value that minimizes the energy function.

  For the realization of the graph cut method, publicly known literature “Graph Cut (Tutorial), Hiroshi Ishikawa, March 2007 CVIM Study Group Tutorial / Information Processing Society of Japan Research Report 2007-CMIM-158- (26) pp. 193-204.” A number of algorithms are known, etc., but the graph cut calculation processing unit 12 in the present invention is configured using an α extension algorithm.

  The α extension algorithm is often used when a set of labels assigned to a node is multivalued. Assigning a label to a node or cutting a graph is called “arrangement”. An α extension from an arrangement L is a movement that only allows the number of nodes to which the arrangement L is assigned α. I mean.

  A graph in which the energy function is defined for the binary value that changes or does not change the label α, and the α expansion that starts with a certain arrangement L and reduces the energy most from the arrangement is binary (α and the original pixel value). By minimizing using a cut and repeating this for all αεL, the best movement is found and the movement destination is repeated as a new arrangement L.

Hereinafter, returning to the description of the embodiment, the graph cut method calculation processing unit 111 defines energy functions Edata (L) and Esmooth (L) used in the graph cut method as follows and stores them in the memory 41. .

  Here, Wu and Wv indicate respective pixel values of adjacent pixels u and v of the smoothed mask image data input from the smoothing processing unit 112. Since the smoothing term is considered as the cost of the edge in the graph cut, the smoothing term is multiplied by the average of the pixel values Wu and Wv as a weighting coefficient. As described above, by multiplying the average values of Wu and Wv, which are pixels of the smoothed mask image data, as spatially non-uniform weighting factors, the range to be smoothed for each location changes substantially.

  H and k are weighting factors (hereinafter, h is a “first factor”, k is a “second factor”, and the weights of the data term and the smoothing term can be adjusted according to the characteristics of the input original pixel component. A value corresponding to the characteristics of the input original pixel component is stored in the memory 41.

  For example, when outputting the luminance component, color difference component, and noise suppression pixel component of photographic image data, human visual spatial resolution perception is more sensitive to the luminance component than to the color difference component. Based on this, when calculating the noise suppression pixel component for the luminance component, the second coefficient k is decreased to reduce the degree of smoothing, and when calculating the noise suppression pixel component for the color difference component, the second coefficient k is increased. Adjustment such as increasing the degree of smoothing becomes possible.

  In addition, in the case of an image in which granular noise is clearly confirmed visually, when adjusting the degree of smoothing as a whole, the first coefficient h is decreased and the second coefficient k is increased. Is also possible. The graphic user interface unit 42 is provided with an application for displaying the editing screen of the first coefficient h and the second coefficient k on the screen of the monitor 34 so that the image can be adjusted while checking the image after the noise suppression processing. It doesn't matter.

  As described above, the graph cut method calculation processing unit 111 receives the smoothed mask image data input from the smoothing processing unit 112, the energy function definition formula, the energy function data term, and the smoothing term from the memory 41, respectively. The first coefficient h and the second coefficient k to be multiplied by, and the noise suppression pixel component in which granular noise is suppressed from the original pixel components of the luminance component Y and the color difference component UV input from the pixel component extraction unit 10 are described above. And is output to the YUV-RGB conversion processing unit 121 of the pixel component conversion unit 12.

The YUV-RGB conversion processing unit 121 inversely converts the noise suppression pixel component indicating the luminance component and the noise suppression pixel component indicating the color difference component obtained by the noise suppression processing unit 11 into RGB color components based on the following expression. And output as noise-suppressed image data.
R = Y−0.000007U + 1.401998V
G = Y−0.344133U−0.714138V
B = Y + 1.772003U + 0.000015V

  Below, the procedure of each process by the granular noise suppression process part 51 mentioned above is demonstrated based on the flowchart shown in FIG.

  First, when RGB component data of each pixel constituting the photographic image data is input to the pixel component extraction unit 10, a pixel component extraction step (S1) is executed.

  The pixel component extraction step (S1) includes an RGB-YUV conversion processing step (S11).

  First, the RGB-YUV conversion processing step is executed, each of the input RGB components of each pixel is converted into a YUV component, and the luminance component is converted into a smoothing processing unit 112 provided in the noise suppression processing unit 11 and a graph cut calculation. The color difference component is output to the processing unit 111, and is output to the graph cut calculation processing unit 111 provided in the noise suppression processing unit 11 (S11).

  Further, the RGB components input in the pixel component extraction step (S1) are output to the smoothing processing unit 112 provided in the noise suppression processing unit 11.

  Subsequently, when the original pixel component output from the pixel component extraction step (S1) is input to the noise suppression processing unit 11, the noise suppression processing step (S2) is executed.

  The noise suppression processing step (S2) includes a smoothing processing step (S21), a graph cut calculation step (S22), and an all pixel component noise suppression end determination step (S23).

  First, the smoothing processing step (S21) is executed, and the smoothing processing unit 112, based on the setting information stored in the memory 41, either one of the input luminance component Y or RGB component RGB. Smoothed mask image data is calculated from the pixel components and output to the graph cut method calculation processing unit 111.

  Subsequently, when the smoothed mask image data is input to the graph cut method calculation processing unit 111, a graph cut method calculation step (S22) is executed.

  The graph cut method calculation step (S22) includes a data term and a smoothing term definition formula stored in the memory 41, a first coefficient h and a second coefficient k stored in the memory 41, and a smoothing processing step (S21). The optimal solution for minimizing the energy function is calculated by the graph cut method using the pixel value of the smoothed mask image data output in (1), and is output as a noise suppression pixel component. There are two input pixel components, a luminance component Y and a color difference component UV, and a noise suppression pixel component is calculated for each. Note that the order in which the luminance component Y or the color difference component UV is processed first does not matter.

  Subsequently, an all pixel component noise suppression end determination step (S23) is executed to check whether two noise suppression pixel components of the luminance component Y and the color difference component UV are generated, and one of the noise suppression pixel components is determined. In the case where there is not, the graph cut calculation processing step (S22) is executed again after waiting for a predetermined time, and the noise suppression pixel component of the pixel component that does not exist is generated.

  When two noise suppression pixel components of the luminance component Y and the color difference component UV are generated, the two noise suppression pixel components are output to the pixel component conversion unit 12.

  When the noise suppression pixel components of the luminance component Y and the color difference component CY are input to the pixel component conversion unit 12, a pixel component conversion step (S3) is executed.

  The pixel component conversion step (S3) includes a YUV-RGB conversion processing step (S31). First, the YUV-RGB conversion processing step (S31) is executed, and the noise suppression pixel component and the color difference component of the luminance component Y are executed. The luminance component Y and the color difference component UV obtained from the UV noise suppression pixel component are converted into the RGB component RGB and output as noise suppression image data.

  Each processing by the granular noise suppression processing unit 51 described above is realized by executing a photographic image processing program of the present invention installed in a hard disk provided in the controller 33.

  That is, a photographic image processing program for generating noise-suppressed image data by suppressing granular noise included in photographic image data, the pixel component extracting step (S1) for extracting a pixel component of a target pixel of photographic image data; A data term Edata (L) indicating a difference between pixel values of an original pixel component which is a pixel component of the target pixel and a noise suppression pixel component which is a pixel component of an assumed pixel corresponding to the target pixel of the noise suppression image data, and noise suppression A smoothing term Esmooth (L) obtained by multiplying the dispersion of pixel values of pixel components by a spatially non-uniform weighting factor, and an energy function E (L ) And a noise suppression processing step (S2) for calculating a noise suppression pixel component whose energy function E (L) has a minimum value by a graph cut method. Image processing program to be executed is installed via a storage medium such as a stored CD or DVD to the computer.

  According to the present invention, the luminance component Y and the color difference component UV obtained from the noise suppression pixel component of the luminance component Y and the noise suppression pixel component of the color difference component Y output from the noise suppression processing step (S2) are converted into RGB components RGB. Even if a photographic image processing program that causes a computer to execute the pixel component conversion step (S3) for converting and outputting noise-suppressed image data is installed via a storage medium such as a CD or DVD that stores the program. I do not care.

  The smoothed mask image calculated using the support vector machine according to the present invention is shown in FIG. 7B as an image obtained by suppressing the granular noise by the bilateral filter with respect to the original image shown in FIG. FIG. 7C shows an image obtained as a result of suppressing granular noise using data, and results of suppressing granular noise using smoothed mask image data calculated using a Sobel filter according to the present invention. The image is shown in FIG.

  When the bilateral filter is used, the granular noise has almost disappeared, but the edge portion becomes thin and fine irregularities appear in the region boundary portion. This becomes conspicuous in a portion where adjacent pixels hardly have many similar pixel values.

  When the graph cut according to the present invention is used, as shown in FIGS. 7C and 7D, it can be seen that the edge is not blurred and the granular noise is removed. Furthermore, the color unevenness has almost disappeared, and the result is better than others. When using the smoothed mask image data calculated with the support vector machine and when using the smoothed mask image data calculated using the Sobel filter, the latter is slightly more flat. There is little brightness unevenness.

  Note that the above-described embodiment is merely an example of the present invention, and it is needless to say that the specific configuration and the like of each block can be changed and designed as appropriate within the scope of the effects of the present invention.

External view of photographic image processing device Illustration of photo printer Functional block diagram of photographic image processing device Functional block diagram of the granular noise suppression processing unit It is explanatory drawing explaining the graph utilized by the graph cut method, (a) is explanatory drawing which shows a graph, (b) is explanatory drawing explaining the cutting of a graph. Flowchart for explaining the processing procedure of the granular noise suppression processing unit It is a photographic image before and after performing granular noise suppression, (a) is a photographic image showing an original image, (b) is a photographic image in which granular noise is suppressed using a bilateral filter, and (c) is a support vector machine. A photographic image in which granular noise is suppressed using the smoothed mask image data calculated in step (d), and (d) is a photographic image in which granular noise is suppressed using the smoothed mask image data calculated using a Sobel filter.

Explanation of symbols

1: Photo image processing device 10: Pixel component extraction unit 11: Noise suppression processing unit 12: Pixel component conversion processing unit 51: Granular noise suppression processing unit 101: RGB-YUV conversion processing unit 111: Graph cut method calculation processing unit 112: Smoothing processing unit 121: YUV-RGB conversion processing unit E (L): energy function Edata (L): data term Esmooth (L): smoothing term S1: pixel component extraction step S11: RGB-YUV conversion processing step S2: Noise suppression processing step S21: Smoothing processing step S22: Graph cut method calculation step S3: Pixel component conversion step S31: YUV-RGB conversion processing step UV: Color difference component Y: Luminance component

Claims (9)

A photographic image processing method for generating noise-suppressed image data by suppressing granular noise included in photographic image data,
A pixel component extraction step of extracting a pixel component of a target pixel of the photographic image data;
A data term indicating a difference between a pixel value of an original pixel component which is a pixel component of the target pixel and a noise suppression pixel component which is a pixel component of an assumed pixel corresponding to the target pixel of the noise suppression image data;
A smoothing term obtained by multiplying a variation in pixel value of the noise suppression pixel component by a spatially non-uniform weighting factor,
A noise suppression processing step of generating an energy function obtained by multiplying each of the data term and the smoothing term by a coefficient, and calculating the noise suppression pixel component having a minimum value by the energy function by a graph cut method;
A photographic image processing method. A smoothing process step of identifying smoothed pixels to be smoothed and non-smoothed pixels not to be smoothed from the pixel components extracted in the pixel component extracting step, and calculating smoothed mask image data based on the identification result In addition,
The data term is generated by multiplying a difference between pixel values of the noise suppression pixel component and the original pixel component with respect to all the pixels of the noise suppression pixel component by a predetermined first coefficient,
The weighting factor is an average value of pixel values of two pixels of the smoothed mask image data corresponding to two adjacent pixels of the noise suppression pixel component,
In the smoothing term, smoothing energy, which is a result of multiplying a difference between pixel values of two adjacent pixels of the noise suppression pixel component by the weighting factor, is applied to all two adjacent pixels of the noise suppression pixel component. The photographic image processing method according to claim 1, wherein the photographic image processing method is generated by a product of a cumulative addition value and a predetermined second coefficient. The pixel component extraction step extracts a luminance component and a color difference component from the photographic image data,
The noise suppression processing unit calculates the noise suppression pixel component of the luminance component and the noise suppression pixel component of the color difference component,
A pixel component conversion step of converting the noise suppression pixel component of the luminance component and the noise suppression pixel component of the color difference component into the noise suppression image data that is the noise suppression pixel component of the RGB component,
The photographic image processing method according to claim 2.   4. The photographic image processing method according to claim 3, wherein the smoothing processing step calculates the smoothed mask image data based on edge information obtained by filtering the luminance component using an edge extraction filter.   The photographic image processing method according to claim 4, wherein the edge extraction filter is a Sobel filter.   The smoothing processing step uses a support vector machine to calculate the smoothed pixel and the non-smoothed pixel for each pixel of the RGB component from the covariance value matrix feature of the RGB component of the photographic image data. The photographic image processing method according to claim 3, wherein the smoothed mask image data is calculated by normalizing an output result before threshold processing and identifying the output.   The noise suppression processing unit calculates a value of the second coefficient when calculating the noise suppression pixel component of the luminance component, and a value of the second coefficient when calculating the noise suppression pixel component of the color difference component. The photographic image processing method according to claim 3, wherein the photographic image processing method is set to a value smaller than the value. A photographic image processing program for generating noise-suppressed image data by suppressing granular noise included in photographic image data,
A pixel component extraction step of extracting a pixel component of a target pixel of the photographic image data;
A data term indicating a difference between a pixel value of an original pixel component which is a pixel component of the target pixel and a noise suppression pixel component which is a pixel component of an assumed pixel corresponding to the target pixel of the noise suppression image data;
A smoothing term obtained by multiplying a variation in pixel value of the noise suppression pixel component by a spatially non-uniform weighting factor,
A noise suppression processing step of generating an energy function obtained by multiplying each of the data term and the smoothing term by a coefficient, and calculating the noise suppression pixel component having a minimum value by the energy function by a graph cut method;
A photographic image processing program for causing a computer to execute. A photographic image processing apparatus for generating noise-suppressed image data by suppressing granular noise included in photographic image data,
A pixel component extraction unit that extracts a pixel component of a target pixel of the photographic image data;
A data term indicating a difference between a pixel value of an original pixel component which is a pixel component of the target pixel and a noise suppression pixel component which is a pixel component of an assumed pixel corresponding to the target pixel of the noise suppression image data;
A smoothing term obtained by multiplying a variation in pixel value of the noise suppression pixel component by a spatially non-uniform weighting factor,
A noise suppression processing unit that generates an energy function obtained by multiplying and adding a coefficient to each of the data term and the smoothing term, and calculates the noise suppression pixel component in which the energy function exhibits a minimum value by a graph cut method;
A photographic image processing apparatus.