US20140321696A1 - Perceptual Rating Of Digital Image Retouching - Google Patents
Perceptual Rating Of Digital Image Retouching Download PDFInfo
- Publication number
- US20140321696A1 US20140321696A1 US14/359,169 US201214359169A US2014321696A1 US 20140321696 A1 US20140321696 A1 US 20140321696A1 US 201214359169 A US201214359169 A US 201214359169A US 2014321696 A1 US2014321696 A1 US 2014321696A1
- Authority
- US
- United States
- Prior art keywords
- image
- rating
- statistics
- retouched
- geometric
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G06T7/003—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Definitions
- Popular image-editing software such as Adobe Photoshop, allows photo editors to easily alter a person's appearance. These alterations may affect the geometry of the subject and may include slimming of legs, hips, and arms, elongating the neck, improving posture, enlarging the eyes, or making faces more symmetric. Other photometric alterations affect skin tone and texture. These changes may include smoothing, sharpening, or other operations that remove or reduce wrinkles, cellulite, blemishes, freckles, and dark circles under the eyes. A combination of geometric and photometric manipulations allows image retouchers to subtly or dramatically alter a person's appearance.
- a method is provides an image retouching rating.
- the method includes the step of receiving at a computer including at least one processor a first set of pixel data of an original digital image and a second set of pixel data of a retouched digital image.
- the method also includes using the at least one processor to determine a plurality of geometric statistics and a plurality of photometric statistics from the first and second sets of pixel data.
- the method further includes the step of using the at least one processor to generate a rating of the retouched image based upon the geometric statistics and photometric statistics to indicate deviation of the retouched image from the original image.
- the method further includes determining a plurality of geometric statistics comprising a first mean and a first standard deviation of geometric distortion of retouching, between the first and second sets of pixel data, corresponding to a first portion of a human body.
- the first portion of the human body is a portion of the face.
- the method further includes determining a second mean and a second standard deviation of geometric distortion of retouching, between the first and second sets of pixel data, corresponding to a second portion of the human body, wherein the second portion is different from the first portion.
- the method further includes determining a plurality of photometric statistics including a third mean and a third standard deviation, between the first and second sets of pixel data, of the frequency response of a linear filter corresponding to a first portion of the human body.
- the method further includes determining a plurality of photometric statistics including a fourth mean and fourth standard deviation of structural similarity (SSIM) characterized by contrast and structural modification.
- SSIM fourth standard deviation of structural similarity
- the step of generating includes executing, using the processor, a non-linear support vector regression (SVR) to compute the image retouching rating based upon the plurality of statistics and characterization data defining the training parameters of the SVR.
- SVR non-linear support vector regression
- the method further includes determining, within a touchup filter, whether the retouched image is to be displayed by a browser based upon the image retouching rating and a rating threshold; displaying the retouched image within the browser when the retouched image is to be displayed, and displaying an image outline within the browser when the retouched image is not to be displayed; wherein the browser is in communication with a web server.
- the method further includes defining within a browser, a rating selector that specifies a maximum image retouching rating value for retouched images to be displayed within the browser; sending the maximum image retouching rating value to a web server in communication with the browser; and receiving a webpage for display within the browser with at least one retouched image having an image retouching rating equal or less than the maximum rating value.
- FIG. 1 is a block diagram illustrating one exemplary system for generating a perceptual rating of digital image retouching, in an embodiment.
- FIG. 2 is a flow chart illustrating of one exemplary method for determining a perceptual rating of digital image retouching using the system of FIG. 1 , in an embodiment.
- FIG. 3 illustrates one exemplary algorithm for determining a perceptual rating of digital image retouching, in an embodiment.
- FIG. 4 illustrates five representations of five original digital images.
- FIG. 5 illustrates five representations of the five original images of FIG. 4 after retouching.
- FIG. 6 illustrates exemplary results from a leave-one-out cross-validation using a nonlinear support vector regression (SVR) to correlate observer ratings of digital image retouching to the geometric and photometric measurements of digital image retouching.
- SVR nonlinear support vector regression
- FIG. 7 illustrates five representations of another five original digital images.
- FIG. 8 illustrates five representations of the five original images of FIG. 7 after retouching.
- FIG. 9 shows one exemplary system for rating and certifying a retouched digital image, in an embodiment.
- FIG. 10 shows an exemplary publication containing the retouched image and displaying a rating determined by the system of FIG. 9 , in an embodiment.
- FIG. 11 shows one exemplary rating system configured for automatically rating observations based upon an observation model and SVR training data, in an embodiment.
- FIG. 12 shows exemplary use of ratings, generated by a rating server, by a browser when viewing a web site, in an embodiment.
- FIG. 13 shows exemplary selection of digital images for display within a web page display of a browser based upon ratings generated by rating server, in an embodiment.
- FIG. 14 shows one exemplary system for generating a rating module that automatically determines a rating for a retouched digital image in comparison to an original digital image, in an embodiment.
- This application discloses, among other things, how to automatically compute a quantitative and perceptually meaningful rating of image retouching.
- the application also discloses rating a photograph based upon a degree by which the photograph has been digitally altered by explicitly modeling and estimating geometric and photometric changes. This rating has been demonstrated to correlate well with perceptual judgments of image retouching and may be used to objectively judge by how much a retouched image has strayed from reality.
- Benefits of the disclosed methods and systems may include providing a perceptually relevant rating of image retouching that helps find a balance between the competing interests of editing images for selling products and discouraging photographic alterations that promote unrealistic expectation of body image.
- Providing an image retouching rating alongside a published image may inform the public of the extent to which images have strayed from reality.
- Such a rating may also provide incentive for publishers and models to reduce usage of the more extreme forms of digital image retouching that are common today. This rating may also help image retouchers and editors because, even when original and retouched images are available, it is often difficult to see and quantify the extent of image alterations.
- Benefits of the disclosed methods and systems may also include providing a fast and automatic rating to users and publishers.
- the core computational component of the disclosed system is typically fully automatic, however a user may annotate the hair/head, face, and body in certain embodiments. When deploying an industry-wide rating system, this annotation could either be done automatically or with fairly minimal user assistance.
- the rating quantifies the perceptual impact of geometric and photometric retouching by modeling common image retouching techniques.
- Geometric changes are modeled, for example, with a dense locally linear, but globally smooth, motion field.
- Photometric changes are modeled, in certain embodiments, with a locally linear filter and a generic measure of local image similarity (SSIM). These model parameters are typically automatically estimated from the original and retouched images.
- the extent of image manipulation is, for example, quantified with eight summary statistics extracted from these models.
- FIG. 1 is a block diagram illustrating one exemplary system 100 for generating a perceptual rating of digital image retouching.
- System 100 includes a computer 112 which has at least one processor 106 and an algorithm 108 comprising machine readable instructions stored in a memory 114 .
- algorithm 108 includes a geometric and photometric calculator 120 for determining statistics 121 , and a support vector regression (SVR) 122 for generating rating 110 based upon characterization data 123 , geometric statistics 132 and photometric statistics 134 .
- SVR support vector regression
- Geometric statistics 132 and photometric statistics 134 are extracted by geometric and photometric calculator 120 by comparing the original image 102 to the retouched image 104 .
- the amount of photo distortion or perceptual distortion is quantified from eight summary statistics 121 that are extracted from geometric and photometric models. These statistics consist of four geometric measurements (e.g., geometric statistics 132 ) and four photometric measurements (e.g., photometric statistics 134 ).
- geometric statistics 132 determined by calculator 120 may use an 8-parameter affine model that is used to model a geometric transformation between local regions in the before and after images, e.g., original and retouched images.
- the luminance transformation is modeled with a 2-parameter model embodying brightness and contrast.
- This 8-parameter model is given by:
- f b and f a are the local regions of the before and after images
- c and b are contrast and brightness terms
- m i are terms of a two by two affine matrix
- t x and t y are translation terms.
- the luminance terms c and b are incorporated only so that the geometric transformation can be estimated in the presence of luminance differences between the before and after images or the original and retouched images.
- a quadratic error function in these parameters is defined, for example, by approximating the right-hand side of Equation (1) with a first-order truncated Taylor series expansion. This error function is then minimized using standard least-squares optimization. Because these geometric parameters are estimated locally throughout the image, the resulting global transformation can lead to unwanted discontinuities. A global penalty on large motions and a smoothness constraint are imposed by penalizing the local model parameters proportional to their magnitude and the magnitude of their local gradient. The addition of this smoothness constraint requires an iterative minimization which is boot-strapped with the result of the least-squares optimization. This optimization is embedded within a coarse-to-fine differential architecture in order to contend with both large- and small-scale geometric changes. A model of missing data is also incorporated that contends with the case when portions of the after image have been entirely removed or added relative to the before image.
- This estimation is performed only on the luminance channel of a color image.
- the before and after images initially are histogram equalized to minimize any overall differences in brightness and contrast.
- the background in each image is replaced with white noise in order to minimize any spurious geometric distortion.
- This geometric model embodies the basic manipulation afforded by, for example, the Photoshop liquify tool used by photo retouchers to alter the global or local shape of a person.
- the first two geometric statistics are the mean and standard deviation of the magnitude of the estimated vector field v(x,y), shown in Equation [2], projected onto the gradient vector of the underlying luminance channel.
- This projection emphasizes geometric distortions that are orthogonal to image features, which are more perceptually salient.
- These two statistics are computed only over the face region, which quantify geometric facial distortion.
- the third and fourth geometric statistics are the mean and standard deviation of the magnitude of the estimated vector field v(x,y), Equation [2], projected onto the gradient vector and computed over the body region. These projected vectors are weighted based on specific body regions.
- bust/waist/thigh regions are weighted by a factor of 2
- head/hair regions are weighted by a factor of 1 ⁇ 2
- the remaining body regions have unit weight (a full range of weights were explored and the final results are not critically dependent on these specific values).
- Photometric statistics 134 are for example determined by calculator 120 using either a linear filter model or a similarity measure SSIM.
- a linear filter model For example, basic photometric retouches between local regions in the after image, i.e. the retouched image, and the geometrically aligned before image, i.e. the original image, are modeled with a linear filter h(x,y), of size nine by nine pixels, given by:
- ⁇ is the convolution operator
- f b is the geometrically aligned before image region, Equation (1).
- the filter h(x,y) is estimated locally using a conjugate gradient descent optimization with a Tikhonov regularization.
- the regularization is used to enforce symmetry (i.e., zero-phase) on the estimated filter h. This estimation is performed only on the luminance channel of a color image.
- Photometric retouches that are not captured by Equation [3] are measured with the similarity measure SSIM.
- This measure embodies contrast and structural retouches C(x,y) as follows:
- ⁇ a and ⁇ b are the standard deviations of the image regions f a and ⁇ f b
- ⁇ ab is the covariance of f a and ⁇ f b
- C 3 C 2 /2. Note that in this implementation of SSIM the brightness term is excluded because it does not impact observers' judgments. For the same reason, SSIM is computed only on the luminance channel of a color image. This photometric model embodies special effects afforded by various Photoshop filters.
- the first two photometric statistics are the mean and standard deviation of the structural similarity (SSIM) or contrast and structural modification C(x,y), in Equation [4], computed over the face region.
- the third and fourth photometric statistics are a measure D of the frequency response of the linear filters h(x,y), shown in Equation [3]. D is expressed as the following:
- H( ⁇ ) and ⁇ F b ( ⁇ ) are unit-sum normalized one-dimensional (1D) frequency responses of the filter h and the local region ⁇ f b which are computed by integrating their 2D Fourier transforms across orientation.
- the parameter D is positive when h is a blurring filter, negative when h is a sharpening filter, and is tailored to the analysis of people in which filtering is commonly used to remove or enhance facial features.
- the mean and standard deviation of parameter D, computed over the face region, are the final two statistics.
- the first four geometric statistics are mean and standard deviation of the estimated vector field computed separately over the face and body.
- the second four photometric statistics are the mean and standard deviation of SSIM and the frequency response of the linear filters.
- Characterization data 123 is, for example, defines a set of training parameters used to train the SVR model 122 used by algorithm 108 .
- To generate characterization data 123 for example, four hundred sixty eight before/after images were collected from a variety of on-line resources, primarily the websites of photo retouchers sselling their services. These images spanned the range from minor to radical amounts of retouching.
- the presentation of images was self-timed and observers could manually toggle between the before and after images as many times as they chose (observers are better able to see the amount of retouching within the images when toggling rather than viewing side-by-side).
- each observer rated a random set of five images three times each. The presentation of these images was evenly distributed throughout the trial. Each observer was paid $3 for their participation and a typical session lasted 30 min. Given the uncontrolled nature of the data collection, some data filtering was necessary. Approximately 9.5% of observers were excluded because they frequently toggled only once between the before and after image and they responded with high variance on the repeated trials.
- the SVR technique was used to generate characterization data 123 as a mapping between the observer ratings and eight summary statistics (e.g., statistics 121 ) extracted from the geometric and photometric models of photo retouching at step 324 . Each statistic was individually scaled into the range [31 1, 1]. Specifically, a nonlinear SVR with a Gaussian radial basis kernel was employed. A leave-one-out cross-validation was performed in which the SVR was trained on 467 of 468 image ratings and tested on the remaining image. This training and testing was repeated 468 times in which each image was individually tested.
- the SVR has two primary degrees of freedom: (1) the scalar y specifies the spatial extent of the kernel function; and (2) the scalar c specifies the penalty applied to deviations of each data point from the regression function. These parameters were selected by performing a dense 2D grid search to maximize the correlation coefficient of each training set. The results of the above crowd sourcing technique may then be stored into memory 114 as characterization data 123 .
- Processor 106 executes algorithm 108 to process original digital image data 102 and retouched digital image data 104 .
- algorithm 108 may be executed by multiple processors 106 .
- Algorithm 108 further generates an output a rating 110 indicating deviation of the retouched digital image data 104 from the original digital image data 102 .
- Output rating 110 may be stored within memory 114 , or transferred as an output to exterior devices other than computer 112 .
- algorithm 108 also includes an eight parameter model.
- Memory 114 may represent one or more of a magnetic or optical disk and/or an electronic memory integrated circuit (e.g., RAM, ROM, FLASH, and so on).
- FIG. 2 is a flow chart illustrating of one exemplary method 200 for determining a perceptual rating of digital image retouching using system 100 of FIG. 1 .
- Method 200 is for example implemented, at least in part, within algorithm 108 of computer 112 .
- step 202 method 200 receives, within computer 112 , original digital image data and retouched digital image data.
- geometric and photometric statistics e.g., statistics 121
- step 206 uses algorithm 108 , for example an SVR algorithm, to determine a rating 110 for the retouched image based upon the statistics 121 and characterization data 123 .
- Step 206 is performed, for example, at least partially by using processor 106 .
- step 208 method 200 reports the rating 110 of the retouched image.
- FIG. 3 shows representations of an original digital image 302 , a retouched digital image 304 , and a visualization of the steps required to create a rating of digital image retouching, for example, by measuring geometric and photometric retouching.
- An image 306 represents measured geometric distortions that are depicted as a vector field, where a superimposed color coding corresponds to vector magnitude. For clarity of illustration, area 332 near the breast and area 334 near the waist are shown in dashed outline and represent areas with higher vector magnitude numbers that correspond to greater distortion.
- An image 308 also within FIG.
- An image 310 represents measured photometric distortions quantified by a linear filter, in which area 340 near the cheek has positive values denoting blurring and area 338 near the eye has negative values denoting sharpening.
- An image 310 represents structural similarity (SSIM) where areas 342 and 344 , near the eyes and mouth, respectively, have smaller values that denote larger image differences.
- Step 312 uses Equation [1] to measure geometric differences for face and body with eight parameters and then computes vector field based upon six of the eight parameters using Equation [2].
- Step 318 computes mean and standard deviation of magnitude of the vector in face and body.
- Step 314 uses Equation [3] to measure photometric differences for the face and then uses Equations [6] to calculate the measure D.
- Step 320 computes mean and standard deviation of D.
- Step 316 uses Equations [4] and [5] to measure photometric differences for the face and computes C(x,y).
- Step 322 computes mean and standard deviation of C(x,y).
- step 324 machine learning techniques, such as SVR, are then used to relate these summary statistics to perceptual judgments made by human observers. It is shown below, in FIGS. 4-8 , that the summary statistics combine to yield a metric that correlates well with perceptual ratings of photo alteration.
- FIG. 5 illustrates five representative retouched digital images 502 , 504 , 506 , 508 , and 510 with a minimal prediction error from original digital images 402 , 404 , 406 , 408 , and 410 , shown in FIG. 4 . Shown along the top is how an average person rated the differences in these images (left), the predicted rating of photo retouching (right) using the present algorithm, and their difference (in parenthesis). An image with little to no retouching is rated a “1” while an image with significant retouching is rated a “5”. The value in the parentheses is the difference between these two values.
- FIG. 6 illustrates the correlation between the mean of 50 observer ratings per image and one exemplary of the present algorithm.
- Each data point corresponds to one of 468 images rated by the test group on a scale of 1 to 5.
- the predicted rating for each image was determined by training an SVR on 467 images using a leave-one-out cross-validation methodology.
- the R-value is 0.80
- the mean/median absolute prediction error is 0.30/0.24 with a standard deviation of 0.24 and a max absolute error of 1.19.
- the absolute prediction error is below 0.5 for 81.4% of the images, and below 0.75 and 1.0 for 94.4% and 99.1% of the images, respectively.
- FIG. 8 illustrates another set of representative retouched digital images 802 , 804 , 806 , 808 , and 810 with maximal prediction error from respective original digital images 702 , 704 , 706 , 708 , 710 , shown in FIG. 7 .
- a model of missing data is also incorporated that contends with the case when portions of the after image have been entirely removed or added relative to the before image.
- the over- and underestimations illustrate some of the limitations of the present model.
- the perceptual distortion in retouched digital image 802 is overestimated because there is a large photometric difference for the young boy (removal of blemishes).
- the perceptual distortion in retouched digital image 804 is overestimated because there is and a large geometric difference for the young woman (change in shape and position of the head). However, neither of these differences corresponds to a large perceptual difference in appearance. On the other hand, the perceptual distortion in the remaining retouched digital images 806 , 808 and 810 is underestimated.
- the change to the symmetry of the young man's face, the addition of make-up to the woman, and the addition of teeth to the man are each relatively small from a photometric and geometric perspective but yield a large perceptual difference in appearance. Even with these limitations, the perceptual distortion is still reasonably measured over a diverse range of image alterations and content.
- the best performing SVR with one statistic consisted of the mean of the geometric facial distortion (statistic (1) as described above), which yielded an R-value of 0.58.
- the best performing SVR with two statistics consisted of the standard deviation of the geometric body distortion and the standard deviation of the photometric SSIM (statistics (4) and (6)), which yielded an R-value of 0.69.
- the best performing SVR with three statistics consisted of adding the standard deviation of the geometric facial distortion to the previous SVR (statistics (4), (5), and (6)), which yielded an R-value of 0.76.
- the best performing SVR of size 6 had an R-value of 0.80, equal to that of the full set of size 8.
- This subset of size 6 consisted of the statistics (1), (2), (4), (6), (7), and (8) as described above. Although six statistics are sufficiently powerful, they are extracted from each component of the geometric and photometric models. Therefore, there is little cost in using all eight statistics in terms of computational complexity or in terms of training the SVR.
- a linear SVR was also tested to validate the use of a nonlinear SVR over a simpler linear SVR.
- the R-value for the linear SVR is 0.72, as compared to 0.80 for the nonlinear SVR.
- the mean absolute prediction error is 0.34 with a standard deviation of 0.27 as compared to 0.30 and 0.24 for the nonlinear SVR.
- the max absolute error jumps from 1.19 to 1.93. Overall, the nonlinear SVR affords a considerably better prediction of observer ratings as compared to a linear SVR.
- the perceptual rating of the present disclosure is compared against two standard image similarity perceptual ratings.
- a perceptual rating based only on the mean and standard deviation of a standard application of SSIM yields an R-value of 0.52 as compared to our approach that had an R-value of 0.80.
- a perceptual rating based on only the mean squared error between the before and after image performed much worse with an R-value of only 0.30.
- Standard image similarity perceptual ratings perform poorly because they do not compensate for, or measure, large-scale geometric distortions.
- the system compares original and retouched images of human subjects.
- the system may also be trained for comparing images of other subjects.
- the system may be trained to compare images of landscape portraits, still-life, and wildlife.
- the system may be trained to compare images of art restorations.
- a rating of restored art may be used for evaluation purposes, wherein a poorly restored image may be awarded a high rating (indicating significant change to the image content).
- FIG. 9 shows one exemplary system 900 for rating and certifying a retouched digital image 924 .
- System 900 includes a server 902 , coupled with the Internet 950 , and having an algorithm 904 and a certificate generator 906 .
- Algorithm 904 may represent algorithm 108 and may implement method 200 of FIG. 2 .
- Certificate generator 906 includes an encryption, and optionally a decryption, algorithm that utilizes a private key 932 of a public/private key pair 930 , to generate a certificate 908 containing a rating 910 . Certificate 908 may then be decrypted using public key 934 of public/private key pair 930 .
- Server 902 may represent one or more computers that are connected to Internet 950 .
- server 902 includes a memory and a processor (not shown for clarity of illustration) that respectively store and execute instructions of algorithm 904 and instructions of certificate generator 906 .
- a publisher 920 desires to publish a retouched digital image 924 together with a certified rating that defines an amount of retouching applied to an original image 922 to form retouched digital image 924 .
- Publisher 920 sends a message 926 , containing original image 922 and retouched image 924 , to server 902 requesting a certified rating of retouched image 924 .
- algorithm 904 is executed to process original image 922 and retouched image 924 and to generate a rating 910 .
- Certificate generator 906 utilizes private key 932 (known only to certificate generator 906 ) to generate certificate 908 containing rating 910 and an image ID 912 that uniquely identifies retouched image 924 .
- image ID 912 is a checksum of data within retouched image 924 .
- image ID 912 is retouched image 924 .
- server 902 sends certificate 908 to publisher 920 as message 914 for example.
- message 914 may also contain public key 934 .
- Publisher 920 decrypts certificate 908 using public key 934 and optionally verifies that rating 910 applies to retouched image 924 using image ID 912 .
- Publisher 920 may retain certificate 908 as evidence of rating 910 for retouched image 924 , for example, to show compliance to a rating standard.
- FIG. 10 shows an exemplary publication 1002 generated by publisher 920 of FIG. 9 and containing retouched digital image 924 with rating 910 overlaid on the image.
- system 900 retouches and returns retouched image 924 to include rating 910 .
- rating 910 may be shown beside the image.
- publication 1002 is a web page
- the user may click on rating 910 to retrieve and validate the rating certificate associated with rating 910 .
- rating 910 is not shown unless the user clicks on (or causes a mouse-over event of) retouched image 924 wherein rating 910 is momentarily displayed.
- the rating system may also be applied to other human judgments, wherein the system may predict how an object may be judged.
- the rating may be applied to judging a photography contest, and rating images for a scientific journal.
- FIG. 11 shows one exemplary rating system 1100 that may be configured for automatically rating observations 1110 based upon an observation model 1104 and SVR training data 1106 .
- Observation model 1104 defined weightings for each of a plurality of features (data points) within digital images of observations 1110 .
- Analyzer 1102 processes observations 1110 based upon observation model 1104 and then applies SVR training data 1106 to determine a rating 1120 .
- Observation model 1104 may be defined for any observable set of features that a user of system wishes to automatically rate. Using the example shown above ( FIGS. 1-10 ), observations 1110 include retouched features of a human as determined by comparing an original digital image and a retouched digital image. Observation model 1104 is thus used to define the features within the images that are of interest, and defines a weighting for each of the features. Observation model 1104 may define parameters for features that are not always present within observations 1110 , but that are applied when the feature is present.
- the mathematical model may also be applied in reverse wherein a user may automatically retouch an image to have a desired rating.
- the system 1100 may be used to reduce the amount of retouching of a heavily retouched image until that image has a rating of 1, thereby meeting the magazine publishing criteria.
- system 900 measures geometric and photometric change between an original digital image and a retouched image
- system 900 may be used to reduce the level of retouching of the retouched image until a determined rating thereof is at a desired rating level.
- the geometric and photometric models are generative in nature, and define how an image is retouched from the original digital image to get to the retouched digital image.
- FIG. 12 shows exemplary use of ratings 1224 , generated by a rating server 1250 , by a browser 1202 when viewing a web site 1220 .
- Web site 1220 is shown with two exemplary retouched images 1222 ( 1 ) and 1222 ( 2 ) that each have a rating 1224 ( 1 ) and 1224 ( 2 ), respectively, generated by rating server 1250 .
- Rating server 1250 utilizes algorithm 108 to determine ratings 1224 of images 1222 based upon those images and matching original untouched images, as previously described.
- rating server 1250 is similar to rating server 902 of FIG. 9 .
- Browser 1202 operating on a user's computer for example, accesses web site 1220 via Internet 950 to generate a web page display 1210 .
- Browser 1202 includes a touch-up filter 1204 that compares ratings 1224 of images 1222 received from web site 1220 to a rating threshold 1206 and does not display images that have a rating greater than rating threshold 1206 .
- rating threshold 1206 has a value of 2
- image 1222 ( 1 ) has a rating 1224 ( 1 ) with a value of 1
- image 1222 ( 2 ) has a rating 1224 ( 2 ) with a value of 3.
- Touch up filter 1204 in cooperation with browser 1202 , allows image 1222 ( 1 ) to be displayed as image 1212 ( 1 ) within web page display 1210 because rating 1224 ( 1 ) is less than or equal to rating threshold 1206 .
- Touch-up filter 1204 in cooperation with browser 1202 , prevents image 1222 ( 2 ) from being displayed within web page display 1210 because rating 1224 ( 2 ) is greater than rating threshold 1206 .
- browser 1202 displays an image placeholder 1212 ( 2 ) to indicate where image 1222 ( 2 ) would have been displayed within web page display 1210 .
- FIG. 13 shows exemplary use of ratings 1310 generated by rating server 1350 to select images for display within a web page display 1322 of a browser 1320 based upon a rating selector 1308 .
- Web server 1302 stores, within a database 1306 for example, at least two retouched images 1308 ( 1 ) and 1308 ( 2 ) of the same subject. Each retouched image 1308 has a different amount of “touch-up” and thereby receive ratings 1310 , from rating server 1350 , that have different values.
- FIG. 13 shows exemplary use of ratings 1310 generated by rating server 1350 to select images for display within a web page display 1322 of a browser 1320 based upon a rating selector 1308 .
- Web server 1302 stores, within a database 1306 for example, at least two retouched images 1308 ( 1 ) and 1308 ( 2 ) of the same subject. Each retouched image 1308 has a different amount of “touch-up” and thereby receive ratings 1310
- image 1308 ( 1 ) has rating 1310 ( 1 ) with a value of 1
- image 1308 ( 2 ) has rating 1310 ( 2 ) with a value of 3, thereby indicating that image 1308 ( 2 ) has more touch-up than image 1308 ( 1 ).
- Browser 1320 running on a user's computer for example, has rating selector 1324 with a value of 2, indicating that browser should display images 1308 with rating 1310 values of 2 or less.
- An application server 1304 running on, or in communication with, web server 1302 , generates web page 1322 for display within browser 1320 based upon a value of rating selector 1324 .
- rating selector 1324 has a value of 2
- application server 1304 generates web page 1322 with image 1308 ( 1 ), and optionally the value of associated rating 1310 ( 1 ), because rating 1310 ( 1 ) has the highest value that is less than or equal to the value of rating selector 1324 .
- rating selector 1324 forms part of a parental control setting for browser 1320 such that browser 1320 does not display images, such as image 1308 ( 2 ), that have a rating (e.g., rating 1310 ( 2 )) value greater than the value of rating selector 1324 .
- a file format is used that stores images 1308 ( 1 ) and 1308 ( 2 ) with the same file together with their respective rating values.
- This file format may store an original image, and any retouches that allow generation of retouched images for a desired rating value.
- FIG. 14 shows one exemplary system 1400 for generating a rating module 1412 that automatically determines a rating for a retouched image 1474 in comparison to an original image 1472 .
- System 1400 has a server 1402 that includes an SVR training module 1404 , evaluation data 1406 , and a rating algorithm 1410 .
- Evaluation data 1406 includes a plurality of original images with a plurality of retouched images that are “touched-up” from the original images.
- Rating algorithm 1410 is characterized by characterization data 1408 , generated by SVR training module 1404 , to provide a rating evaluation of a retouched image as compared to an original image, as described above.
- Characterization data 1408 and rating algorithm 1410 in combination, form a rating module 1412 that may represent algorithm 108 .
- Rating module 1412 may be operated on other computers, such as publisher 1470 , to automatically determine a rating for a retouched image 1474 in comparison to an original image 1472 .
- server 1402 sends, via Internet 950 , evaluation data 1406 to a crowd-source server 1460 , where a plurality of evaluators 1462 evaluate evaluation data 1406 to generate results 1464 .
- Results 1464 include a rating value for each retouched image as perceived by each evaluator 1462 , together with other values of the evaluator's interaction with crowded-source server 1460 , such as one or more of time taken, a number of times the evaluator switched between each original and retouched image, and so on.
- Results 1464 are sent to server 1402 where they are processed by SVR training module 1404 to generate characterization data 1408 that controls rating algorithm 1410 to automatically rate a retouched image in comparison to an original image.
- rating module 1412 may be transferred to other computers to determine a rating of retouched images as compared to original images.
- rating module 1412 is a software package that is purchased by publisher 1470 and used to rate retouched image 1474 against original image 1472 to determine rating 1476 .
- Rating module 1412 is implemented as instructions, stored on computer readable media, that when executed by a processor determines the rating of the retouched image as described above.
- v -> ⁇ ( x , y ) ( m 1 ⁇ x + m 2 ⁇ y + t x - x m 3 ⁇ x + m 4 ⁇ y + t y - y ) .
- v -> ⁇ ( x , y ) ( m 1 ⁇ x + m 2 ⁇ y + t x - x m 3 ⁇ x + m 4 ⁇ y + t y - y ) .
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
A method is provided for automatically providing a digital image rating of photo retouching. The method includes the step of receiving at a computer, including a processor, a first set of pixel data of an original image and a second set of pixel data of a retouched image. The method also includes using the processor to determine a plurality of geometric statistics and a plurality of photometric statistics from the first and second sets of pixel data. The method further includes the step of using the processor to quantify a rating of the retouched image based upon the geometric statistics and photometric statistics to indicate deviation of the retouched image from the original image. A system is also provided to perform the steps.
Description
- This application claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 61/562,934, filed Nov. 22, 2011, which is incorporated by reference herein. This application further claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 61/606,775, filed on Mar. 5, 2012, which is incorporated by reference herein.
- This invention was made with government support under contract number cns-0708209 awarded by the national science foundation (NSF). The government has certain rights in the invention.
- In recent years, advertisers and magazine editors have been widely criticized for taking digital image retouching to an extreme Impossibly thin, tall, and wrinkle- and blemish-free models are routinely splashed onto billboards, advertisements, and magazine covers. The images, however, are often the result of digital image retouching.
- Retouched images are ubiquitous and have created an idealized and unrealistic representation of physical beauty. A significant amount of literature has established a link between these images and men's and women's satisfaction with their own physical appearance. Such concern for public health has led the American Medical Association to recently adopt a policy to “discourage the altering of photographs in a manner that could promote unrealistic expectations of appropriate body image.” Concern for public health and for the general issue of truth in advertising has also led the United Kingdom, France, and Norway to consider legislation that would require digitally altered images to be labeled.
- Popular image-editing software, such as Adobe Photoshop, allows photo editors to easily alter a person's appearance. These alterations may affect the geometry of the subject and may include slimming of legs, hips, and arms, elongating the neck, improving posture, enlarging the eyes, or making faces more symmetric. Other photometric alterations affect skin tone and texture. These changes may include smoothing, sharpening, or other operations that remove or reduce wrinkles, cellulite, blemishes, freckles, and dark circles under the eyes. A combination of geometric and photometric manipulations allows image retouchers to subtly or dramatically alter a person's appearance.
- In an embodiment, a method is provides an image retouching rating. The method includes the step of receiving at a computer including at least one processor a first set of pixel data of an original digital image and a second set of pixel data of a retouched digital image. The method also includes using the at least one processor to determine a plurality of geometric statistics and a plurality of photometric statistics from the first and second sets of pixel data. The method further includes the step of using the at least one processor to generate a rating of the retouched image based upon the geometric statistics and photometric statistics to indicate deviation of the retouched image from the original image.
- In another embodiment, the method further includes determining a plurality of geometric statistics comprising a first mean and a first standard deviation of geometric distortion of retouching, between the first and second sets of pixel data, corresponding to a first portion of a human body. In certain embodiments, the first portion of the human body is a portion of the face.
- In another embodiment, the method further includes determining a second mean and a second standard deviation of geometric distortion of retouching, between the first and second sets of pixel data, corresponding to a second portion of the human body, wherein the second portion is different from the first portion.
- In another embodiment, the method further includes determining a plurality of photometric statistics including a third mean and a third standard deviation, between the first and second sets of pixel data, of the frequency response of a linear filter corresponding to a first portion of the human body.
- In yet another embodiment, the method further includes determining a plurality of photometric statistics including a fourth mean and fourth standard deviation of structural similarity (SSIM) characterized by contrast and structural modification.
- In yet another embodiment, the step of generating includes executing, using the processor, a non-linear support vector regression (SVR) to compute the image retouching rating based upon the plurality of statistics and characterization data defining the training parameters of the SVR.
- In another embodiment, the method further includes determining, within a touchup filter, whether the retouched image is to be displayed by a browser based upon the image retouching rating and a rating threshold; displaying the retouched image within the browser when the retouched image is to be displayed, and displaying an image outline within the browser when the retouched image is not to be displayed; wherein the browser is in communication with a web server.
- In another embodiment, the method further includes defining within a browser, a rating selector that specifies a maximum image retouching rating value for retouched images to be displayed within the browser; sending the maximum image retouching rating value to a web server in communication with the browser; and receiving a webpage for display within the browser with at least one retouched image having an image retouching rating equal or less than the maximum rating value.
- A system is also provided to perform the steps in the above embodiments. Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the invention. A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings.
-
FIG. 1 is a block diagram illustrating one exemplary system for generating a perceptual rating of digital image retouching, in an embodiment. -
FIG. 2 is a flow chart illustrating of one exemplary method for determining a perceptual rating of digital image retouching using the system ofFIG. 1 , in an embodiment. -
FIG. 3 illustrates one exemplary algorithm for determining a perceptual rating of digital image retouching, in an embodiment. -
FIG. 4 illustrates five representations of five original digital images. -
FIG. 5 illustrates five representations of the five original images ofFIG. 4 after retouching. -
FIG. 6 illustrates exemplary results from a leave-one-out cross-validation using a nonlinear support vector regression (SVR) to correlate observer ratings of digital image retouching to the geometric and photometric measurements of digital image retouching. -
FIG. 7 illustrates five representations of another five original digital images. -
FIG. 8 illustrates five representations of the five original images ofFIG. 7 after retouching. -
FIG. 9 shows one exemplary system for rating and certifying a retouched digital image, in an embodiment. -
FIG. 10 shows an exemplary publication containing the retouched image and displaying a rating determined by the system ofFIG. 9 , in an embodiment. -
FIG. 11 shows one exemplary rating system configured for automatically rating observations based upon an observation model and SVR training data, in an embodiment. -
FIG. 12 shows exemplary use of ratings, generated by a rating server, by a browser when viewing a web site, in an embodiment. -
FIG. 13 shows exemplary selection of digital images for display within a web page display of a browser based upon ratings generated by rating server, in an embodiment. -
FIG. 14 shows one exemplary system for generating a rating module that automatically determines a rating for a retouched digital image in comparison to an original digital image, in an embodiment. - The present disclosure may be understood by reference to the following detailed description taken in conjunction with the drawings as described below. For purposes of illustrative clarity, certain elements in the drawings are not drawn to scale.
- This application discloses, among other things, how to automatically compute a quantitative and perceptually meaningful rating of image retouching. The application also discloses rating a photograph based upon a degree by which the photograph has been digitally altered by explicitly modeling and estimating geometric and photometric changes. This rating has been demonstrated to correlate well with perceptual judgments of image retouching and may be used to objectively judge by how much a retouched image has strayed from reality.
- Benefits of the disclosed methods and systems may include providing a perceptually relevant rating of image retouching that helps find a balance between the competing interests of editing images for selling products and discouraging photographic alterations that promote unrealistic expectation of body image. Providing an image retouching rating alongside a published image may inform the public of the extent to which images have strayed from reality. Such a rating may also provide incentive for publishers and models to reduce usage of the more extreme forms of digital image retouching that are common today. This rating may also help image retouchers and editors because, even when original and retouched images are available, it is often difficult to see and quantify the extent of image alterations.
- Benefits of the disclosed methods and systems may also include providing a fast and automatic rating to users and publishers. The core computational component of the disclosed system is typically fully automatic, however a user may annotate the hair/head, face, and body in certain embodiments. When deploying an industry-wide rating system, this annotation could either be done automatically or with fairly minimal user assistance.
- The rating quantifies the perceptual impact of geometric and photometric retouching by modeling common image retouching techniques. Geometric changes are modeled, for example, with a dense locally linear, but globally smooth, motion field. Photometric changes are modeled, in certain embodiments, with a locally linear filter and a generic measure of local image similarity (SSIM). These model parameters are typically automatically estimated from the original and retouched images. The extent of image manipulation is, for example, quantified with eight summary statistics extracted from these models.
-
FIG. 1 is a block diagram illustrating oneexemplary system 100 for generating a perceptual rating of digital image retouching.System 100 includes acomputer 112 which has at least oneprocessor 106 and analgorithm 108 comprising machine readable instructions stored in amemory 114. - In one embodiment,
algorithm 108 includes a geometric andphotometric calculator 120 for determiningstatistics 121, and a support vector regression (SVR) 122 for generatingrating 110 based uponcharacterization data 123,geometric statistics 132 andphotometric statistics 134. -
Geometric statistics 132 and photometric statistics 134 (together, statistics 121), ofFIG. 1 , are extracted by geometric andphotometric calculator 120 by comparing theoriginal image 102 to the retouchedimage 104. In one embodiment, the amount of photo distortion or perceptual distortion is quantified from eightsummary statistics 121 that are extracted from geometric and photometric models. These statistics consist of four geometric measurements (e.g., geometric statistics 132) and four photometric measurements (e.g., photometric statistics 134). - For example,
geometric statistics 132 determined bycalculator 120 may use an 8-parameter affine model that is used to model a geometric transformation between local regions in the before and after images, e.g., original and retouched images. The luminance transformation is modeled with a 2-parameter model embodying brightness and contrast. This 8-parameter model is given by: -
cf a(x, y)+b=f b(m 1 x+m 2 y+t x m 3 x+m 4 y+t y), [1] - where fb and fa are the local regions of the before and after images, c and b are contrast and brightness terms, mi are terms of a two by two affine matrix, and tx and ty are translation terms. The luminance terms c and b are incorporated only so that the geometric transformation can be estimated in the presence of luminance differences between the before and after images or the original and retouched images.
- A quadratic error function in these parameters is defined, for example, by approximating the right-hand side of Equation (1) with a first-order truncated Taylor series expansion. This error function is then minimized using standard least-squares optimization. Because these geometric parameters are estimated locally throughout the image, the resulting global transformation can lead to unwanted discontinuities. A global penalty on large motions and a smoothness constraint are imposed by penalizing the local model parameters proportional to their magnitude and the magnitude of their local gradient. The addition of this smoothness constraint requires an iterative minimization which is boot-strapped with the result of the least-squares optimization. This optimization is embedded within a coarse-to-fine differential architecture in order to contend with both large- and small-scale geometric changes. A model of missing data is also incorporated that contends with the case when portions of the after image have been entirely removed or added relative to the before image.
- Once estimated, the geometric transformation is represented as a dense two-dimensional (2D) vector field:
-
- This estimation is performed only on the luminance channel of a color image. The before and after images initially are histogram equalized to minimize any overall differences in brightness and contrast. The background in each image is replaced with white noise in order to minimize any spurious geometric distortion. This geometric model embodies the basic manipulation afforded by, for example, the Photoshop liquify tool used by photo retouchers to alter the global or local shape of a person.
- In this example, the first two geometric statistics (e.g statistics 1-2) are the mean and standard deviation of the magnitude of the estimated vector field v(x,y), shown in Equation [2], projected onto the gradient vector of the underlying luminance channel. This projection emphasizes geometric distortions that are orthogonal to image features, which are more perceptually salient. These two statistics are computed only over the face region, which quantify geometric facial distortion. The third and fourth geometric statistics (e.g statistics 3-4) are the mean and standard deviation of the magnitude of the estimated vector field v(x,y), Equation [2], projected onto the gradient vector and computed over the body region. These projected vectors are weighted based on specific body regions.
- In a particular embodiment, the bust/waist/thigh regions are weighted by a factor of 2, the head/hair regions are weighted by a factor of ½, and the remaining body regions have unit weight (a full range of weights were explored and the final results are not critically dependent on these specific values). These two statistics quantify geometric body distortion, and are computed separately from the facial distortion because observers weight facial and body distortions differently.
-
Photometric statistics 134 are for example determined bycalculator 120 using either a linear filter model or a similarity measure SSIM. For example, basic photometric retouches between local regions in the after image, i.e. the retouched image, and the geometrically aligned before image, i.e. the original image, are modeled with a linear filter h(x,y), of size nine by nine pixels, given by: -
f a(x, y)=h(x, y)★{tilde over (f)} b(x, y), [3] - where ★ is the convolution operator, and fb is the geometrically aligned before image region, Equation (1). The filter h(x,y) is estimated locally using a conjugate gradient descent optimization with a Tikhonov regularization. The regularization is used to enforce symmetry (i.e., zero-phase) on the estimated filter h. This estimation is performed only on the luminance channel of a color image.
- Photometric retouches that are not captured by Equation [3] are measured with the similarity measure SSIM. This measure embodies contrast and structural retouches C(x,y) as follows:
-
- where σa and σb are the standard deviations of the image regions fa and ˜fb, and σab is the covariance of fa and ˜fb. The various constants are β=1, γ=1, C2=(0.03)2, and C3=C2/2. Note that in this implementation of SSIM the brightness term is excluded because it does not impact observers' judgments. For the same reason, SSIM is computed only on the luminance channel of a color image. This photometric model embodies special effects afforded by various Photoshop filters.
- In this particular example, the first two photometric statistics (e.g. statistics 5-6) are the mean and standard deviation of the structural similarity (SSIM) or contrast and structural modification C(x,y), in Equation [4], computed over the face region. The third and fourth photometric statistics (e.g. statistics 7-8) are a measure D of the frequency response of the linear filters h(x,y), shown in Equation [3]. D is expressed as the following:
-
D='ω |{tilde over (F)} b(ω)|ω−Σω |H(ω){tilde over (F)} b(ω)|ω, [6] - where H(ω) and ˜Fb(ω) are unit-sum normalized one-dimensional (1D) frequency responses of the filter h and the local region ˜fb which are computed by integrating their 2D Fourier transforms across orientation. The parameter D is positive when h is a blurring filter, negative when h is a sharpening filter, and is tailored to the analysis of people in which filtering is commonly used to remove or enhance facial features. The mean and standard deviation of parameter D, computed over the face region, are the final two statistics.
- In summary, a total of eight summary statistics have been described. The first four geometric statistics are mean and standard deviation of the estimated vector field computed separately over the face and body. The second four photometric statistics are the mean and standard deviation of SSIM and the frequency response of the linear filters.
-
Characterization data 123 is, for example, defines a set of training parameters used to train theSVR model 122 used byalgorithm 108. To generatecharacterization data 123, for example, four hundred sixty eight before/after images were collected from a variety of on-line resources, primarily the websites of photo retouchers showcasing their services. These images spanned the range from minor to radical amounts of retouching. - A group of 390 observers was recruited to perform perceptual ratings through Amazon's Mechanical Turk. This crowd sourcing utility has become popular among social scientists as a way to quickly collect large amounts of data from human observers around the world. Observers were initially shown a representative set of 20 before/after images in order to help them gauge the range of distortions they could expect to see. Observers were then shown 70 pairs of before/after images and asked to rate how different the person looked between the images on a scale of “1” to “5”. A score of “1” means “very similar” and a score of “5” means “very different.” This yielded a total of 50 ratings per each of 468 images. The presentation of images was self-timed and observers could manually toggle between the before and after images as many times as they chose (observers are better able to see the amount of retouching within the images when toggling rather than viewing side-by-side). In order to measure the consistency of observer responses each observer rated a random set of five images three times each. The presentation of these images was evenly distributed throughout the trial. Each observer was paid $3 for their participation and a typical session lasted 30 min. Given the uncontrolled nature of the data collection, some data filtering was necessary. Approximately 9.5% of observers were excluded because they frequently toggled only once between the before and after image and they responded with high variance on the repeated trials.
- The SVR technique was used to generate
characterization data 123 as a mapping between the observer ratings and eight summary statistics (e.g., statistics 121) extracted from the geometric and photometric models of photo retouching atstep 324. Each statistic was individually scaled into the range [31 1, 1]. Specifically, a nonlinear SVR with a Gaussian radial basis kernel was employed. A leave-one-out cross-validation was performed in which the SVR was trained on 467 of 468 image ratings and tested on the remaining image. This training and testing was repeated 468 times in which each image was individually tested. The SVR has two primary degrees of freedom: (1) the scalar y specifies the spatial extent of the kernel function; and (2) the scalar c specifies the penalty applied to deviations of each data point from the regression function. These parameters were selected by performing a dense 2D grid search to maximize the correlation coefficient of each training set. The results of the above crowd sourcing technique may then be stored intomemory 114 ascharacterization data 123. -
Processor 106 executesalgorithm 108 to process originaldigital image data 102 and retoucheddigital image data 104. Those skilled in the art would appreciate thatalgorithm 108 may be executed bymultiple processors 106.Algorithm 108 further generates an output arating 110 indicating deviation of the retoucheddigital image data 104 from the originaldigital image data 102.Output rating 110 may be stored withinmemory 114, or transferred as an output to exterior devices other thancomputer 112. In one embodiment,algorithm 108 also includes an eight parameter model.Memory 114 may represent one or more of a magnetic or optical disk and/or an electronic memory integrated circuit (e.g., RAM, ROM, FLASH, and so on). -
FIG. 2 is a flow chart illustrating of oneexemplary method 200 for determining a perceptual rating of digital image retouching usingsystem 100 ofFIG. 1 .Method 200 is for example implemented, at least in part, withinalgorithm 108 ofcomputer 112. Instep 202,method 200 receives, withincomputer 112, original digital image data and retouched digital image data. Instep 204, geometric and photometric statistics (e.g., statistics 121) are for example determined usingprocessor 106. In step 206,method 200 usesalgorithm 108, for example an SVR algorithm, to determine arating 110 for the retouched image based upon thestatistics 121 andcharacterization data 123. Step 206 is performed, for example, at least partially by usingprocessor 106. Instep 208,method 200 reports therating 110 of the retouched image. -
FIG. 3 shows representations of an originaldigital image 302, a retoucheddigital image 304, and a visualization of the steps required to create a rating of digital image retouching, for example, by measuring geometric and photometric retouching. Animage 306 represents measured geometric distortions that are depicted as a vector field, where a superimposed color coding corresponds to vector magnitude. For clarity of illustration, area 332 near the breast andarea 334 near the waist are shown in dashed outline and represent areas with higher vector magnitude numbers that correspond to greater distortion. Animage 308, also withinFIG. 3 , represents measured photometric distortions quantified by a linear filter, in whicharea 340 near the cheek has positive values denoting blurring andarea 338 near the eye has negative values denoting sharpening. Animage 310 represents structural similarity (SSIM) whereareas - Step 312 uses Equation [1] to measure geometric differences for face and body with eight parameters and then computes vector field based upon six of the eight parameters using Equation [2]. Step 318 computes mean and standard deviation of magnitude of the vector in face and body. Step 314 uses Equation [3] to measure photometric differences for the face and then uses Equations [6] to calculate the
measure D. Step 320 computes mean and standard deviation ofD. Step 316 uses Equations [4] and [5] to measure photometric differences for the face and computes C(x,y). Step 322 computes mean and standard deviation of C(x,y). - In
step 324, machine learning techniques, such as SVR, are then used to relate these summary statistics to perceptual judgments made by human observers. It is shown below, inFIGS. 4-8 , that the summary statistics combine to yield a metric that correlates well with perceptual ratings of photo alteration. -
FIG. 5 illustrates five representative retoucheddigital images digital images FIG. 4 . Shown along the top is how an average person rated the differences in these images (left), the predicted rating of photo retouching (right) using the present algorithm, and their difference (in parenthesis). An image with little to no retouching is rated a “1” while an image with significant retouching is rated a “5”. The value in the parentheses is the difference between these two values. -
FIG. 6 illustrates the correlation between the mean of 50 observer ratings per image and one exemplary of the present algorithm. Each data point corresponds to one of 468 images rated by the test group on a scale of 1 to 5. The predicted rating for each image was determined by training an SVR on 467 images using a leave-one-out cross-validation methodology. The R-value is 0.80, the mean/median absolute prediction error is 0.30/0.24 with a standard deviation of 0.24 and a max absolute error of 1.19. The absolute prediction error is below 0.5 for 81.4% of the images, and below 0.75 and 1.0 for 94.4% and 99.1% of the images, respectively. -
FIG. 8 illustrates another set of representative retoucheddigital images digital images FIG. 7 . A model of missing data is also incorporated that contends with the case when portions of the after image have been entirely removed or added relative to the before image. The over- and underestimations illustrate some of the limitations of the present model. The perceptual distortion in retoucheddigital image 802 is overestimated because there is a large photometric difference for the young boy (removal of blemishes). The perceptual distortion in retoucheddigital image 804 is overestimated because there is and a large geometric difference for the young woman (change in shape and position of the head). However, neither of these differences corresponds to a large perceptual difference in appearance. On the other hand, the perceptual distortion in the remaining retoucheddigital images - To determine which of the eight summary statistics were most critical for predicting observer ratings, we trained and tested 255 SVRs, one for each possible subset of
size 1 to 8. The best performing SVR with one statistic consisted of the mean of the geometric facial distortion (statistic (1) as described above), which yielded an R-value of 0.58. The best performing SVR with two statistics consisted of the standard deviation of the geometric body distortion and the standard deviation of the photometric SSIM (statistics (4) and (6)), which yielded an R-value of 0.69. And, the best performing SVR with three statistics consisted of adding the standard deviation of the geometric facial distortion to the previous SVR (statistics (4), (5), and (6)), which yielded an R-value of 0.76. The best performing SVR of size 6 had an R-value of 0.80, equal to that of the full set of size 8. This subset of size 6 consisted of the statistics (1), (2), (4), (6), (7), and (8) as described above. Although six statistics are sufficiently powerful, they are extracted from each component of the geometric and photometric models. Therefore, there is little cost in using all eight statistics in terms of computational complexity or in terms of training the SVR. - A linear SVR was also tested to validate the use of a nonlinear SVR over a simpler linear SVR. The R-value for the linear SVR is 0.72, as compared to 0.80 for the nonlinear SVR. The mean absolute prediction error is 0.34 with a standard deviation of 0.27 as compared to 0.30 and 0.24 for the nonlinear SVR. The max absolute error jumps from 1.19 to 1.93. Overall, the nonlinear SVR affords a considerably better prediction of observer ratings as compared to a linear SVR.
- The perceptual rating of the present disclosure is compared against two standard image similarity perceptual ratings. A perceptual rating based only on the mean and standard deviation of a standard application of SSIM yields an R-value of 0.52 as compared to our approach that had an R-value of 0.80. A perceptual rating based on only the mean squared error between the before and after image performed much worse with an R-value of only 0.30. Standard image similarity perceptual ratings perform poorly because they do not compensate for, or measure, large-scale geometric distortions.
- In the above embodiments, the system compares original and retouched images of human subjects. However, the system may also be trained for comparing images of other subjects. For example, the system may be trained to compare images of landscape portraits, still-life, and wildlife. Similarly, the system may be trained to compare images of art restorations. For example, a rating of restored art may be used for evaluation purposes, wherein a poorly restored image may be awarded a high rating (indicating significant change to the image content).
-
FIG. 9 shows oneexemplary system 900 for rating and certifying a retoucheddigital image 924.System 900 includes aserver 902, coupled with theInternet 950, and having analgorithm 904 and acertificate generator 906.Algorithm 904 may representalgorithm 108 and may implementmethod 200 ofFIG. 2 .Certificate generator 906 includes an encryption, and optionally a decryption, algorithm that utilizes aprivate key 932 of a public/privatekey pair 930, to generate acertificate 908 containing arating 910.Certificate 908 may then be decrypted usingpublic key 934 of public/privatekey pair 930. -
Server 902 may represent one or more computers that are connected toInternet 950. For example,server 902 includes a memory and a processor (not shown for clarity of illustration) that respectively store and execute instructions ofalgorithm 904 and instructions ofcertificate generator 906. - In one example of operation, a
publisher 920 desires to publish a retoucheddigital image 924 together with a certified rating that defines an amount of retouching applied to anoriginal image 922 to form retoucheddigital image 924.Publisher 920 sends amessage 926, containingoriginal image 922 and retouchedimage 924, toserver 902 requesting a certified rating of retouchedimage 924. Withinserver 902,algorithm 904 is executed to processoriginal image 922 and retouchedimage 924 and to generate arating 910.Certificate generator 906 utilizes private key 932 (known only to certificate generator 906) to generatecertificate 908 containingrating 910 and animage ID 912 that uniquely identifies retouchedimage 924. In one example,image ID 912 is a checksum of data within retouchedimage 924. In another example,image ID 912 is retouchedimage 924. - Once
certificate 908 is generated,server 902 sendscertificate 908 topublisher 920 asmessage 914 for example.Message 914 may also containpublic key 934.Publisher 920 decryptscertificate 908 usingpublic key 934 and optionally verifies thatrating 910 applies to retouchedimage 924 usingimage ID 912.Publisher 920 may retaincertificate 908 as evidence ofrating 910 for retouchedimage 924, for example, to show compliance to a rating standard. -
FIG. 10 shows anexemplary publication 1002 generated bypublisher 920 ofFIG. 9 and containing retoucheddigital image 924 withrating 910 overlaid on the image. In one embodiment,system 900 retouches and returns retouchedimage 924 to includerating 910. Alternatively,rating 910 may be shown beside the image. In one embodiment, wherepublication 1002 is a web page, the user may click onrating 910 to retrieve and validate the rating certificate associated withrating 910. In another example,rating 910 is not shown unless the user clicks on (or causes a mouse-over event of) retouchedimage 924 whereinrating 910 is momentarily displayed. - The rating system may also be applied to other human judgments, wherein the system may predict how an object may be judged. For example, the rating may be applied to judging a photography contest, and rating images for a scientific journal.
-
FIG. 11 shows oneexemplary rating system 1100 that may be configured for automatically ratingobservations 1110 based upon anobservation model 1104 andSVR training data 1106.Observation model 1104 defined weightings for each of a plurality of features (data points) within digital images ofobservations 1110.Analyzer 1102 processesobservations 1110 based uponobservation model 1104 and then appliesSVR training data 1106 to determine arating 1120. -
Observation model 1104 may be defined for any observable set of features that a user of system wishes to automatically rate. Using the example shown above (FIGS. 1-10 ),observations 1110 include retouched features of a human as determined by comparing an original digital image and a retouched digital image.Observation model 1104 is thus used to define the features within the images that are of interest, and defines a weighting for each of the features.Observation model 1104 may define parameters for features that are not always present withinobservations 1110, but that are applied when the feature is present. - The mathematical model may also be applied in reverse wherein a user may automatically retouch an image to have a desired rating. For example, where a magazine desires to publish only lightly retouched images, the
system 1100 may be used to reduce the amount of retouching of a heavily retouched image until that image has a rating of 1, thereby meeting the magazine publishing criteria. Sincesystem 900 measures geometric and photometric change between an original digital image and a retouched image,system 900 may be used to reduce the level of retouching of the retouched image until a determined rating thereof is at a desired rating level. Specifically, the geometric and photometric models are generative in nature, and define how an image is retouched from the original digital image to get to the retouched digital image. -
FIG. 12 shows exemplary use ofratings 1224, generated by arating server 1250, by abrowser 1202 when viewing aweb site 1220.Web site 1220 is shown with two exemplary retouched images 1222(1) and 1222(2) that each have a rating 1224(1) and 1224(2), respectively, generated byrating server 1250.Rating server 1250 utilizesalgorithm 108 to determineratings 1224 ofimages 1222 based upon those images and matching original untouched images, as previously described. In one embodiment,rating server 1250 is similar torating server 902 ofFIG. 9 . -
Browser 1202, operating on a user's computer for example, accessesweb site 1220 viaInternet 950 to generate aweb page display 1210.Browser 1202 includes a touch-up filter 1204 that comparesratings 1224 ofimages 1222 received fromweb site 1220 to arating threshold 1206 and does not display images that have a rating greater thanrating threshold 1206. - In the example of
FIG. 12 ,rating threshold 1206 has a value of 2, image 1222(1) has a rating 1224(1) with a value of 1, and image 1222(2) has a rating 1224(2) with a value of 3. Touch upfilter 1204, in cooperation withbrowser 1202, allows image 1222(1) to be displayed as image 1212(1) withinweb page display 1210 because rating 1224(1) is less than or equal torating threshold 1206. Touch-up filter 1204, in cooperation withbrowser 1202, prevents image 1222(2) from being displayed withinweb page display 1210 because rating 1224(2) is greater thanrating threshold 1206. In one embodiment,browser 1202 displays an image placeholder 1212(2) to indicate where image 1222(2) would have been displayed withinweb page display 1210. -
FIG. 13 shows exemplary use ofratings 1310 generated byrating server 1350 to select images for display within aweb page display 1322 of abrowser 1320 based upon arating selector 1308.Web server 1302 stores, within adatabase 1306 for example, at least two retouched images 1308(1) and 1308(2) of the same subject. Each retouchedimage 1308 has a different amount of “touch-up” and thereby receiveratings 1310, fromrating server 1350, that have different values. In the example ofFIG. 13 , image 1308(1) has rating 1310(1) with a value of 1, and image 1308(2) has rating 1310(2) with a value of 3, thereby indicating that image 1308(2) has more touch-up than image 1308(1). -
Browser 1320, running on a user's computer for example, hasrating selector 1324 with a value of 2, indicating that browser should displayimages 1308 withrating 1310 values of 2 or less. Anapplication server 1304, running on, or in communication with,web server 1302, generatesweb page 1322 for display withinbrowser 1320 based upon a value ofrating selector 1324. In the example ofFIG. 13 ,rating selector 1324 has a value of 2, andapplication server 1304 generatesweb page 1322 with image 1308(1), and optionally the value of associated rating 1310(1), because rating 1310(1) has the highest value that is less than or equal to the value ofrating selector 1324. - In one example of operation,
rating selector 1324 forms part of a parental control setting forbrowser 1320 such thatbrowser 1320 does not display images, such as image 1308(2), that have a rating (e.g., rating 1310(2)) value greater than the value ofrating selector 1324. - In one embodiment, a file format is used that stores images 1308(1) and 1308(2) with the same file together with their respective rating values. This file format may store an original image, and any retouches that allow generation of retouched images for a desired rating value.
-
FIG. 14 shows oneexemplary system 1400 for generating arating module 1412 that automatically determines a rating for a retouchedimage 1474 in comparison to anoriginal image 1472.System 1400 has aserver 1402 that includes an SVR training module 1404,evaluation data 1406, and arating algorithm 1410.Evaluation data 1406 includes a plurality of original images with a plurality of retouched images that are “touched-up” from the original images.Rating algorithm 1410 is characterized bycharacterization data 1408, generated by SVR training module 1404, to provide a rating evaluation of a retouched image as compared to an original image, as described above.Characterization data 1408 andrating algorithm 1410, in combination, form arating module 1412 that may representalgorithm 108.Rating module 1412 may be operated on other computers, such aspublisher 1470, to automatically determine a rating for a retouchedimage 1474 in comparison to anoriginal image 1472. - In one example of operation,
server 1402 sends, viaInternet 950,evaluation data 1406 to a crowd-source server 1460, where a plurality ofevaluators 1462 evaluateevaluation data 1406 to generateresults 1464.Results 1464 include a rating value for each retouched image as perceived by eachevaluator 1462, together with other values of the evaluator's interaction with crowded-source server 1460, such as one or more of time taken, a number of times the evaluator switched between each original and retouched image, and so on.Results 1464 are sent toserver 1402 where they are processed by SVR training module 1404 to generatecharacterization data 1408 that controlsrating algorithm 1410 to automatically rate a retouched image in comparison to an original image. - Once
characterization data 1408 andrating algorithm 1410 are combined to formrating module 1412,rating module 1412 may be transferred to other computers to determine a rating of retouched images as compared to original images. In the example ofFIG. 14 ,rating module 1412 is a software package that is purchased bypublisher 1470 and used to rate retouchedimage 1474 againstoriginal image 1472 to determine rating 1476.Rating module 1412 is implemented as instructions, stored on computer readable media, that when executed by a processor determines the rating of the retouched image as described above. - Having described several embodiments, it will be recognized by those skilled in the art that various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the invention. Additionally, a number of well known processes and elements have not been described in order to avoid unnecessarily obscuring the present invention. Accordingly, the above description should not be taken as limiting the scope of the invention.
- It should thus be noted that the matter contained in the above description or shown in the accompanying drawings should be interpreted as illustrative and not in a limiting sense. For example, the above example describe processing of two (original and retouched) digital images; however, the systems and methods may also be used to process video images, which may be considered a sequence of still images. The following claims are intended to cover generic and specific features described herein, as well as all statements of the scope of the present method and system, which, as a matter of language, might be said to fall there between. In particular, the following embodiments are specifically contemplated, as well as any combinations of such embodiments that are compatible with one another:
-
- A. A method for automatically providing an image retouching rating including: receiving, within a computer having a processor and memory, a first set of pixel data of an original image and a second set of pixel data of a retouched image; determining, from the first and second sets of pixel data using the processor, a plurality of statistics based upon deviation of the retouched image from the original image; and generating, using the processor, the image retouching rating to define a level of modification of the retouched image based upon the plurality of statistics.
- B. The method of embodiment A, wherein the step of determining the plurality of statistics comprises determining a plurality of geometric statistics and determining a plurality of photometric statistics.
- C. The method of embodiments A or B, wherein the step of determining the plurality of statistics includes determining a plurality of geometric statistics comprising a first mean and a first standard deviation of geometric distortion of retouching, between the first and second sets of pixel data, corresponding to the first portion of a human body.
- D. The method of embodiments A through C, wherein the step of determining the plurality of statistics includes determining a plurality of geometric statistics including a second mean and a second standard deviation of geometric distortion of retouching, between the first and second sets of pixel data, corresponding to a second portion of the human body, wherein the second portion is different from the first portion.
- E. The method of embodiments C or D, wherein the first portion of the human body is a portion of the face.
- F. The method of embodiments A through E, wherein the step of determining the plurality of statistics further includes executing, using the processor, an algorithm stored as computer readable instructions within the memory, to model a geometric transformation between the first and second sets of pixel data corresponding to respective local regions in the original image and the retouched image.
- G. The method of embodiment F, wherein the algorithm includes an eight parameter model expressed by:
-
cf a(x, y)+b=f b(m 1 x+m 2 y+t x m 3 x+m 4 y+t y) -
-
- wherein fa(x,y) and fb(x,y) are local regions of the original image and the retouched image, respectively, c and b are contrast and brightness terms, mi are terms of a two by two affine matrix, and tx and ty are translation terms.
- H. The method of embodiments C through G, wherein the geometric distortion is expressed as a magnitude of a vector field v(x,y), wherein vector field v(x,y) is expressed by
-
-
-
- I. The method of embodiments A through H, wherein the step of determining the plurality of statistics includes determining a plurality of photometric statistics comprising a third mean and a third standard deviation, between the first and second sets of pixel data, of the frequency response of a linear filter corresponding to a first portion of the human body.
- J. The method of embodiment I, wherein the frequency response is characterized by,
-
D=Σ ω |{tilde over (F)} b(ω)|ω−Σω |H(ω){tilde over (F)} b(ω)|ω, -
-
- wherein H(ω) and ˜Fb(ω) are unit-sum normalized one-dimensional (1D) frequency responses of the linear filter h(x,y) and a local region ˜fb, corresponding to the first portion of the human body, which are computed by integrating their 2D Fourier transforms across orientation,
- wherein each linear filter h(x,y) is characterized by:
-
-
f a(x, y)=h(x, y)★{tilde over (f)} b(x, y), -
-
- wherein ★ is a convolution operator, and fa and fb are geometrically aligned after and before image regions, respectively.
- K. The method of embodiments A through J, wherein the step of determining the plurality of statistics includes determining a plurality of photometric statistics comprising a fourth mean and a fourth standard deviation of structural similarity (SSIM) characterized by contrast and structural modification.
- L. The method of embodiment K, wherein the SSIM is characterized by contrast and structural modification C(x,y) as follows:
-
-
C(x, y)=c(x, y)β s(x, y)γ, -
-
-
- wherein σa, σb are the standard deviations of image regions fa and ˜fb, and σab is the covariance of fa and ˜fb.
- M. The method of embodiments A through L, wherein the step of generating includes executing, using the processor, a non-linear support vector regression (SVR) to compute the image retouching rating based upon the plurality of statistics and characterization data defining the training parameters of the SVR.
- N. The method of embodiments A through M, wherein the characterization data is defined by a plurality of perceived ratings received from a plurality of users each viewing a plurality of selected original images and a plurality of retouched images.
- O. The method of embodiments A through N, further including: determining, within a touchup filter, whether the retouched image is to be displayed by a browser based upon the image retouching rating and a rating threshold; displaying the retouched image within the browser when the retouched image is to be displayed; and displaying an image outline within the browser when the retouched image is not to be displayed; wherein the browser is in communication with a web server.
- P. The method of embodiments A through O, further including: defining, within a browser, a rating selector that specifies a maximum image retouching rating value for retouched images to be displayed within the browser; sending the maximum image retouching rating value to a web server in communication with the browser; and receiving a web page for display within the browser with at least one retouched image having an image retouching rating equal or less than the maximum rating value.
- Q. A system for generating a digital image retouching rating of a retouched image in comparison to an original image, the system including: a processor in communication with memory; an algorithm stored within the memory as computer readable instructions and executed by the processor to: (i) process the original image and the retouched image to determine a plurality of geometric statistics and a plurality of photometric statistics of differences between the original image and the retouched image, and (ii) generate the digital image retouching rating based upon at least the plurality of geometric statistics and the plurality of photometric statistics.
- R. The system of embodiment Q, wherein the algorithm further includes computer readable instructions stored in the memory and executed by the processor to: (iii) model the plurality of geometric statistics with a motion field; (vi) model the plurality of photometric statistics with a generic measure of local image similarity; and (v) extract four geometric measurements from the modeled geometric statistics and four photometric measurements from the modeled photometric statistics; wherein the algorithm generates the digital image retouching rating using a non-linear support vector regression (SVR) and the four geometric measurements, the four photometric measurements, and characterization data defining training parameters of the SVR.
- S. The system of embodiments Q or R, further including computer readable instructions stored in memory and executed by the processor to issue the digital image retouching rating within a digitally signed certificate that identifies the retouched image.
- T. The system of embodiments Q through S, further including computer readable instructions stored in memory and executed by the processor to embed the digital image retouching rating within the retouched image.
- U. The system of embodiments Q through T, wherein the plurality of geometric statistics includes a first mean and a first standard deviation of a geometric distortion between the original image and the retouched image corresponding to a first portion of a human body.
- V. The system of embodiment U wherein the first portion of the human body is a portion of the face.
- W. The system of embodiments Q through U, wherein the plurality of geometric statistics includes a second mean and a second standard deviation of a geometric distortion between the original image and the retouched image corresponding to a second portion of the human body, the second portion being different from the first portion.
- X. The system of embodiments Q through W, wherein the processor executes the algorithm to implement an eight parameter model expressed by:
-
-
cf a(x, y)+b=f b(m 1 x+m 2 y+t x m 3 x+m 4 y+t y) -
-
- wherein fa(x,y) and fb(x,y) are local regions of the original image and the retouched image, respectively, c and b are contrast and brightness terms, mi are terms of a two by two affine matrix, and tx and ty are translation terms.
- Y. The system of embodiments U through X, wherein the geometric distortion is expressed by magnitude of a vector v(x,y) is expressed by
-
-
-
- Z. The system of embodiments Q through Y, wherein the plurality of photometric statistics includes a third mean and a third standard deviation, between the original image and the retouched image, of the frequency response of a linear filter for the first portion of the human body.
- AA. The system of embodiment Z, wherein the frequency response is characterized by:
-
D=Σ ω |{tilde over (F)} b(ω)|ω−Σω |H(ω){tilde over (F)} b(ω)|ω, -
-
- wherein H(ω) and ˜Fb(ω) are unit-sum normalized one-dimensional (1D) frequency responses of the linear filter h(x,y) and a local region ˜fb which are computed by integrating their 2D Fourier transforms across orientation, and
- wherein each linear filter h(x,y) is characterized by:
-
-
f a(x, y)=h(x, y)★{tilde over (f)} b(x, y), -
-
- wherein where ★ is the convolution operator, and fa and fb are a geometrically aligned after and before image region, respectively.
- BB. The system of embodiments Q through AA, wherein the plurality of photometric statistics comprises a fourth mean and a fourth standard deviation of structural similarity (SSIM) between the original image and the retouched image.
- CC. The system of claim BB, wherein the SSIM is characterized by contrast and structural modification C(x,y) as follows:
-
-
-
-
- wherein σa, σb are the standard deviations of image regions fa and ˜fb, and σab is the covariance of fa and ˜fb.
- DD. The system of embodiments Q through CC, wherein the algorithm includes a non-linear support vector regression (SVR) to compute the image retouching rating based upon the plurality of statistics and characterization data.
- EE. The system of embodiment DD, wherein the characterization data is defined from a perceptual rating defined by each of a plurality of users for each of a plurality of original image and retouched image pairs.
-
Claims (31)
1. A method for automatically providing an image retouching rating comprising:
receiving, within a computer having a processor and memory, a first set of pixel data of an original image and a second set of pixel data of a retouched image;
determining, from the first and second sets of pixel data using the processor, a plurality of statistics based upon deviation of the retouched image from the original image; and
generating, using the processor, the image retouching rating to define a level of modification of the retouched image based upon the plurality of statistics.
2. The method of claim 1 , wherein the step of determining the plurality of statistics comprises determining a plurality of geometric statistics and determining a plurality of photometric statistics.
3. The method of claim 2 , wherein the step of determining the plurality of geometric statistics comprises:
determining a first mean and a first standard deviation of geometric distortion of retouching, between the first and second set of pixel data, corresponding to a first portion of a human body
4. The method of claim 3 , wherein the step of determining the plurality of geometric statistics comprises:
determining a second mean and a second standard deviation of geometric distortion of retouching, between the first and second sets of pixel data, corresponding to a second portion of the human body, wherein the second portion is different from the first portion.
5. The method of claim 4 , wherein the first portion of the human body is a portion of the face.
6. The method of claim 5 , wherein the step of determining the plurality of geometric statistics further comprises executing, using the processor, an algorithm stored as computer readable instructions within the memory, to model the geometric transformation between the first and second sets of pixel data corresponding to respective local regions in the original image and the retouched image.
7. The method of claim 6 , wherein the step of determining the plurality of photometric statistics comprises determining a third mean and a third standard deviation, between the first and second sets of pixel data, of the frequency response of a linear filter corresponding to the first portion of the human body.
8. The method of claim 7 , wherein the step of determining the plurality of photometric statistics comprises determining a fourth mean and a fourth standard deviation of structural similarity (SSIM) characterized by contrast and structural modification.
9. The method of claim 8 , wherein the step of generating comprises executing, using the processor, a non-linear support vector regression (SVR) to compute the image retouching rating based upon the plurality of statistics, and characterization data defining the training parameters of the SVR.
10. The method of claim 9 , wherein the characterization data is defined by a plurality of perceived ratings received from a plurality of users each viewing a plurality of selected original images and a plurality of retouched images.
11. The method of claim 10 , further comprising:
determining, within a touchup filter, whether the retouched image is to be displayed by a browser based upon the image retouching rating and a rating threshold;
displaying the retouched image within the browser when the retouched image is to be displayed; and
displaying an image outline within the browser when the retouched image is not to be displayed;
wherein the browser is in communication with a web server.
12. The method of claim 11 , further comprising:
defining, within the browser, a rating selector that specifies a maximum image retouching rating value for retouched images to be displayed within the browser;
sending the maximum image retouching rating value to the web server; and
receiving a web page for display within the browser with at least one retouched image having an image retouching rating equal or less than the maximum rating value.
13. The method of claim 12 , wherein the algorithm comprises an eight parameter model expressed by:
cf a(x, y)+b=f b(m 1 x+m 2 y+t x m 3 x+m 4 y+t y)
cf a(x, y)+b=f b(m 1 x+m 2 y+t x m 3 x+m 4 y+t y)
wherein fa(x,y) and fb(x,y) are local regions of the original image and the retouched image, respectively, c and b are contrast and brightness terms, mi are terms of a two by two affine matrix, and tx and ty are translation terms.
14. The method of claim 13 , wherein the geometric distortion is expressed as a magnitude of a vector field v(x,y), wherein vector field v(x,y) is expressed by
15. The method of claim 14 , wherein the frequency response is characterized by,
D=Σ ω |{tilde over (F)} b(ω)|ω−Σω |H(ω){tilde over (F)} b(ω)|ω,
D=Σ ω |{tilde over (F)} b(ω)|ω−Σω |H(ω){tilde over (F)} b(ω)|ω,
wherein H(ω) and ˜Fb(ω) are unit-sum normalized one-dimensional (1D) frequency responses of a linear filter h(x,y) and a local region ˜fb, corresponding to the first portion of the human body, which are computed by integrating their 2D Fourier transforms across orientation,
wherein each linear filter h(x,y) is characterized by:
f a(x, y)=h(x, y)★{tilde over (f)} b(x, y),
f a(x, y)=h(x, y)★{tilde over (f)} b(x, y),
wherein where ★ is a convolution operator, and fa and fb are geometrically aligned after and before image regions, respectively.
16. The method of claim 15 , wherein the SSIM is characterized by contrast and structural modification C(x,y) as follows:
wherein σa, σb are the standard deviations of image regions fa and ˜fb, and σab is the covariance of fa and ˜fb.
17. A system for generating a digital image retouching rating of a retouched image in comparison to an original image, the system comprising:
a processor in communication with memory;
an algorithm stored within the memory as computer readable instructions and executed by the processor to:
(i) process the original image and the retouched image to determine a plurality of geometric statistics and a plurality of photometric statistics of differences between the original image and the retouched image, and
(ii) generate the digital image retouching rating based upon the plurality of geometric statistics and the plurality of photometric statistics.
18. The system of claim 17 , wherein the algorithm further comprises computer readable instructions stored in the memory and executed by the processor to:
(iii) model the plurality of geometric statistics with a motion field;
(vi) model the plurality of photometric statistics with a generic measure of local image similarity; and
(v) extract four geometric measurements from the modeled geometric statistics and four photometric measurements from the modeled photometric statistics;
wherein: the algorithm generates the digital image retouching rating using a non-linear support vector regression (SVR) and the four geometric measurements, the four photometric measurements, and characterization data defining the training parameters of the SVR.
19. The system of claim 18 further comprising computer readable instructions stored in memory and executed by the processor to issue the digital image retouching rating within a digitally signed certificate that identifies the retouched image.
20. The system of claim 19 further comprising computer readable instructions stored in memory and executed by the at least one processor to imbed the digital image rating within the retouched image.
21. The system of claim 20 , wherein the plurality of geometric statistics comprises:
a first mean and a first standard deviation of a geometric distortion in the retouched image of a first portion of a human body.
22. The system of claim 21 , wherein the plurality of geometric statistics comprises:
a second mean and a second standard deviation of a geometric distortion in the retouched image of a second portion of the human body, the second portion being different from the first portion.
23. The system of claim 22 , wherein the first portion of the human body is a portion of the face.
24. The system of claim 23 , wherein the plurality of photometric statistics comprises a third mean and a third standard deviation of photometric differences for the first portion of the human body.
25. The system of claim 24 , wherein the plurality of photometric statistics comprises a fourth mean and a fourth standard deviation of structural similarity (SSIM) between the original image and the retouched image.
26. The system of claim 25 , wherein the algorithm comprises a non-linear support vector regression (SVR) to compute the image retouching rating based upon the plurality of statistics and characterization data.
27. The system of claim 26 , wherein the characterization data is defined from a perceptual rating defined by each of a plurality of users for each of a plurality of original image and retouched image pairs.
28. The system of claim 27 , wherein the at least one processor executes the algorithm to implement an eight parameter model expressed by:
cf a(x, y)+b=f b(m 1 x+m 2 y+t x m 3 x+m 4 y+t y);
cf a(x, y)+b=f b(m 1 x+m 2 y+t x m 3 x+m 4 y+t y);
wherein fa(x,y) and fb(x,y) are local regions of the original image and the retouched image, respectively, c and b are contrast and brightness terms, mi are terms of a two by two affine matrix, and tx and ty are translation terms.
29. The system of claim 28 , wherein the geometric distortion is expressed by magnitude of a vector v(x,y) is expressed by
30. The system of claim 29 , wherein the photometric differences are characterized by frequency responses of linear filters,
D=Σ ω |{tilde over (F)} b(ω)|ω−Σω |H(ω){tilde over (F)} b(ω)|ω,
D=Σ ω |{tilde over (F)} b(ω)|ω−Σω |H(ω){tilde over (F)} b(ω)|ω,
wherein H(ω) and ˜Fb(ω) are unit-sum normalized one-dimensional (1D) frequency responses of a linear filter h(x,y) and the local region ˜fb which are computed by integrating their 2D Fourier transforms across orientation, wherein each linear filter h(x,y) is characterized by:
f a(x, y)=h(x, y)★f b(x, y),
f a(x, y)=h(x, y)★f b(x, y),
wherein where ★ is the convolution operator, and fa and fb are the geometrically aligned after and before image region, respectively.
31. The system of claim 30 , wherein the SSIM is characterized by contrast and structural modification C(x,y) as follows:
wherein σa, σb are the standard deviations of image regions fa and −fb, and σab is the covariance of fa and ˜fb.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/359,169 US20140321696A1 (en) | 2011-11-22 | 2012-11-21 | Perceptual Rating Of Digital Image Retouching |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201161562934P | 2011-11-22 | 2011-11-22 | |
US201261606775P | 2012-03-05 | 2012-03-05 | |
PCT/US2012/066385 WO2013078404A1 (en) | 2011-11-22 | 2012-11-21 | Perceptual rating of digital image retouching |
US14/359,169 US20140321696A1 (en) | 2011-11-22 | 2012-11-21 | Perceptual Rating Of Digital Image Retouching |
Publications (1)
Publication Number | Publication Date |
---|---|
US20140321696A1 true US20140321696A1 (en) | 2014-10-30 |
Family
ID=48427035
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/359,169 Abandoned US20140321696A1 (en) | 2011-11-22 | 2012-11-21 | Perceptual Rating Of Digital Image Retouching |
US13/683,954 Active 2033-09-19 US9135690B2 (en) | 2011-11-22 | 2012-11-21 | Perceptual rating of digital image retouching |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/683,954 Active 2033-09-19 US9135690B2 (en) | 2011-11-22 | 2012-11-21 | Perceptual rating of digital image retouching |
Country Status (2)
Country | Link |
---|---|
US (2) | US20140321696A1 (en) |
WO (1) | WO2013078404A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10594938B2 (en) * | 2017-02-22 | 2020-03-17 | Canon Kabushiki Kaisha | Image processing apparatus, imaging apparatus, and method for controlling image processing apparatus |
US20220256245A1 (en) * | 2019-05-23 | 2022-08-11 | Lg Electronics Inc. | Display device |
Families Citing this family (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140321696A1 (en) * | 2011-11-22 | 2014-10-30 | The Trustees Of Dartmouth College | Perceptual Rating Of Digital Image Retouching |
GB2507558A (en) * | 2012-11-05 | 2014-05-07 | Toshiba Res Europ Ltd | Image processing with similarity measure of two image patches |
CN103413153B (en) * | 2013-08-28 | 2016-09-14 | 中山大学 | Facial image mill skin evidence collecting method based on SVM study |
JP5708868B1 (en) * | 2014-08-20 | 2015-04-30 | 富士ゼロックス株式会社 | Program, information processing apparatus and method |
CN105243371B (en) * | 2015-10-23 | 2018-12-11 | 厦门美图之家科技有限公司 | A kind of detection method, system and the camera terminal of face U.S. face degree |
US10896429B2 (en) * | 2016-03-09 | 2021-01-19 | Talon Systems Software, Inc. | Method and system for auditing and verifying vehicle identification numbers (VINs) with crowdsourcing |
US11423417B2 (en) | 2016-03-09 | 2022-08-23 | Positioning Universal, Inc. | Method and system for auditing and verifying vehicle identification numbers (VINs) on transport devices with audit fraud detection |
US10796317B2 (en) * | 2016-03-09 | 2020-10-06 | Talon Systems Software, Inc. | Method and system for auditing and verifying vehicle identification numbers (VINs) with audit fraud detection |
US20180373293A1 (en) * | 2017-06-21 | 2018-12-27 | Newtonoid Technologies, L.L.C. | Textile display system and method |
CN111814520A (en) * | 2019-04-12 | 2020-10-23 | 虹软科技股份有限公司 | Skin type detection method, skin type grade classification method, and skin type detection device |
US11069036B1 (en) | 2020-01-03 | 2021-07-20 | GE Precision Healthcare LLC | Method and system for real-time and offline de-identification of facial regions from regular and occluded color video streams obtained during diagnostic medical procedures |
JP2022084116A (en) * | 2020-11-26 | 2022-06-07 | キヤノン株式会社 | Image processing device, control method therefor, imaging device, and program |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100239163A1 (en) * | 2009-03-19 | 2010-09-23 | Electronics And Telecommunications Research Institute | Image searching method and apparatus |
US20120120269A1 (en) * | 2010-11-11 | 2012-05-17 | Tessera Technologies Ireland Limited | Rapid auto-focus using classifier chains, mems and/or multiple object focusing |
US20120281924A1 (en) * | 2011-05-04 | 2012-11-08 | Stephane Coulombe | Method and system for increasing robustness of visual quality metrics using spatial shifting |
US20130129234A1 (en) * | 2011-11-22 | 2013-05-23 | The Trustees Of Dartmouth College | Perceptual Rating Of Digital Image Retouching |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3646931B2 (en) * | 2001-08-29 | 2005-05-11 | セイコーエプソン株式会社 | Image retouching program |
JP4345426B2 (en) | 2003-10-07 | 2009-10-14 | ソニー株式会社 | Image collation method, program, and image collation apparatus |
JP2007257585A (en) * | 2006-03-27 | 2007-10-04 | Fujifilm Corp | Image processing method, device and program |
US8781175B2 (en) * | 2007-05-07 | 2014-07-15 | The Penn State Research Foundation | On-site composition and aesthetics feedback through exemplars for photographers |
US7973977B2 (en) * | 2007-05-18 | 2011-07-05 | Reliance Media Works | System and method for removing semi-transparent artifacts from digital images caused by contaminants in the camera's optical path |
EP2003896A1 (en) * | 2007-06-12 | 2008-12-17 | Panasonic Corporation | Statistical image enhancement |
KR101366255B1 (en) * | 2008-02-20 | 2014-02-20 | 삼성전자주식회사 | Digital image processing apparatus, method for controlling the same, and recording medium storing program to implement the method |
US8269849B2 (en) | 2009-01-23 | 2012-09-18 | Hewlett-Packard Development Company, L.P. | Method and system for testing image pipelines |
US8666193B2 (en) | 2009-04-27 | 2014-03-04 | St-Ericsson (France) Sas | Geometric method of transforming a two-dimensional image |
KR20110066011A (en) | 2009-12-10 | 2011-06-16 | 한국전자통신연구원 | Apparatus and method for similar shader search using image feature extraction |
US8811686B2 (en) * | 2011-08-19 | 2014-08-19 | Adobe Systems Incorporated | Methods and apparatus for automated portrait retouching using facial feature localization |
-
2012
- 2012-11-21 US US14/359,169 patent/US20140321696A1/en not_active Abandoned
- 2012-11-21 WO PCT/US2012/066385 patent/WO2013078404A1/en active Application Filing
- 2012-11-21 US US13/683,954 patent/US9135690B2/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100239163A1 (en) * | 2009-03-19 | 2010-09-23 | Electronics And Telecommunications Research Institute | Image searching method and apparatus |
US20120120269A1 (en) * | 2010-11-11 | 2012-05-17 | Tessera Technologies Ireland Limited | Rapid auto-focus using classifier chains, mems and/or multiple object focusing |
US20120281924A1 (en) * | 2011-05-04 | 2012-11-08 | Stephane Coulombe | Method and system for increasing robustness of visual quality metrics using spatial shifting |
US20130129234A1 (en) * | 2011-11-22 | 2013-05-23 | The Trustees Of Dartmouth College | Perceptual Rating Of Digital Image Retouching |
Non-Patent Citations (1)
Title |
---|
Wang, Zhou, et al. "Image quality assessment: from error visibility to structural similarity." IEEE transactions on image processing 13.4 (2004): 600-612. * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10594938B2 (en) * | 2017-02-22 | 2020-03-17 | Canon Kabushiki Kaisha | Image processing apparatus, imaging apparatus, and method for controlling image processing apparatus |
US20220256245A1 (en) * | 2019-05-23 | 2022-08-11 | Lg Electronics Inc. | Display device |
Also Published As
Publication number | Publication date |
---|---|
WO2013078404A1 (en) | 2013-05-30 |
US20130129234A1 (en) | 2013-05-23 |
US9135690B2 (en) | 2015-09-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9135690B2 (en) | Perceptual rating of digital image retouching | |
US10489683B1 (en) | Methods and systems for automatic generation of massive training data sets from 3D models for training deep learning networks | |
US11443480B2 (en) | Method and system for remote clothing selection | |
US8265351B2 (en) | Method, system and computer program product for automatic and semi-automatic modification of digital images of faces | |
US8660319B2 (en) | Method, system and computer program product for automatic and semi-automatic modification of digital images of faces | |
Kee et al. | A perceptual metric for photo retouching | |
US9760935B2 (en) | Method, system and computer program product for generating recommendations for products and treatments | |
US9142054B2 (en) | System and method for changing hair color in digital images | |
US8620038B2 (en) | Method, system and computer program product for automatic and semi-automatic modification of digital images of faces | |
CN110599528A (en) | Unsupervised three-dimensional medical image registration method and system based on neural network | |
US20050180657A1 (en) | System and method for image-based surface detail transfer | |
WO2013005447A1 (en) | Face impression analysis method, cosmetic counseling method, and face image generation method | |
Qin et al. | Robust match fusion using optimization | |
US11507781B2 (en) | Methods and systems for automatic generation of massive training data sets from 3D models for training deep learning networks | |
WO2015017687A2 (en) | Systems and methods for producing predictive images | |
Yeganeh et al. | Objective assessment of tone mapping algorithms | |
WO2023088277A1 (en) | Virtual dressing method and apparatus, and device, storage medium and program product | |
Dai et al. | Burn images segmentation based on Burn-GAN | |
Cui et al. | TPET: two-stage perceptual enhancement transformer network for low-light image enhancement | |
Cui et al. | Progressive dual-branch network for low-light image enhancement | |
Fang et al. | PG-VTON: A novel image-based virtual try-on method via progressive inference paradigm | |
Ponton et al. | Fitted avatars: automatic skeleton adjustment for self-avatars in virtual reality | |
Florea et al. | High dynamic range imaging by perceptual logarithmic exposure merging | |
Bardis et al. | WebGL enabled smart avatar warping for body weight animated evolution | |
Xia et al. | Robust patchmatch HDR image reconstruction for deghosting |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |