Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T11/00—2D [Two Dimensional] image generation
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/12—Computing arrangements based on biological models using genetic models
  • G06N3/126—Evolutionary algorithms, e.g. genetic algorithms or genetic programming

Definitions

• the present invention is in the field of computer applications and, more particularly, computer applications that improve the quality of digital images.
• This digital image editing tool should be simple to operate, powerful in its ability to alter the digital image in beneficial ways, and efficient.
• the present invention is directed to providing such a digital imaging tool.
• each image has an associated genotype, with the associated genotype having a plurality of genes.
• Each gene defines an attribute of the associated image such as brightness, color saturation, and contrast.
• the genotype may also contain genes that define methods for enhancing the digital image. For example, the gene may define a method for making a global color shift in the digital image or applying special effects to the digital image.
• the genotype may be a predefined template genotype or may be a genotype that is evolved through a selection process described below.
• the selection process includes receiving a fitness rating for each of a plurality of digital images within a current generation of images.
• Each current generation of images generally includes a leader image and a plurality of child images.
• a copy of an original digital image is made the leader image of a first generation of images and the leader image is assigned a neutral leader genotype.
• a plurality of child genotypes are applied to the leader image with the appUcation of each child genotype yielding a child image.
• the child images are sequentially compared to the leader image and a fitness rating is assigned to each child image.
• An improved image may be chosen at any time from any of the images in the current generation of images or the evolution can continue by making the current generation of images a previous generation of images and evolving a next generation of images to become the current generation of images.
• the method of evolution creates a next generation of images by evolving a plurality of next generation child images from a next generation leader image.
• the next generation child images are created by applying a plurality of next generation child genotypes that are evolved from the leader and child genotypes of the previous generation of images.
• the leader genotype from the previous generation of images is recombined with a corresponding child genotype from the previous generation of images.
• the next generation child genotype is then mutated.
• a child image from the previous generation that received the highest fitness rating is made the next generation leader image.
• Each of the next generation child genotypes is applied to the next generation leader image to produce a next generation of child images. Since the next generation leader image and the next generation child images become the current generation of images for the selection process, the method of evolution in accordance with the present invention may continue through unlimited generations.
• a genotype includes a plurality of genes. Each gene includes one or more parameters.
• the genotypes, genes and parameters are evolved through methods for recombination and mutation.
• the boundary gradation curves (and thus the interpolated gradation curves) may be linear or non-linear and may be predefined or generated specifically for the digital image being processed.
• the gradation curve defines the replacement values for current values in the digital image when the gene is applied to the digital image.
• any genotype may be stored independent of a digital image file and later applied to a digital image as a template genotype.
• the stored genotype comprises a relatively small data file that may be easily transmitted between computers across a communication network.
• Other template genotypes may be evolved by third parties or shipped with a product incorp orating the present invention.
• a printer calibration genotype may be provided as a template genotype or evolved according to a method and system of the present invention. The printer calibration genotype adjusts the digital image for the particular printing characteristics of the printer being calibrated. Before the digital image is sent to the printer, the printer calibration genotype is applied to the digital image.
• FIGURE 1 is a block diagram of a general purpose computer system suitable for implementing the present invention
• FIGURE 2A is a block diagram illustrating a common storage method for storing a digital image file
• FIGURE 2B is an illustration of a color space model used by an actual embodiment of the present invention.
• FIGURES 3A, 3B and 4 are pictorial representations of a user interface provided by an actual embodiment of the present invention
• FIGURE 5 is a functional flow diagram of a main processing loop, in accordance with the present invention.
• FIGURE 6 is a functional flow diagram illustrating an evolution of a digital image, in accordance with the present invention.
• FIGURE 7 is a functional flow diagram illustrating a process of creating a set of child images from a leader image using a method of genetic evolution, in accordance with the invention
• FIGURE 8 is a functional flow diagram illustrating a process for recombining genotypes, in accordance with the present invention.
• FIGURE 9 is a functional flow diagram illustrating a process of recombining a leader genotype with a fitness-rated child genotype, in accordance with the present invention.
• FIGURE 10A is a functional flow diagram illustrating the process of mutating a genotype, in accordance with the present invention.
• FIGURE 10B is a mutation rate table containing values for mutation rates used in the process illustrated in FIGURE 10A;
• FIGURE 11 is a functional flow diagrams illustrating the process of applying a genotype to a digital image, in accordance with the present invention.
• FIGURES 12A-12C are a functional flow diagram illustrating the process for creating RGB gradation curves for gene 1 (global color shift) of the present invention
• FIGURES 12D-12E are graphs of boundary gradation curves used by the process illustrated in FIGURES 12A-12C, in accordance with the present invention
• FIGURE 12F is an illustration of interpolating an interpolated gradation curve between a boundary gradation curve and a neutral gradation curve
• FIGURE 13A is a functional flow diagram illustrating a process of providing the RGB gradation curves for gene 2 (color temperature) in accordance with the present invention
• FIGURES 13B-13D are graphs of boundary gradation curves used in the method illustrated in FIGURE 13 A;
• FIGURE 14 is a functional flow diagram illustrating a method for specifying the RGB gradation curves for gene 3 (effects), in accordance with the present invention
• FIGURE 15A is a functional flow diagram illustrating the method for deteiTJiiiiing a blackpoint and wbitepoint gradation curve for gene 8 (blackpoint/wbitepoint), in accordance with the present invention
• FIGURE 15B is a graph of an example blackpoint and wbitepoint gradation curve
• FIGURE 16A is a functional flow diagram illustrating process of providing the RGB gradation curves for gene 4 (brightness), in accordance with the present invention
• FIGURES 16B-16C are graphs of boundary gradation curves referenced by the processes of FIGURES 16A and 18;
• FIGURE 17 is a functional flow diagram illustrating the process of providing a contrast gradation curve for gene 5 (contrast), in accordance with the present invention
• FIGURE 18 is a functional flow diagram illustrating the process of adjusting a pixel's color saturation value using gene 6 (color saturation), in accordance with the present invention
• FIGURES 19A-19B are a functional flow diagram illustrating the method for adjusting the black level in primary colors using gene 7a (black level), in accordance with the present invention
• FIGURES 19C-19D illustrate curves referenced by the method of FIGURES 19A-19B;
• FIGURES 20A-20C are a functional flow diagram illustrating the method for selective color correction of a pixel using gene 7b (selective color correction), in accordance with the present invention.
• FIGURE 21A is a functional flow diagram illustrating a process for preparing a contrast gradation curve for gene 9 (contrast gradation curve), in accordance with the present invention
• FIGURES 2 IB and 21C are pictorial representations of a histogram and a gradation curve, respectively, calculated by the process illustrated in FIGURE 21 A, in accordance with the present invention
• FIGURE 22 is a functional flow diagram of a process for removing optical distortion in digital images (dewarp), in accordance with the present invention.
• FIGURES 23A-23C are a functional flow diagram illustrating a process for calibrating a printer using a genetic algorithm, in accordance with the present invention.
• FIGURE 1 illustrates a computer system 100 of conventional design that is suitable for practicing the present invention.
• the computer system includes a central processing unit (CPU) 110 that executes program instructions and manipulates digital data.
• the central processing unit 110 is coupled to a volatile memory 112 and a nonvolatile memory 114 that store the program instructions and data.
• the central processing unit 110 is coupled to a video monitor 116, which provides a human recognizable medium for the display of digital images, text and other data to a user.
• a keyboard 119 and a pointer device 117 (a "mouse") are coupled to the CPU 110 for providing input to the computer 100.
• a printer 118 may be coupled to the CPU 100 to provide printouts of images, text and other data.
• the CPU 110 may also receive input from devices such as a digital camera 120 or optical scanner 122. Both the digital camera 120 and the optical scanner 122 record photographic images as computer-readable digital image files.
• a plurality of computers 100 may be connected to a communication network 124 such as a local area network (LAN), a wide area network (WAN), or global network such as the Internet.
• a communication network 124 such as a local area network (LAN), a wide area network (WAN), or global network such as the Internet.
• LAN local area network
• WAN wide area network
• Internet global network
• a remote computer 126 is generally similar in configuration to that described for computer 100.
• a digital image file 210 (FIGURE 2A), such as that created by the digital camera 120 or the optical scanner 122 (FIGURE 1), comprises data representing a plurality of pixels 212. When rendered on the video monitor 116 or the printer 118, the plurality of pixels visually form an image.
• the digital image file 210 is assumed to be 480 x 480 pixels in size. Of course, other digital image file sizes, formats and resolutions can be used when practicing the present invention.
• An exemplary pixel 212 is shown in a partial view in FIGURE 2A.
• the pixel 212 is defined by a red (R) data value 214, a green (G) data value 216, and a blue (B) data value 218 (collectively the "RGB values").
• a red (R) data value 214, a green (G) data value 216, and a blue (B) data value 218 define a color for the pixel 212 from a set of colors available in a color space.
• a color space is a mathematical representation of a set of colors. In other words, by specifying an R value 214, a G value 216, and a B value 218, a color in the color space can be replicated for each pixel 212 on the video monitor 116, a printer 118, or other device.
• a RGB color space is illustrated in FIGURE 2B as a hexagon 220, with each of the additive and subtractive primary colors indicated at a vertex.
• the primary colors red 222, green 224, and yellow 226 are known as the additive primary colors, and the primary colors magenta 228, cyan 230, and blue 232 are known as the subtractive primary colors.
• any color within the color space can be specified.
• the number of colors that can be specified in the color space depends on the resolution of each RGB value 214, 216, and 218.
• the resolution is defined by the number of bits of data that are used to capture and store each of the RGB values—the more bits, the more colors that can be described In the following discussion, it will be assumed that the RGB values have a resolution of eight bits, or 256 possible values. Each pixel, therefore, can specify more man 1.6 million different colors in the RGB color space defined by hexagon 220. However, the resolution and size (number of pixels) in the digital image is not important to processes of the present invention.
• the digital image file 210 may have a resolution that is higher or lower depending on the quality requirements for the digital image, and practical matters such as storage requirements for the digital image file 210.
• Other color space models may be used without departing from the scope of the present invention. Examples of other color space encoding include YIQ, YUV, YCbCr, HSI, and HSV. All of these color spaces can be translated to the RGB color space 220 described herein using well-known conversion methods.
• the digital image file 210 may be stored in many different formats that are well known in the art.
• the RGB values 214, 216, 218 can be individually stored for each pixel in the digital image file 210 in a bitmap, or these RGB values may be translated, compressed and stored in a variety of other formats such as TIFF, JPEG, or MPEG, among other known digital image file 210 formats. All file formats may be processed by the present invention.
• FIGURES 3 A-3B and 4 A user interface 300 provided by an actual embodiment of the present invention is illustrated in FIGURES 3 A-3B and 4.
• the user interface 300 includes a menu bar 310 that contains menus named File 312, Edit 314, View 316, Window 318, Special 320, and Help 322.
• the File menu 312 includes a drop-down menu that includes items for saving and retrieving the digital image file 210 to and from the non-volatile memory 114, including options to Open, Open Special, Close, Save, Save As, and Save Genotype As.
• the Open Special option will be discussed further with reference to FIGURE 3B.
• a genotype may be saved in a separate data file using the Save Genotype As option in this menu (genotypes are discussed below beginning with the discussion of image evolution in FIGURE 5).
• the digital image file 210 may also be loaded into the application controlling the user interface 300 using file menu options to Acquire the image from a device such as scanner 122 (FIGURE 1) or from another device using a Select Source menu option.
• the image may be printed on the printer 118 using File menu options for Print, Print Preview (on the display), Printer Setup, and Printer Calibration. Printer calibration is described below with reference to FIGURE 26A-C.
• the user may also select an option for Preferences or may select an Exit option from the drop- down menu to end the application.
• the Edit menu 314 includes options to Undo the Last Action, Perform a Dewarp on the Image, to Start Evolution, and to Rotate Left, Rotate Right, Resize Image, Crop Image, and to Remove Red Eyes.
• the Perform a Dewarp option is discussed below with reference to FIGURE 22 and the Start Evolution option is discussed below beginning with FIGURE 6.
• the View menu 316 allows for the selection of user interface elements that the user wishes to appear in user interface 300. These user interface elements include a Toolbar menu item (toolbar 324), a Status Bar menu item (status bar 326), a Navigator window menu item (navigator 328), and Genotypes window menu item (genotypes 330).
• the View menu 316 also provides the options to Compare to Original and to present the Optimal View of the image.
• the Window and Help menus 318 and 322 offer menu items well known to those skilled in the art and will not be discussed further. Referring to FIGURE 3 A, the user interface 300 has an image display area 332 that displays a digital image file 210 that has been opened for enhancement.
• the digital image file 210 will be referred to as a parent image 334.
• a miniature (thumbnail) version of the parent image 334 is displayed in a navigator 328 window.
• a slider control 336 By using a slider control 336, a minus command button 338 or a plus command button 340, a user can select a percentage of the total image and only a portion of the parent image 334 will be displayed in the display area 332.
• the genotypes list box 330 contains a list of template genotypes that may be applied to the parent image 334 to change the parent image 334 according to the predefined genes and gene parameters of the template genotype selected. For instance, if the inverse (INV) template genotype 335 is selected from the list box and an Apply command button 340 is activated, the TNV template genotype 335 would be applied (FIGURE 11) to the parent image 334 and would result in an evolved image replacing the parent image in the display area 332. In the example shown, the evolved image would appear as a black background with a white outline of a sailboat. If a Reject control button 342 is activated, the parent image 334 will revert to the state it was in just before the TNV template genotype 335 was applied.
• the toolbar 324 includes command buttons for many of the functions described as menu items in the menus 310 described above.
• a Dewarp command button 344 activates the image Dewarp process illustrated in FIGURE 22
• a color temperature button 346 adjusts the color temperature as illustrated in FIGURE 13 A
• an image cropping command button 348 activates a function for cropping the parent image 334
• a remove red eyes command button 350 activates a remove red eyes function.
• the user interface presented when the Open Special method of the present invention available from the drop-down file menu is invoked as illustrated in FIGURE 3B.
• the Open Special user interface 360 includes a directory tree window 362 for selecting the directory 364 containing image data files which may be imported into the present invention.
• Each image 366A, 366B, 366C, 366D, 366E . . . stored in the directory 364 is displayed as a thumbnail image that, when selected (as indicated by a dark border surrounding the image), is opened for editing by the present invention by activating an Open button 368.
• a drop-down list box 370 lists available template genotypes. When an available template genotype is selected, selected thumbnail image (or images) are changed by the selected template genotype. When loaded, the loaded image is changed by the selected template genotype.
• the present invention quickly and efficiently improves the quality of digital images through evolution.
• the evolution process is initiated by activating the Start Evolution command button 352 (FIGURE 3A).
• the Start Evolution command button 352 when the Start Evolution command button 352 is actuated, the image display area 332 changes to display a copy of the parent image 334 as a leader image 334A in the left-hand portion 417 of the image display area 332 and one of a set of four child images 334B, 334C, 334D, and 334E in the right-hand portion 415 of the image display area 332.
• Thumbnail images of the leader image 334A and of the four child images 334B-E are displayed in a legend area 410 together with an indication 412 of the generation to which the images 334A-E belong.
• Each child image 334B-E is sequentially displayed in the right-hand portion 415 of the image display area 332 for comparison with the leader image 334A that is displayed in the left-hand portion 417 of the image display area 332.
• the child image 334B is compared to the leader image 334A and fitness rated by selecting a portion of a graduated control 414.
• the graduated control 414 has a "no better or worse" portion 416 and a graduated portion 418 associated with values from 1% to 100% better.
• a fitness value equal to the percentage selected from the graduated control 414 is associated with the child image 334B.
• the next child image 334C is displayed in the right-hand portion 415 of the image display area 332 for comparison with the leader image 334A.
• the process of assigning a fitness rating for images 334D and 334E continues in the same manner just described.
• the status bar 326 displays an indication of the current generation and the number of the child image 334B-E that is currently being compared.
• the evolution can be stopped by activating a Stop Evolution command button 420.
• a user dialog is displayed asking whether the left image (leader image 334A) or the right image (child image 334B-E) should be kept.
• the kept image can then be saved as a new digital image file 210 or otherwise manipulated by the user's computer 100.
• next generation leader image 422A is displayed in the right-hand portion 417 of the video display area 332.
• the next generation leader image is created from a (new) leader genotype that is produced by recombining the (previous) leader genotype with the genotype of the highest fitness-rated child from the last generation.
• next generation child images 422B- E are evolved and displayed in the legend area 410 with an indication 412 that the second generation is being viewed.
• the child images 422B-E are then compared to the leader image using the graduated control 414 to assign fitness ratings as is discussed above.
• a parent image is imported 510 from the non-volatile memory 114 (or other source such as the digital camera 120 or optical scanner 122) into a workspace available to the present invention in the volatile memory 112.
• the encoded information describing the parent image e.g., in formats such as bitmap, JPEG, TIFF, etc.
• RGB red, green, and blue
• the parent image can be modified according to the present invention by image evolution 512, applying a template genotype 514, or conducting an image Dewarp 516. Any digital image 210 can be accurately printed following a printer calibration 518 according to the method of the present invention. These four processes 512, 514, 516, 518 will be discussed below. Any or all of the above methods 512, 514, 516, 518 may be conducted in any order on a parent image until the program is exited 520.
• evolution is performed 512 beginning with the process illustrated in FIGURE 6.
• the evolution method 512 sets 610 a leader genotype to a neutral genotype.
• a genotype is a set of genes (described below) that are used to alter a digital image in a number of ways.
• a neutral genotype contains genes that do not affect the digital image to which they are applied.
• the evolution of the first generation of child images may be produced using predefined genotypes or evolved genotypes. If predefined genotypes are selected 612, then the method 512 selects a plurality of predefined template genotypes that are used as the initial child genotypes. Each child genotype is apphed 616 to the leader image to produce a corresponding child image. The child images are displayed 618 and the method 512 receives the fitness ratings of each of the child images from the user (as was described above with reference to FIGURE 4).
• the evolution continues 620 by recombining 622 the genotype of the child image that received the highest fitness rating with the leader genotype as a next generation leader genotype. As is illustrated begiiuding in FIGURE 7, a next generation of child genotypes is evolved 624 using this next generation leader genotype. Alternatively, if the evolution is ended 620, the method 512 is done 626.
• the evolve next generation method 710 begins by recombining 712 the leader genotype with a child genotype from the previous generation into a next generation child genotype. Recombination is discussed below with reference to FIGURE 8.
• the next generation child genotype is then mutated 714 in the manner illustrated in FIGURE 10 and described below. Having been recombined and mutated, the next generation child genotype is apphed 716 (FIGURE 11) to the currently loaded image (the leader image) to produce a next generation child image.
• the next generation child genotype is then associated with the next generation child image.
• an actual embodiment of the present invention produces four child images in each generation for comparison with that generation's leader image. Of course, greater or fewer child images can be produced in each generation.
• the evolve next generation method 710 repeats (starting with recombining 712 the child genotype with the leader genotype into a next generation child genotype).
• a decision 720 directs the evolve next generation method 710 to return 722 all the next generation child images, together with their associated next generation child genotypes, as child images for presentation as a (new) current generation in the selection process described above (FIGURE 4). The method 710 is then done 724.
• the method 810 for recombining the leader genotype with each child genotype to produce a plurality of next generation child genotypes is illustrated in FIGURE 8.
• the method 810 retrieves 812 a leader genotype. (It is to be understood that each of the methods described herein may retrieve, create or be passed data.)
• a child genotype is then retrieved 814.
• the individual genes of the leader genotype are recombined 816 with the genes of the child genotype into a next generation genotype, in the manner illustrated in FIGURE 9 and described below.
• a decision 818 returns the method 810 to the retrieval 814 of another child genotype and the process is repeated.
• the next generation genotypes are returned 820 as the (new) child genotypes to block 712 in FIGURE 7.
• the method 810 is then done 822.
• the method 910 of the present invention for recombining the leader genotype with a child genotype begins by retrieving 912 a leader gene from a set of genes contained in leader genotype. A corresponding child gene is also retrieved 912 from the child genotype. Each gene may contain one or more parameters which are retrieved 914 one at a time from the leader gene and then the corresponding child gene. A next generation gene parameter is computed 916 by taking a weighted average of the leader gene parameter and the child gene parameter. In this computation 916, the child gene parameter is weighted by a factor that equals the fitness rating that was assigned to the child genotype (as discussed above with reference to FIGURE 4). For instance, if the resolution was eight bits (256 values), the weighting formula would be:
• This weighted average formula produces a next generation gene parameter that is stored 918 in a next generation gene.
• a decision 920 is made to determine if there is another gene parameter to process. If so, the method 910 returns to retrieve 914 the (next) leader gene parameter from the leader gene and a 0 corresponding child gene parameter from the child gene. If all gene parameters have been processed, the decision 920 directs the method 910 to store 922 the next generation gene in the next generation genotype. If there is another leader gene to process, a decision 924 returns the process to the beginning of method 910 to retrieve 912 the next leader gene and a corresponding child gene. If the decision 924 5 determines that there is not another leader gene to process, then the method 910 is done 926.
• the method 1010 for mutating a genotype is shown in FIGURE 10A.
• the method 1010 retrieves a genotype to be mutated from a block 714 in FIGURE 7.
• a gene from the genotype is retrieved 1012 and a mutation rate is obtained for the gene 0 from a mutation rate table (FIGURE 10B).
• the mutation rate table provides the mutation rate to use for the gene being processed (column) and the number of the generation currently being processed (row). For instance, if gene 4 is being mutated from generation 5, cell 1016 would be selected to obtain a value of "18" hexadecimal. If the evolution has progressed through more than 16 generations, the mutation rate is 5 selected from the generation 16 column according to the genes row. For instance, all mutations after the 16th generation for gene 5 will have a mutation rate found in cell 1018 that indicates a hexadecimal value of "LA".
• the mutation rate is multiplied 1022 by a Gauss-random value to produce a mutation value from a Gauss-random function.
• the Gauss-random function returns a value that makes small changes to the mutation rate very probable while big changes will seldom occur. Gauss-random functions are well known to those skilled in the art and will not be further discussed.
• the mutation value is then added to the gene to produce a mutated gene. If the gene has multiple parameters, all parameters are mutated.
• the mutated gene is then stored 1026 in a mutated genotype. If another gene in the genotype needs to be processed, the method 1010 is directed 1028 to retrieve 1012 the next gene in the genotype. If another gene is not present in the genotype to be mutated, the method 1010 is directed 1030 to return 1032 the mutated genotype as the next generation genotype. The method 1010 is then done 1034.
• the method 1110 for applying a genotype to a digital image is shown in FIGURE 11.
• the leader (currently loaded) image is retrieved 1112.
• the child genotype to be apphed to the leader image is also retrieved 1114.
• a pixel is retrieved 1116 from the currently loaded image and a gene is retrieved 1118 from the genotype.
• the gene is apphed 1120 to the pixel according to a method associated with each gene, as is discussed below in detail with reference to FIGURES 12A-21.
• a decision 1122 determines if there is another gene in the genotype that still needs to be processed. If so, the method 1110 directs 1124 the retrieval 1118 of the next gene from the genotype. If there is not another gene in the genotype to be processed, the decision 1122 directs 1126 the method 1110 to store 1128 the newly altered pixel in a child image.
• a decision 1130 determines if there is another pixel in the currently loaded image that still needs to be processed, and directs 1132 the method 1110 to repeat starting with retrieving 1116 the next pixel from the currently loaded image. If there is not another pixel in the currently loaded image, the decision 1130 returns 1136 the child image and the associated genotype to the calling method. The method 1110 is then done 1138.
• FIGURES 12A-21 detail a set of nine genes in a genotype provided by an actual embodiment of the invention. Other genes may be added to the genotype and still be within the scope and spirit of the present invention. For instance, a gene could be developed to create artistic effects for the image using the methods and systems described below. While the order that the genes are apphed does not have to be strictly defined, the genes below are discussed in the order that they are apphed in an actual embodiment of the invention.
• genes described below include methods for defining a new gradation curve for each of the R, G, and B values.
• the apphcation of the gradation curves specified by these genes occurs in FIGURE 11.
• a gradation curve (e.g., FIGURE 12D) is generated. This generated gradation curve is then referenced to assign a new value (the y-coordinate) to the R 214, G 216, or B 218 value for each pixel in the digital image.
• gradation curves shown in the FIGURES are illustrated as continuous curves both to more clearly point out the concept of the present invention and to point out that the methods and systems of the present invention apply equally to digital images of all resolutions, e.g., a gradation curve may be apphed to a digital image with 8-bit resolution equally as well as to a digital image with 32-bit resolution simply by calculating a greater range of data points for (or from) the gradation curve.
• interpolating a gradation curve (FIGURE 12F) is actually computing a new value (y-coordinate) for each possible old value (x-coordinate).
• predefined boundary gradation curves described for the use of the genes discussed herein have been determined empirically and may be changed as further experience dictates. The same is true for the predefined data points used by some genes to dynamically calculate boundary gradation curves.
• the interpolated gradation curve 1206 in this example represents a weight (or parameter) value of 89. If the gene is defined by an 8-bit signed integer, giving 127 possible positive values (and 127 possible positive interpolated gradation curves), the positive boundary gradation curve 1204 represents the maximum parameter value (127).
• the interpolated gradation curve 1206 therefore comprises 256 new values (ynew) that may be stored in any convenient data structure for use in the methods and systems described herein, such as the lookup table described above or each ynew point may be calculated as needed.
• the global color shift of gene 1 is accomplished by calculating a gradation curve for each of the RGB values.
• Gene 1 has three parameters, with the first parameter specifying which two of the three RGB gradation curves will be interpolated. If the gene 1 first parameter is 1, new red and green gradation curves are interpolated and the blue gradation curve is assigned a neutral gradation curve 1202. When the first parameter of gene 1 is equal to 2, new green and blue gradation curves are created and the red gradation curve is assigned the neutral gradation curve 1202. If the first parameter of gene 1 is equal to 3, new red and blue gradation curves are calculated and the green gradation curve is assigned the neutral gradation curve 1202.
• the second and third parameters of gene 1 are specified by a signed 8-bit integer with a range of -128 to +127.
• a gradation curve is calculated for every possible value of parameter 2 (and 3) by calculating a weighted interpolation between a predefined boundary curve and the neutral curve.
• a decision 1212 directs 1214 the process to a decision 1216 that checks if the second parameter is positive. If the second parameter is positive, the decision 1216 directs 1218 a method 1220 to use the second parameter of the leader genotype gene 1 as a weight value to interpolate a new red global color shift gradation curve between a positive global color shift boundary gradation curve (FIGURE 12D) and the neutral gradation curve 1202 (FIGURE 12F).
• the method 1210 continues 1222 by using the leader genotype gene 1 second parameter as the weight value to interpolate a new red global color shift gradation curve 1224 between the negative global color shift gradation curve (FIGURE 12E) and the neutral gradation curve 1202 (FIGURE 12F).
• a decision 1226 is made whether the third parameter of the leader genotype gene 1 is positive. If so, the third parameter of the leader genotype gene 1 is used as the weight value to interpolate a new green color shift gradation curve between the positive global color shift gradation curve (FIGURE 12D) and the neutral gradation curve 1202 (FIGURE 12F).
• the third parameter of the leader genotype gene 1 is used as the weight value to interpolate 1232 a new green global color shift gradation curve between the negative global color shift gradation curve (FIGURE 12E) and the neutral gradation curve (FIGURE 12F).
• a new blue gradation curve is assigned 1233 the neutral gradation curve (FIGURE 12F).
• the method 1210 continues as illustrated in FIGURE 12B. If gene 1 of a leader genotype has a first parameter equal to two, a decision 1234 directs 1236 the process to a decision 1238 that checks if the second parameter is positive.
• the decision 1238 directs 1240 the method 1210 to use the second parameter of the leader genotype gene 1 as a weight value to interpolate 1242 a new green global color shift gradation curve between the positive global color shift boundary gradation curve (FIGURE 12D) and the neutral gradation curve 1202 (FIGURE 12F).
• the method 1210 continues 1244 by using the leader genotype gene 1 second parameter as the weight value to interpolate a new green global color shift gradation curve 1246 between the negative global color shift gradation curve (FIGURE 12E) and the neutral gradation curve (FIGURE 12F).
• a decision 1248 is made whether the third parameter of the leader genotype gene 1 is positive. If so, the third parameter of the leader genotype gene 1 is used as the weight value to interpolate a new blue color shift gradation curve 1252 between the positive global color shift gradation curve (FIGURE 12D) and the neutral gradation curve (FIGURE 12F). If in the decision 1248 it is determined that the third parameter is not positive, the third parameter of the leader genotype gene 1 is used as the weight value to interpolate a new blue global color shift gradation curve 1256 between the negative global color shift gradation curve (FIGURE 12E) and the neutral gradation curve 1202 (FIGURE 12F). A new red gradation curve is assigned 1257 the neutral gradation curve 1202 (FIGURE 12F). After the RGB gradation curves have been interpolated or assigned 1252,
• a decision 1260 directs 1262 the process to a decision 1264 that checks if the second parameter is positive. If the second parameter is positive, the decision 1264 directs 1265 the method 1210 to use the second parameter of the leader genotype gene 1 as a weight value to interpolate 1266 a new red global color shift gradation curve between the positive global color shift boundary gradation curve (FIGURE 12D) and the neutral gradation curve (FIGURE 12F).
• the method 1210 continues by using the leader genotype gene 1 second parameter as the weight value to interpolate a new red global color shift gradation curve 1268 between the negative global color shift gradation curve (FIGURE 12E) and the neutral gradation curve (FIGURE 12F).
• a decision 1270 is made whether the third parameter of the leader genotype gene 1 is positive. If so, the third parameter of the leader genotype gene 1 is used as the weight value to interpolate a new blue color shift gradation curve 1274 between the positive global color shift gradation curve (FIGURE 12D) and the neutral gradation curve (FIGURE 12F). If in the decision 1270 it is determined that the third parameter is not positive, the third parameter of the leader genotype gene 1 is used as the weight value to interpolate a new blue global color shift gradation curve 1278 between the negative global color shift gradation curve (FIGURE 12E) and the neutral gradation curve (FIGURE 12F).
• a new green gradation curve is assigned 1279 the neutral gradation curve (FIGURE 12F).
• Gene 2 adjusts the color temperature of the child image 334B-E. Color temperature may be thought of as simulating a "warmer” or "colder” ambient light in the digital image.
• Gene 2 has a single parameter defined by a signed 8-bit integer (values from -128 to +127). Altered by gene 2, a negative parameter value result in the image looking warmer (lower color temperature caused by a shift toward orange- red in the image) and a positive parameter will make the image look colder (higher color temperature caused by a shift toward blue in the image).
• FIGURE 13 A The method for calculating the RGB gradation curves 1310 to adjust the color temperature of an image is shown in FIGURE 13 A.
• a single new value (y-coordinates) is calculated for each of the RGB gradation curves, as needed, to adjust a single pixel instead of calculating and storing all of the new values (y-coordinates) of an interpolated gradation curve.
• the following will continue to refer to these new values as being part of a gradation curve to better illustrate the concept behind the method 1310 of gene 2.
• the parameter value of gene 2 is used as a weight value to interpolate 1312 a red color temperature gradation curve ("RTemp") 1312 between a low temperature red boundary gradation curve (FIGURE 13B) and the neutral gradation curve (FIGURE 12F).
• RTemp red color temperature gradation curve
• GTemp green color temperature gradation curve
• a blue color temperature gradation curve (“BTemp") is interpolated 1316 by using the gene 2 parameter as the weight value in a weighted interpolation between a low temperature blue boundary gradation curve (FIGURE 13D) and the neutral gradation curve (FIGURE 12F).
• the new RGB values are obtained 1322 from a set of high temperature RGB gradation curves that mirror the low temperature RGB gradation curves about the neutral gradation curve.
• the high temperature RGB gradation curves may be expressed mathematically as:
• the high-temperature RGB boundary gradation curves may be defined separately from the low-temperature boundary gradation curves and the high- temperature RGB gradation curves interpolated between those separately defined high-temperature RGB gradation curves and the neutral gradation curve (FIGURE 12F).
• the new RGB values obtained in blocks 1320 or 1322 are then returned 1324 as a new pixel for the child image.
• Gene 3 provides RGB gradation curves that define special effects.
• a method 1410 of gene 2 is illustrated in FIGURE 14.
• Gene 3 has three parameters that specify a gradation curve for each red, green, and blue value.
• the first parameter specifies 1412 a predefined gradation curve defining an effect to the red component of each pixel.
• the second parameter of gene 3 specifies 1414 a predefined effect gradation curve that defines an effect to the green component of the pixel.
• a third parameter specifies 1416 a predefined effect gradation curve that defines an effect to the blue value of each pixel.
• the first, second and third parameters may each specify the same gradation curve or a different gradation curve for each of the red, green and blue values.
• the method 1410 is then done 1418.
• Gene 8 determines a blackpoint and wbitepoint gradation curve that is apphed to each of the RGB values.
• the method 1510 of gene 8 is shown in FIGURE 15 A, where the brightest color tone in the image is assigned 1512 as the whitepoint and then the darkest color tone in the image is assigned 1514 as the blackpoint. Using the coordinates (0, whitepoint and 255, blackpoint) a linear gradation curve is computed 1516 between these coordinates. The method 1510 of gene 8 is then done 1518.
• An example of the gradation curve produced by gene 8 is shown in FIGURE 15B with the linear gradation curve 1520 extending between the blackpoint 1522 and the whitepoint 1524.
• the RGB gradation curves calculated by gene 4 affect the brightness of the image.
• Gene 4 has one parameter that is a signed 8-bit integer, ranging from -128 (maximum brightness) to +127 (minimum brightness). In an actual embodiment of the invention, the same calculated brightness gradation curve is used for all the RGB values.
• the method 1610 for calculating the brightness gradation curve of gene 4 is shown in FIGURE 16A and begins with a decision 1612 that determines whether the value of the gene 4 parameter is positive.
• a brightness gradation curve is interpolated 1614 by using the value of a gene 4 parameter as the weight value in a weighted interpolation between a positive brightness/color saturation boundary curve (FIGURE 16B) and the neutral gradation curve (FIGURE 12F). If the decision 1612 determines that the value of the gene 4 parameter is not positive, the weighted interpolation is made using the gene 4 parameter as the weight value between a negative brightness/color saturation boundary gradation curve (FIGURE 16C) and the neutral gradation curve (FIGURE 12F). After calculating the brightness gradation curve 1614 or 1616, the method 1610 is done 1618.
• the contrast gradation curve produced by gene 5 is apphed equally to each of the RGB values to alter contrast as the red, green and blue gradation curve for contrast.
• Gene 5 has one parameter that is a signed 8-bit integer ranging from -128 (lowest contrast) to +127 (maximum contrast).
• the method 1710 for finding the contrast gradation curve of gene 5 begins by retrieving 1712 the contrast parameter of gene 5.
• the contrast parameter is used as the weight value to interpolate 1714 a new contrast gradation curve between a boundary gradation curve generated by gene 9 (discussed below with reference to FIGURE 21) and a neutral gradation curve (FIGURE 12F).
• a new contrast value is determined directly from the new contrast gradation curve 1716 for each of the RGB values of the pixel.
• the method 1710 is then done 1718.
• the method 1810 for adjusting the color saturation of the image using gene 6 is illustrated in FIGURE 18.
• Gene 6 has a single parameter that comprises a signed 8- bit integer in an actual embodiment of the invention.
• the RGB values of the pixel being passed to method 1810 are each translated 1814 from their additive primary color values to the equivalent subtractive primary colors using the formulas:
• the black level (K) is assigned 1816 the minimum value of the C, M or Y:
• the subtractive primary colors are adjusted 1820 (FIGURE 11) using the
• the subtractive primary color values are translated back 1824 to their equivalent additive primary colors using the formulas:
• the method 1810 then returns 1826 the new pixel for storage in the adjusted image and is done 1830.
• the method 1910 of gene 7A for adjusting the black level in the primary colors of the image is illustrated in FIGURES 19A and 19B beginning with the method 1910 retrieving 1912 the RGB values of a pixel.
• the RGB values are translated 1914 from the additive primary color values to the equivalent subtractive primary colors using the formulas:
• the black level is temporarily removed 1918 from the C, M & K values using the formulas:
• Each intermediate separation value is adjusted 1924 by applying a corresponding gene parameter from gene 7A.
• the parameters of gene 7A are changed by a regulator gene (gr) that is evolved during the evolution process.
• the regulator gene (gr) is set by the evolution process, i.e., by mutation and recombination.
• the six gene parameters of gene 7A have the values that are calculated using a predefined weighting factor to adjust the value of the regulator gene, i.e.:
• the weight values used in gl-g6 have been determined empirically and may be changed as experience dictates. In choosing the weight values, however, it should be considered that the reduction in the black value of color should not be equal for all colors or unpleasant artifacts may result. It has been found that green, blue and cyan are the most robust against these artifacts and may be weighted more heavily, while red and yellow are the least robust against manipulation and should have smaller weight values.
• the intermediate color separation values are adjusted 1924 using the following formulas:
• the method 1910 next determines 1926 a black level correction factor (kf) from a predefined curve shown in FIGURE 19D (the "kfCurve"):
• the method 1910 is then done 1938.
• the method 2010 for the selective color correction of the image by gene 7B is illustrated in FIGURES 20A-C.
• the pixel passed to the method 2010 is separated 2012 into the pixel's RGB values, which are translated 2014 to the equivalent subtractive primary colors, using the formulas:
• the method 2010 then retrieves the eighteen parameters of gene 7B which are stored as unsigned 8-bit integers for each color except black: an amplification factor for the color being processed, an amplification factor for the clockwise adjacent color on the color hexagon 220 (FIGURE 2B) to the color being processed, and an amplification factor for the counterclockwise adjacent color on the color hexagon 220 (FIGURE 2B) to the color being processed.
• gradation curves (3 for each color except black) are computed 2024 using the points: (0,0), (xpoint, ypoint), (255,255), where ypoint is equal to xpoint if xpoint is less than 230 or ypoint is equal to 230 if xpoint is greater than 230.
• 230 as the breakpoint has been dete ⁇ riined empirically and may change as further experience dictates.
• the resulting 18 gradation curves are: BBCorrf] BlueCorrection for Blue
• the primary colors are selectively corrected 2028, as foUows:
• the method 2110 of gene 9 provides the contrast gradation boundary curve that is used by the method 1710 of gene 5 (FIGURE 17).
• the contrast gradation boundary curve is calculated by first converting 2112 each pixel into gray according to a gray scale.
• a histogram is then created 2114 that records the gray tone value of every pixel in the image with the x-axis of the histogram representing the gray scale value and the y-axis of the histogram representing the frequency that that gray tone appears in the image.
• An example of such a histogram 2130 is lustrated in FIGURE 21B. The histogram is integrated from its minimum value until 2% of the data members in the histogram has been reached.
• the value of the x-axis at this point is recorded 2116, as a first parameter (A) of gene 9. Similarly, the histogram is integrated from its maximum value toward its minimum value until 2% of the total data members has been reached. The value of the x-axis at this point is recorded 2118 as a second parameter (B) of gene 9.
• the integration percentage (2%) was dete ⁇ nined empiricaUy and may be changed as experience dictates.
• a S-shaped contrast gradation curve 2132 is calculated 2120 using (0,0) and (255,255) as the end points 2134, 2140 and the points (A, 12) and (B, 242) as the inflection points 2136, 2138.
• An example of this S-shaped contrast gradation curve with the end points and inflection points 2136, 2138 just described is iUustrated in FIGURE 21C. Since, spline interpolation algorithms are weU known in the art, they wiU not be discussed further here. Retaining to FIGURE 5, it is also possible to apply 514 a template genotype to the image without going through the evolution process.
• the template genotype may be predefined and shipped as part of the product incorporating the invention or may be a genotype that has been saved from a previous evolution. For example, a user may prefer a genotype that the user has evolved for photographs of mountains that the user wishes to apply to aU pictures of mountains that the user works on.
• the genotype may be selected 524 and the genotype apphed 514 to the image as was discussed above with reference to FIGURE 11.
• any saved or template genotype forms a relatively smaU data file. This data file may be exchanged between appUcations employing the present invention, for instance, by sending the genotype data file from the user's computer 100 over the cornmunication network 124 to another user's remote computer 126 (FIGURE 1).
• the present invention also provides the option 526 to Dewarp the image.
• Performing 516 a Dewarp is iUustrated in FIGURE 22.
• the not perfect shape of the lens may introduce a perceptible aberration in the image.
• the process of "Dewarping" an image minimizes the effect of this spherical aberration by redistributing the pixels in the image.
• the method 2210 of Dewarping the image begins by retrieving 2212 a predefined set of co ⁇ ection coefficients A, B, C.
• a focal point is then determined 2214 in the parent image. For each pixel in the parent image, the pixel is retrieved 2216 and a vectored radius between the location of the pixel and the focal point is determined 2218.
• the pixel is then stored 2222 in a chUd image at the new vectored radius (rnew). To make the image look more smooth, a bi-cubic interpolation is used. If there is another pixel to process in the parent image, a decision 2224 retums control to the retrieval 2216 of the next pixel and the method continues as described above. If there is not another pixel to process in the parent image, the decision directs the method 2210 to end 2226.
• the last option shown in FIGURE 5 is to select 528 printer cahbration.
• a printer cahbration is performed 518 according to the method shown in FIGURES 23A-C.
• Printer cahbrations are necessary because the image produced by any given printer 118 may appear different than the image as it is displayed on the video monitor 116.
• a genotype may be developed that is associated with the printer and that is used to harmonize the image printed by the printer 118 with the image displayed on the video monitor 116.
• the printer cahbration method 2310 begins by retrieving 2312 a gray scale image.
• a gray scale template genotype is apphed 2314 to the gray scale image to create a test image.
• the gray scale template genotype has genes that are neutral for aU attributes other than brightness (altered by gene 4) and contrast (altered by gene 5).
• a decision 2316 determines if another gray scale test image should be produced. If yes, a different gray scale template genotype is apphed 2314 to the gray scale image to create another gray scale test image. In an actual embodiment of the invention, five gray scale test images are produced. Each of the gray scale test images is displayed 2318 on the video monitor 116. The same gray scale test images are then printed 2320 on the printer 118 to be calibrated.
• the test images should be arranged on the printed page as they are displayed on the display so that the user may intuitively select 2322 the "best" test image appearing on the printed test page. (The "best” test image is a subjective determination by the user.)
• the genotype of the image is assigned as a printer gray scale genotype.
• a red-dominant test image containing subject matter with many red tones is retrieved 2324.
• the printer gray scale genotype is apphed 2326 to the red-dominant test image (as is described above with reference to FIGURE 11) to create a gray scale adjusted test image.
• a red-dominant template genotype is apphed 2328 to the gray scale adjusted test image to create a red-dominant test image.
• a decision 2330 returns control to the block 2328 to apply a different red-dominant template genotype to the gray scale adjusted test image to produce another red-dominant test image.
• the decision 2330 directs control to the retrieval 2332 of a green- dominant test image.
• the printer gray scale genotype is apphed 2334 to the green- dominant test image to create a gray scale adjusted test image.
• a set of green- dominant template genotypes are apphed 2336 to the gray scale adjusted test image to create a set of green-dominant test images.
• a decision 2338 returns control to the block 2336 until aU green-dominant test images have been produced.
• the process is next repeated for a blue-dominant test image that is retrieved 2340 and adjusted by applying 2342 the printer gray scale genotype to the blue-dominant test image to create a gray scale adjusted test image.
• a set of blue- dominant genotypes are apphed 2344 to create a set of blue- dominant test images using the control loop provided by decision 2346.
• the decision 2346 directs control to the display 2348 of each of the red-dominant, green- dominant, and blue-dominant test images on the video monitor 116.
• Each of the red-dominant, green-dominant and blue-dominant test images are printed 2350 on a test page by the printer 118.
• the test images should be arranged as they are displayed on the display so that the user can intuitively select the images from the printed page on the video display.
• the user chooses 2352 a "best" red-dominant test image that appears on the printed test page and selects the corresponding red-dominant test image on the video monitor 116. This selects the genotype of the selected test image as the printer red- dominant genotype.
• the user chooses 2354 the "best" green-dominant test image that appears on the printed test page and selects the corresponding green- dominant test image on the display. This assigns the genotype of the selected test image as the printer green-dominant genotype.
• the user then chooses 2356 the "best" blue- dominant test image that appears on the printed test page and selects the corresponding blue-dominant test image on the display.
• the genotype of the selected test image is assigned as the printer blue-dominant genotype.
• the printer gray scale genotype, the printer red- dominant genotype, the printer green- dominant genotype, and the printer blue-dominant genotype are recombined 2358 to form a printer cahbration genotype that is stored as a printer cahbration template genotype and associated with the printer 118 being cahbrated.
• the image to be printed can be modified using the printer cahbration genotype as it is sent to the printer to be printed so that the image printed matches the image displayed as closely as possible.
• the printer cahbration method is then done 2360. WhUe the preferred embodiment of the invention has been iUustrated and described, it wiU be appreciated that various changes can be made therein without departing from the spirit and scope of the invention.

Applications Claiming Priority (9)

Publications (1)

Family

ID=27493777

Family Applications (1)

Country Status (3)

Families Citing this family (7)

Family Cites Families (2)

• 1999
  • 1999-12-03 WO PCT/US1999/028676 patent/WO2000033207A1/en not_active Application Discontinuation
  • 1999-12-03 AU AU23521/00A patent/AU2352100A/en not_active Abandoned
  • 1999-12-03 EP EP99967186A patent/EP1147471A1/de not_active Withdrawn

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events