US7668402B2 - Method for generating a composite image - Google Patents

Method for generating a composite image Download PDF

Info

Publication number
US7668402B2
US7668402B2 US10/982,386 US98238604A US7668402B2 US 7668402 B2 US7668402 B2 US 7668402B2 US 98238604 A US98238604 A US 98238604A US 7668402 B2 US7668402 B2 US 7668402B2
Authority
US
United States
Prior art keywords
image
composite image
overlaid
separate image
separate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US10/982,386
Other versions
US20060097139A1 (en
Inventor
Gene A. Grindstaff
Sheila G. Whitaker
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Intergraph Corp
Original Assignee
Intergraph Technologies Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Intergraph Technologies Co filed Critical Intergraph Technologies Co
Priority to US10/982,386 priority Critical patent/US7668402B2/en
Assigned to INTERGRAPH HARDWARE TECHNOLOGIES COMPANY reassignment INTERGRAPH HARDWARE TECHNOLOGIES COMPANY ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GRINDSTAFF, GENE A., WHITAKER, SHEILA G.
Priority to BRPI0517669-7A priority patent/BRPI0517669A/en
Priority to EP05802850A priority patent/EP1807804B1/en
Priority to RU2007120940/07A priority patent/RU2475850C2/en
Priority to AU2005305293A priority patent/AU2005305293B2/en
Priority to CN2005800429980A priority patent/CN101080744B/en
Priority to PCT/US2005/035075 priority patent/WO2006052335A2/en
Priority to CA002586242A priority patent/CA2586242A1/en
Priority to DE602005027884T priority patent/DE602005027884D1/en
Priority to AT05802850T priority patent/ATE508437T1/en
Priority to JP2007540307A priority patent/JP4767258B2/en
Priority to TW094134888A priority patent/TWI464705B/en
Publication of US20060097139A1 publication Critical patent/US20060097139A1/en
Assigned to INTERGRAPH SOFTWARE TECHNOLOGIES reassignment INTERGRAPH SOFTWARE TECHNOLOGIES ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: INTERGRAPH HARDWARE TECHNOLOGIES
Assigned to INTERGRAPH SOFTWARE TECHNOLOGIES COMPANY reassignment INTERGRAPH SOFTWARE TECHNOLOGIES COMPANY ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: INTERGRAPH HARDWARE TECHNOLOGIES COMPANY
Assigned to MORGAN STANLEY & CO. INCORPORATED reassignment MORGAN STANLEY & CO. INCORPORATED FIRST LIEN INTELLECTUAL PROPERTY SECURITY AGREEMENT Assignors: COBALT HOLDING COMPANY, COBALT MERGER CORP., DAISY SYSTEMS INTERNATIONAL, INC., INTERGRAPH (ITALIA), LLC, INTERGRAPH ASIA PACIFIC, INC., INTERGRAPH CHINA, INC., INTERGRAPH COMPUTER SYSTEMS HOLDING, INC., INTERGRAPH CORPORATION, INTERGRAPH DC CORPORATION - SUBSIDIARY 3, INTERGRAPH DISC, INC., INTERGRAPH EUROPEAN MANUFACTURING, LLC, INTERGRAPH HARDWARE TECHNOLOGIES COMPANY, INTERGRAPH PROPERTIES COMPANY, INTERGRAPH SERVICES COMPANY, INTERGRAPH SOFTWARE TECHNOLOGIES COMPANY, M & S COMPUTING INVESTMENTS, INC., WORLDWIDE SERVICES, INC., Z/I IMAGING CORPORATION
Assigned to MORGAN STANLEY & CO. INCORPORATED reassignment MORGAN STANLEY & CO. INCORPORATED SECOND LIEN INTELLECTUAL PROPERTY SECURITY AGREEMENT Assignors: COBALT HOLDING COMPANY, COBALT MERGER CORP., DAISY SYSTEMS INTERNATIONAL, INC., INTERGRAPH (ITALIA), LLC, INTERGRAPH ASIA PACIFIC, INC., INTERGRAPH CHINA, INC., INTERGRAPH COMPUTER SYSTEMS HOLDING, INC., INTERGRAPH CORPORATION, INTERGRAPH DC CORPORATION - SUBSIDIARY 3, INTERGRAPH DISC, INC., INTERGRAPH EUROPEAN MANUFACTURING, LLC, INTERGRAPH HARDWARE TECHNOLOGIES COMPANY, INTERGRAPH PROPERTIES COMPANY, INTERGRAPH SERVICES COMPANY, INTERGRAPH SOFTWARE TECHNOLOGIES COMPANY, M & S COMPUTING INVESTMENTS, INC., WORLDWIDE SERVICES, INC., Z/I IMAGING CORPORATION
Priority to IL182962A priority patent/IL182962A/en
Publication of US7668402B2 publication Critical patent/US7668402B2/en
Application granted granted Critical
Assigned to INTERGRAPH SERVICES COMPANY, Z/I IMAGING CORPORATION, INTERGRAPH PP&M US HOLDING, INC., WORLDWIDE SERVICES, INC., M&S COMPUTING INVESTMENTS, INC., COADE INTERMEDIATE HOLDINGS, INC., INTERGRAPH DC CORPORATION - SUBSIDIARY 3, INTERGRAPH DISC, INC., Intergraph Technologies Company, COADE HOLDINGS, INC., INTERGRAPH (ITALIA), LLC, INTERGRAPH CHINA, INC., INTERGRAPH CORPORATION, INTERGRAPH EUROPEAN MANUFACTURING, LLC, ENGINEERING PHYSICS SOFTWARE, INC., INTERGRAPH ASIA PACIFIC, INC., INTERGRAPH HOLDING COMPANY (F/K/A COBALT HOLDING COMPANY) reassignment INTERGRAPH SERVICES COMPANY TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST Assignors: MORGAN STANLEY & CO. INCORPORATED
Assigned to INTERGRAPH ASIA PACIFIC, INC., COADE INTERMEDIATE HOLDINGS, INC., INTERGRAPH HOLDING COMPANY (F/K/A COBALT HOLDING COMPANY), INTERGRAPH CORPORATION, INTERGRAPH CHINA, INC., Z/I IMAGING CORPORATION, WORLDWIDE SERVICES, INC., INTERGRAPH PP&M US HOLDING, INC., COADE HOLDINGS, INC., INTERGRAPH SERVICES COMPANY, INTERGRAPH (ITALIA), LLC, INTERGRAPH DISC, INC., M&S COMPUTING INVESTMENTS, INC., INTERGRAPH EUROPEAN MANUFACTURING, LLC, Intergraph Technologies Company, ENGINEERING PHYSICS SOFTWARE, INC., INTERGRAPH DC CORPORATION - SUBSIDIARY 3 reassignment INTERGRAPH ASIA PACIFIC, INC. TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST Assignors: WACHOVIA BANK, NATIONAL ASSOCIATION
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation

Definitions

  • the present invention relates to video processing, and, more particularly, to generating a composite image.
  • the lighting in the store might have been poor.
  • the subject might not have been facing the camera or part of the subject's face might have been hidden by bystanders.
  • the image might have been recorded on a medium such as magnetic tape subject to deterioration.
  • Construction of a composite image is an interactive process.
  • a witness working with a police artist guides manipulation of the appearance and location of features such as ears and eyes until she feels that the composite image resembles her memory of the face.
  • a person working with a separate image of a face instead of with a witness has the advantage of not relying on memory.
  • the person working with a separate image lacking a witness to tell him when to stop, relies on his own criteria in deciding when resemblance of the composite image to the separate image is close enough.
  • the person working with a separate image may find himself trying one combination after another, unable to home in on the combination associated with a composite image most similar to the separate image.
  • a method for generating a composite image from a separate image includes receiving the separate image into a computer system and comparing the separate image to the composite image, producing a mismatch value in the process. At least one of the composite image and the separate image is modified so as to reduce the mismatch value.
  • a nondigital image may be digitized to form a separate image.
  • a method for generating a composite image may include modifying a feature such as a mouth, a nose, an eye, an eyebrow, and ear, a hairstyles and a face shape and the location of the feature in the composite image.
  • a method for generating a composite image may include modifying a position, an orientation, a size, and an appearance in the separate image.
  • a method for generating a composite image may include displaying a toolbar for modifying the separate image, and, may further include displaying toolbar buttons for modifying the position of the separate image by shifting the separate image left, right, up, and down, for modifying the orientation of the separate image by rotation clockwise and counterclockwise, by modifying the size of the separate image by enlargement and reduction, and for modifying the appearance of the separate image by selection of a color and grayscale image.
  • the toolbar may include buttons for increasing the intensity of the composite image and decreasing the intensity of the separate image and for decreasing the intensity of the composite image and increasing the intensity of the separate image.
  • a method for generating a composite image may include displaying an overlay image containing the modified composite image and the modified separate image, where the modified composite image and the modified separate image may be displayed individually.
  • a method for generating a composite image may include comparing the separate image to the composite image pixel by pixel. Blank white areas of the composite image may be treated as blank black areas. A composite image average light level may be calculated and subtracted from composite image. A separate image average light level may be calculated and subtracted from the separate image.
  • a method for calculating a mismatch value includes calculating an overlaid composite image average pixel intensity includes calculating an adjusted overlaid composite image pixel intensity by subtracting the overlaid composite image average pixel intensity from an intensity of each pixel in the overlaid composite image, calculating an overlaid separate image average pixel intensity, calculating an adjusted overlaid separate image pixel intensity by subtracting the overlaid separate image average pixel intensity from an intensity of each pixel in the overlaid separate image, calculating an overlaid pixel intensity difference by subtracting each adjusted overlaid composite image pixel intensity from the adjusted separate image pixel intensity for each pixel, calculating an overlaid pixel intensity difference absolute value for each pixel, calculating a total overlaid intensity difference by adding the overlaid pixel intensity difference absolute value of all pixels, calculating a total overlaid separate intensity difference by adding the adjusted overlaid separate image pixel intensities of all pixels, and calculating a mismatch value by dividing the total overlaid intensity difference
  • a system for generating a composite image based on a separate image includes means for receiving a separate image, means for comparing the separate image to the composite image, producing a mismatch, value, and means for modifying at least one of the composite image and the separate image to reduce the mismatch value.
  • means for receiving may include an antenna and means for comparing the separate image to the composite image may include a computer and a monitor.
  • the computer may include a memory storing a computer code for execution by the computer to compare the separate image to the composite image and produce a mismatch value.
  • the means for modifying at least one of the composite image and the separate image to reduce the mismatch value may include a device for selecting a modification.
  • the selection device may include a mouse.
  • a computer program product for use on a computer system for generating a composite image based on a separate image comprises a computer usable medium having computer readable program code where the computer readable program code includes program code for comparing the separate image to the composite image, producing a mismatch value and program code for modifying at least one of the composite image and the separate image to reduce the mismatch value.
  • the program code for comparing the separate image to the composite image may include program code for calculating an overlaid composite image average pixel intensity, program code for calculating an adjusted overlaid composite image pixel intensity by subtracting the overlaid composite image average pixel intensity from an intensity of each pixel in the overlaid composite image, program code for calculating an overlaid separate image average pixel intensity, program code for calculating an adjusted overlaid separate image pixel intensity by subtracting the overlaid separate image average pixel intensity from an intensity of each pixel in the overlaid separate image, program code for calculating an overlaid pixel intensity difference by subtracting each adjusted overlaid composite image pixel intensity from the adjusted separate image pixel intensity for each pixel, program code for calculating an overlaid pixel intensity difference absolute value for each pixel, program code for calculating a total overlaid intensity difference by adding thee overlaid pixel intensity difference absolute value of all pixels, program code for calculating a total overlaid separate intensity difference by adding the adjusted overlaid separate image pixel intensities of
  • the program code for modifying at least one of the composite image and the separate image to reduce the mismatch value may include program code for incorporating a feature into the composite image, program code for modifying the location of a feature in the composite image, program code for modifying at least one of a position, an orientation, a size, and an appearance of the separate image, program code for modifying the position of the separate image by shifting the separate image left, right, up, and down, program code for modifying the orientation of the separate image by rotating the separate image clockwise and counterclockwise, program code for modifying the size of the separate image by enlargement and by reduction, program code for modifying the appearance of the separate image by selecting to display the separate image in color and in grayscale, and program code for increasing the intensity of the composite image and decreasing the intensity of the separate image and for decreasing the intensity of the composite image and increasing the intensity of the separate image.
  • FIG. 1 is a schematic illustration of a computer system for generating a composite image
  • FIG. 2 illustrates a computer display window including a composite image
  • FIG. 3 illustrates a computer display window including a composite image, a separate image, an overlay image, and a mismatch value
  • FIG. 4 illustrates a toolbar containing buttons for modifying a separate image and a composite image
  • FIG. 5 illustrates a process for calculating a mismatch value
  • FIG. 6 illustrates the process for generating the composite image based on the separate image.
  • a composite image is a digital image of an individual created by individual selection and assembly of features to represent the appearance of the individual.
  • a separate image is a digital representation of an actual image taken of an individual.
  • An overlay image is a digital image of the composite image overlaying the separate image.
  • the sizes, orientations, positions, and intensities of the images may be modified before intensities at corresponding pixels for the modified composite and separate images are combined.
  • Combining may correspond to summing the intensities at corresponding pixels for the modified composite and separate images, followed by averaging the sum.
  • the present invention enables a composite image to be created and compared to a separate image. Based on the comparison, the composite image may be edited to better resemble the separate image through use of an indicator of resemblance between the composite and separate images.
  • FIG. 1 shows a computer system 100 containing a computer 105 , a display 110 , and a mouse 115 on which a computer program for generating a composite image operates.
  • a computer program for generating a composite image operates.
  • One example of an existing composite image generating program is Faces, The Ultimate Composite PictureTM.
  • an operator 120 uses the composite image generating program, an operator 120 displays and modifies a composite image 127 .
  • FIG. 2 shows a composite image window 125 associated with the composite image generating program on the display 110 .
  • the composite image 127 contained in composite image window 125 reflects all of the insertions, deletions, and modifications of features.
  • the operator 120 selects features 200 from a library that includes features such as eyes 210 , eyebrows 215 , ears 220 , noses 225 , mouths 230 , lips 245 , teeth 240 , hairstyles 235 , moles 250 , scars 255 , and faces 260 .
  • the operator may further select features 200 on the basis of qualities such as color (e.g., eyes, hair, and face), shape (e.g., eyes, face, ears, and nose), and orientation (e.g., eyes and face).
  • the operator may alter feature locations by dragging a feature 200 across the display 110 of the composite image 127 under the control of the mouse 115 .
  • FIG. 3 shows a composite image evaluation window 130 also located on the display 110 .
  • the composite image evaluation window 130 includes three images: a composite image 131 , a separate image 132 , and an overlay image 133 .
  • On the left is the composite image 131 , a copy of composite image 127 associated with the composite image window 125 and is regularly updated, for example, every 500 milliseconds.
  • In the middle is the separate image 132 .
  • On the right is the overlay image 133 that includes the composite image 131 overlaying the separate image 132 .
  • the composite image evaluation window 130 also contains a Mismatch value 134 . As the operator 120 alters the composite image 131 to match the separate image 132 , the Mismatch value 134 varies from a value approximately equal to 1 that is indicative of a complete mismatch to a value approximately equal to 0 that is indicative of a perfect match.
  • the separate image 132 in digital form, is stored in the computer 105 from a digital source 140 that may include a digital camera and the Internet and from a nondigital source 142 that may include a film camera and a video cassette recorder (VCR).
  • a nondigital image 131 from nondigital source 142 is first digitized by a digitizer 144 before being stored as separate image 132 .
  • the separate image 132 may be a file type such as BMP, JPEG, WMF, GIF, PNG, TIFF, and Exif.
  • the separate image 132 Prior to comparison of the composite image 131 with the separate image 132 , the separate image 132 may be modified to remove or reduce the influence of aspects not dependent on features 200 . Examples of modifications to the separate image 132 include shifting (up, down, left, and right), rotation (clockwise and counterclockwise), sizing (enlargement and reduction), and display (color and grayscales).
  • the cumulative effects of the above modifications to the separate image 132 are displayed in the overlay image 133 where the composite image 315 is overlaid with the modified separate image 310 .
  • the overlaid composite 315 and the modified separate 310 images may be further modified relative to one another.
  • the intensity of the overlaid modified composite image 315 may be increased and the intensity of the overlaid modified separate image 310 decreased or the intensity of the overlaid modified separate image 310 may be decreased and the intensity of the overlaid modified composite image 315 decreased.
  • FIG. 4 illustrates an embodiment of a toolbar 135 incorporated into the display 110 for implementing modifications to the overlaid separate 310 and overlaid composite 315 images.
  • the toolbar 135 may contain buttons 400 for effecting modifications to the overlaid separate image 310 (shifting up 420 , down 425 , left 410 , and right 415 ; rotation clockwise 430 and counterclockwise 435 ; enlargement 450 and reduction 455 ; display in color 440 and in greyscale 445 ) and to the overlaid separate image 310 relative to the overlaid composite image 315 (increase overlaid separate image intensity and decrease overlaid composite image intensity 465 ; decrease overlaid separate image intensity and increase overlaid composite image intensity 460 ).
  • the operator 120 may use the mouse 115 to selectively apply the modifications by selecting particular buttons. Each selection of a button 400 may apply a small increment of the indicated change, thereby allowing the operator 120 to make fine adjustments to the overlaid images 310 and 315 .
  • the display 110 For each overlay image 133 containing an overlaid composite image 315 and an overlaid separate image 310 , the display 110 also contains a Mismatch value 134 .
  • the overlaid composite image 315 and the overlaid separate image 310 are displayed at the same resolution, i.e., as similar numbers of pixels, for example, corresponding to arrays of 600 ⁇ 800 pixels.
  • the Mismatch value 134 quantifies the resemblance of the overlaid composite image 315 with the overlaid separate image 310 and is calculated on a pixel by pixel basis.
  • FIG. 5 illustrates the process 500 for calculation of the Mismatch value 134 .
  • the intensities of all of the pixels in the overlaid composite image 315 are added and divided by the total number of pixels in the overlaid composite image 315 to find the average pixel intensity for the overlaid composite image 315 , Av(oci).
  • Av(oci) is subtracted from the intensity of each pixel in the overlaid composite image 315 to find the adjusted overlaid composite image pixel intensities.
  • step 508 the intensities of all of the pixels in the overlaid separate image 310 are added and divided by the total number of pixels in the overlaid separate image 310 to find the average pixel intensity for the overlaid separate image, Av(osi).
  • Av(osi) is subtracted from the intensity of each pixel in the overlaid separate image 310 to find the adjusted overlaid separate image pixel intensities.
  • each adjusted overlaid composite image pixel intensity is subtracted from the corresponding adjusted overlaid separate image pixel intensity to find the overlaid pixel intensity differences.
  • each adjusted overlaid separate image pixel intensity may be subtracted from the corresponding adjusted overlaid composite image pixel intensity to find the overlaid pixel intensity differences.
  • the absolute value is taken of each overlaid pixel intensity difference.
  • the absolute values of all overlaid pixel intensity differences are added to find the total overlaid pixel intensity difference.
  • all the adjusted overlaid separate image pixel intensities are added to find the total overlaid separate image pixel intensity.
  • the total overlaid pixel intensity difference is divided by the total overlaid separate image pixel intensity to find the Mismatch value 134 .
  • the intensities of the pixels of the overlaid composite image and separate image may be adjusted to eliminate systemic differences between the overlaid images.
  • the backgrounds of the overlaid composite image and overlaid separate image may be made the same intensity. Since the background of the composite image is often white and the background of the separate image dark, the background of the overlaid composite image may be treated as black in the calculation of the Mismatch value 134 to yield a lower Mismatch value 134 with less skewing than would result from absolute pixel comparison.
  • the Mismatch value 134 changes accordingly, increasing as a result of a poorer match and decreasing as a result of a better match.
  • the composite image 127 corresponding to the best match to date may be stored in the computer 105 .
  • the Mismatch value 134 first reaches a predetermined threshold value or when the Mismatch value 134 fails to fall below the Mismatch value of the currently stored best composite image over a predetermined number of changes to the composite image, the currently stored best composite image may be declared to be the final composite image.
  • FIG. 6 illustrates the process 600 for generating the composite image based on the separate image 132 .
  • the computer system receives the separate image.
  • the separate image 132 is compared with the composite image 131 , producing the Mismatch value 134 in the process. If the Mismatch value 134 is satisfactory, the process 600 ends. If the Mismatch value 134 is not satisfactory, the process 600 proceeds to step 608 where at least one of the composite image 131 and the separate image 132 is modified to reduce the Mismatch value 134 . Process 600 then returns to step 606 for another comparison.
  • embodiments of the invention may be implemented at least in part in any conventional computer programming language. For example, some embodiments may be implemented in a procedural programming language (e.g., “C”), or in an object oriented programming language (e.g., “C++”). Other embodiments of the invention may be implemented as preprogrammed hardware elements (e.g., application specific integrated circuits, FPGAs, and digital signal processors), or other related components.
  • C procedural programming language
  • object oriented programming language e.g., “C++”.
  • preprogrammed hardware elements e.g., application specific integrated circuits, FPGAs, and digital signal processors
  • the disclosed apparatus and method may be implemented as a computer program product for use with a computer system.
  • Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium.
  • the medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., WIFI, microwave, infrared or other transmission techniques).
  • the series of computer instructions can embody all or part of the functionality previously described herein with respect to the system.
  • Such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems.
  • such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies.
  • Such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web).
  • a computer system e.g., on system ROM or fixed disk
  • a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web).
  • some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments of the invention are implemented as entirely hardware, or entirely software.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Editing Of Facsimile Originals (AREA)
  • Processing Or Creating Images (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)
  • Heating, Cooling, Or Curing Plastics Or The Like In General (AREA)
  • Ink Jet Recording Methods And Recording Media Thereof (AREA)
  • Non-Silver Salt Photosensitive Materials And Non-Silver Salt Photography (AREA)

Abstract

A method for generating a composite image includes receiving a separate image into a computer system, comparing the separate image to the composite image so as to generate a mismatch value, and modifying at least one of the composite image and the separate image to reduce the mismatch value.

Description

TECHNICAL FIELD
The present invention relates to video processing, and, more particularly, to generating a composite image.
BACKGROUND
Often, there is a need to identify a person through comparison of the person's face with a recorded image of a face. Such situations regularly arise in police investigations where, for example, the police search for a suspect on the basis of a security camera image.
Difficulty arises when the image used for comparison is of poor quality. The lighting in the store might have been poor. The subject might not have been facing the camera or part of the subject's face might have been hidden by bystanders. The image might have been recorded on a medium such as magnetic tape subject to deterioration.
There are advantages in converting the recorded image to a standard frontal and profile composite image similar to the image composed by a person under the guidance of a police artist. Such an image can easily be the basis for comparison. The image can maintain its fidelity despite noise added in transmission and copying. The image can be updated to allow for the passage of time or for modifications associated with a particular area, such as a beard.
Construction of a composite image is an interactive process. A witness working with a police artist guides manipulation of the appearance and location of features such as ears and eyes until she feels that the composite image resembles her memory of the face. A person working with a separate image of a face instead of with a witness has the advantage of not relying on memory. However, the person working with a separate image, lacking a witness to tell him when to stop, relies on his own criteria in deciding when resemblance of the composite image to the separate image is close enough. With an unlimited number of combinations of features and locations available, the person working with a separate image may find himself trying one combination after another, unable to home in on the combination associated with a composite image most similar to the separate image.
SUMMARY OF THE INVENTION
In accordance with one aspect of the invention, a method for generating a composite image from a separate image includes receiving the separate image into a computer system and comparing the separate image to the composite image, producing a mismatch value in the process. At least one of the composite image and the separate image is modified so as to reduce the mismatch value.
In some embodiments, a nondigital image may be digitized to form a separate image.
In certain embodiments, a method for generating a composite image may include modifying a feature such as a mouth, a nose, an eye, an eyebrow, and ear, a hairstyles and a face shape and the location of the feature in the composite image.
In other embodiments, a method for generating a composite image may include modifying a position, an orientation, a size, and an appearance in the separate image. A method for generating a composite image may include displaying a toolbar for modifying the separate image, and, may further include displaying toolbar buttons for modifying the position of the separate image by shifting the separate image left, right, up, and down, for modifying the orientation of the separate image by rotation clockwise and counterclockwise, by modifying the size of the separate image by enlargement and reduction, and for modifying the appearance of the separate image by selection of a color and grayscale image. Further, the toolbar may include buttons for increasing the intensity of the composite image and decreasing the intensity of the separate image and for decreasing the intensity of the composite image and increasing the intensity of the separate image.
In illustrative embodiments, a method for generating a composite image may include displaying an overlay image containing the modified composite image and the modified separate image, where the modified composite image and the modified separate image may be displayed individually.
In still further embodiments, a method for generating a composite image may include comparing the separate image to the composite image pixel by pixel. Blank white areas of the composite image may be treated as blank black areas. A composite image average light level may be calculated and subtracted from composite image. A separate image average light level may be calculated and subtracted from the separate image.
In accordance with a further aspect of the invention, a method for calculating a mismatch value includes calculating an overlaid composite image average pixel intensity includes calculating an adjusted overlaid composite image pixel intensity by subtracting the overlaid composite image average pixel intensity from an intensity of each pixel in the overlaid composite image, calculating an overlaid separate image average pixel intensity, calculating an adjusted overlaid separate image pixel intensity by subtracting the overlaid separate image average pixel intensity from an intensity of each pixel in the overlaid separate image, calculating an overlaid pixel intensity difference by subtracting each adjusted overlaid composite image pixel intensity from the adjusted separate image pixel intensity for each pixel, calculating an overlaid pixel intensity difference absolute value for each pixel, calculating a total overlaid intensity difference by adding the overlaid pixel intensity difference absolute value of all pixels, calculating a total overlaid separate intensity difference by adding the adjusted overlaid separate image pixel intensities of all pixels, and calculating a mismatch value by dividing the total overlaid intensity difference by the total overlaid separate image intensity difference.
In accordance with an additional aspect of the invention, a system for generating a composite image based on a separate image includes means for receiving a separate image, means for comparing the separate image to the composite image, producing a mismatch, value, and means for modifying at least one of the composite image and the separate image to reduce the mismatch value.
In certain embodiments, means for receiving may include an antenna and means for comparing the separate image to the composite image may include a computer and a monitor. The computer may include a memory storing a computer code for execution by the computer to compare the separate image to the composite image and produce a mismatch value. The means for modifying at least one of the composite image and the separate image to reduce the mismatch value may include a device for selecting a modification. The selection device may include a mouse.
In accordance with another aspect of the invention, a computer program product for use on a computer system for generating a composite image based on a separate image comprises a computer usable medium having computer readable program code where the computer readable program code includes program code for comparing the separate image to the composite image, producing a mismatch value and program code for modifying at least one of the composite image and the separate image to reduce the mismatch value.
In additional embodiments, the program code for comparing the separate image to the composite image may include program code for calculating an overlaid composite image average pixel intensity, program code for calculating an adjusted overlaid composite image pixel intensity by subtracting the overlaid composite image average pixel intensity from an intensity of each pixel in the overlaid composite image, program code for calculating an overlaid separate image average pixel intensity, program code for calculating an adjusted overlaid separate image pixel intensity by subtracting the overlaid separate image average pixel intensity from an intensity of each pixel in the overlaid separate image, program code for calculating an overlaid pixel intensity difference by subtracting each adjusted overlaid composite image pixel intensity from the adjusted separate image pixel intensity for each pixel, program code for calculating an overlaid pixel intensity difference absolute value for each pixel, program code for calculating a total overlaid intensity difference by adding thee overlaid pixel intensity difference absolute value of all pixels, program code for calculating a total overlaid separate intensity difference by adding the adjusted overlaid separate image pixel intensities of all pixels, and program code for calculating a mismatch value by dividing the total overlaid intensity difference by the total overlaid separate image intensity difference.
In other embodiments, the program code for modifying at least one of the composite image and the separate image to reduce the mismatch value may include program code for incorporating a feature into the composite image, program code for modifying the location of a feature in the composite image, program code for modifying at least one of a position, an orientation, a size, and an appearance of the separate image, program code for modifying the position of the separate image by shifting the separate image left, right, up, and down, program code for modifying the orientation of the separate image by rotating the separate image clockwise and counterclockwise, program code for modifying the size of the separate image by enlargement and by reduction, program code for modifying the appearance of the separate image by selecting to display the separate image in color and in grayscale, and program code for increasing the intensity of the composite image and decreasing the intensity of the separate image and for decreasing the intensity of the composite image and increasing the intensity of the separate image.
BRIEF DESCRIPTION OF THE DRAWINGS
The foregoing features of the invention will be more readily understood by reference to the following detailed description, taken with reference to the accompanying drawings, in which:
FIG. 1 is a schematic illustration of a computer system for generating a composite image;
FIG. 2 illustrates a computer display window including a composite image;
FIG. 3 illustrates a computer display window including a composite image, a separate image, an overlay image, and a mismatch value;
FIG. 4 illustrates a toolbar containing buttons for modifying a separate image and a composite image;
FIG. 5 illustrates a process for calculating a mismatch value; and
FIG. 6 illustrates the process for generating the composite image based on the separate image.
 DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS
Definitions. As used in this description and the accompanying claims, the following terms shall have the meanings indicated, unless the context otherwise requires:
A composite image is a digital image of an individual created by individual selection and assembly of features to represent the appearance of the individual.
A separate image is a digital representation of an actual image taken of an individual.
An overlay image is a digital image of the composite image overlaying the separate image. In overlaying the composite image onto the separate image, the sizes, orientations, positions, and intensities of the images may be modified before intensities at corresponding pixels for the modified composite and separate images are combined. Combining may correspond to summing the intensities at corresponding pixels for the modified composite and separate images, followed by averaging the sum.
In one embodiment, the present invention enables a composite image to be created and compared to a separate image. Based on the comparison, the composite image may be edited to better resemble the separate image through use of an indicator of resemblance between the composite and separate images.
FIG. 1 shows a computer system 100 containing a computer 105, a display 110, and a mouse 115 on which a computer program for generating a composite image operates. One example of an existing composite image generating program is Faces, The Ultimate Composite Picture™. Using the composite image generating program, an operator 120 displays and modifies a composite image 127.
FIG. 2 shows a composite image window 125 associated with the composite image generating program on the display 110. The composite image 127 contained in composite image window 125 reflects all of the insertions, deletions, and modifications of features. The operator 120 selects features 200 from a library that includes features such as eyes 210, eyebrows 215, ears 220, noses 225, mouths 230, lips 245, teeth 240, hairstyles 235, moles 250, scars 255, and faces 260. The operator may further select features 200 on the basis of qualities such as color (e.g., eyes, hair, and face), shape (e.g., eyes, face, ears, and nose), and orientation (e.g., eyes and face). The operator may alter feature locations by dragging a feature 200 across the display 110 of the composite image 127 under the control of the mouse 115.
FIG. 3 shows a composite image evaluation window 130 also located on the display 110. The composite image evaluation window 130 includes three images: a composite image 131, a separate image 132, and an overlay image 133. On the left is the composite image 131, a copy of composite image 127 associated with the composite image window 125 and is regularly updated, for example, every 500 milliseconds. In the middle is the separate image 132. On the right is the overlay image 133 that includes the composite image 131 overlaying the separate image 132. The composite image evaluation window 130 also contains a Mismatch value 134. As the operator 120 alters the composite image 131 to match the separate image 132, the Mismatch value 134 varies from a value approximately equal to 1 that is indicative of a complete mismatch to a value approximately equal to 0 that is indicative of a perfect match.
The separate image 132, in digital form, is stored in the computer 105 from a digital source 140 that may include a digital camera and the Internet and from a nondigital source 142 that may include a film camera and a video cassette recorder (VCR). A nondigital image 131 from nondigital source 142 is first digitized by a digitizer 144 before being stored as separate image 132. The separate image 132 may be a file type such as BMP, JPEG, WMF, GIF, PNG, TIFF, and Exif. Prior to comparison of the composite image 131 with the separate image 132, the separate image 132 may be modified to remove or reduce the influence of aspects not dependent on features 200. Examples of modifications to the separate image 132 include shifting (up, down, left, and right), rotation (clockwise and counterclockwise), sizing (enlargement and reduction), and display (color and grayscales).
The cumulative effects of the above modifications to the separate image 132 are displayed in the overlay image 133 where the composite image 315 is overlaid with the modified separate image 310. The overlaid composite 315 and the modified separate 310 images may be further modified relative to one another. The intensity of the overlaid modified composite image 315 may be increased and the intensity of the overlaid modified separate image 310 decreased or the intensity of the overlaid modified separate image 310 may be decreased and the intensity of the overlaid modified composite image 315 decreased.
FIG. 4 illustrates an embodiment of a toolbar 135 incorporated into the display 110 for implementing modifications to the overlaid separate 310 and overlaid composite 315 images. The toolbar 135 may contain buttons 400 for effecting modifications to the overlaid separate image 310 (shifting up 420, down 425, left 410, and right 415; rotation clockwise 430 and counterclockwise 435; enlargement 450 and reduction 455; display in color 440 and in greyscale 445) and to the overlaid separate image 310 relative to the overlaid composite image 315 (increase overlaid separate image intensity and decrease overlaid composite image intensity 465; decrease overlaid separate image intensity and increase overlaid composite image intensity 460). The operator 120 may use the mouse 115 to selectively apply the modifications by selecting particular buttons. Each selection of a button 400 may apply a small increment of the indicated change, thereby allowing the operator 120 to make fine adjustments to the overlaid images 310 and 315.
For each overlay image 133 containing an overlaid composite image 315 and an overlaid separate image 310, the display 110 also contains a Mismatch value 134. The overlaid composite image 315 and the overlaid separate image 310 are displayed at the same resolution, i.e., as similar numbers of pixels, for example, corresponding to arrays of 600×800 pixels. The Mismatch value 134 quantifies the resemblance of the overlaid composite image 315 with the overlaid separate image 310 and is calculated on a pixel by pixel basis.
FIG. 5 illustrates the process 500 for calculation of the Mismatch value 134. In step 504, the intensities of all of the pixels in the overlaid composite image 315 are added and divided by the total number of pixels in the overlaid composite image 315 to find the average pixel intensity for the overlaid composite image 315, Av(oci). In step 506, Av(oci) is subtracted from the intensity of each pixel in the overlaid composite image 315 to find the adjusted overlaid composite image pixel intensities. In step 508, the intensities of all of the pixels in the overlaid separate image 310 are added and divided by the total number of pixels in the overlaid separate image 310 to find the average pixel intensity for the overlaid separate image, Av(osi). In step 510, Av(osi) is subtracted from the intensity of each pixel in the overlaid separate image 310 to find the adjusted overlaid separate image pixel intensities. In step 512, each adjusted overlaid composite image pixel intensity is subtracted from the corresponding adjusted overlaid separate image pixel intensity to find the overlaid pixel intensity differences. (Corresponding overlaid composite and separate image pixels have the same relative location in the image, i.e. the same horizontal and vertical position, as indicated schematically with pixels 320 and 325 in FIG. 3). Equivalently, each adjusted overlaid separate image pixel intensity may be subtracted from the corresponding adjusted overlaid composite image pixel intensity to find the overlaid pixel intensity differences. In step 514, the absolute value is taken of each overlaid pixel intensity difference. In step 516, the absolute values of all overlaid pixel intensity differences are added to find the total overlaid pixel intensity difference. In step 518, all the adjusted overlaid separate image pixel intensities are added to find the total overlaid separate image pixel intensity. In step 520, the total overlaid pixel intensity difference is divided by the total overlaid separate image pixel intensity to find the Mismatch value 134.
Prior to calculation of the Mismatch value 134, the intensities of the pixels of the overlaid composite image and separate image may be adjusted to eliminate systemic differences between the overlaid images. The backgrounds of the overlaid composite image and overlaid separate image may be made the same intensity. Since the background of the composite image is often white and the background of the separate image dark, the background of the overlaid composite image may be treated as black in the calculation of the Mismatch value 134 to yield a lower Mismatch value 134 with less skewing than would result from absolute pixel comparison.
As the operator 120 changes the composite image 127, the Mismatch value 134 changes accordingly, increasing as a result of a poorer match and decreasing as a result of a better match. The composite image 127 corresponding to the best match to date may be stored in the computer 105. When the Mismatch value 134 first reaches a predetermined threshold value or when the Mismatch value 134 fails to fall below the Mismatch value of the currently stored best composite image over a predetermined number of changes to the composite image, the currently stored best composite image may be declared to be the final composite image.
FIG. 6 illustrates the process 600 for generating the composite image based on the separate image 132. In step 604, the computer system receives the separate image. In step 606, the separate image 132 is compared with the composite image 131, producing the Mismatch value 134 in the process. If the Mismatch value 134 is satisfactory, the process 600 ends. If the Mismatch value 134 is not satisfactory, the process 600 proceeds to step 608 where at least one of the composite image 131 and the separate image 132 is modified to reduce the Mismatch value 134. Process 600 then returns to step 606 for another comparison.
Various embodiments of the invention may be implemented at least in part in any conventional computer programming language. For example, some embodiments may be implemented in a procedural programming language (e.g., “C”), or in an object oriented programming language (e.g., “C++”). Other embodiments of the invention may be implemented as preprogrammed hardware elements (e.g., application specific integrated circuits, FPGAs, and digital signal processors), or other related components.
In an alternative embodiment, the disclosed apparatus and method may be implemented as a computer program product for use with a computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. The medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., WIFI, microwave, infrared or other transmission techniques). The series of computer instructions can embody all or part of the functionality previously described herein with respect to the system.
Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies.
Such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web). Of course, some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments of the invention are implemented as entirely hardware, or entirely software.
Although various exemplary embodiments of the invention are disclosed above, it should be apparent that those skilled in the art can make various changes and modifications that will achieve some of the advantages of the invention without departing from the true scope of the invention.

Claims (46)

1. A method for generating a composite image in a computer based on a separate image, the method comprising:
receiving the separate image into a computer system;
using a digital process, comparing the separate image to the composite image in the computer pixel by pixel, producing a mismatch value; and
modifying at least one of the composite image and the separate image to reduce the mismatch value, thereby generating a final composite image.
2. A method for generating a composite image according to claim 1, further including digitizing a nondigital image to form the separate image.
3. A method for generating a composite image according to claim 1, wherein modifying the composite image includes incorporating a feature into the composite image.
4. A method for generating a composite image according to claim 3, wherein the feature includes at least one of a mouth, a nose, an eye, an eyebrow, an ear, a hairstyle, and a face shape.
5. A method for generating a composite image according to claim 3, wherein modifying the composite image includes modifying the location of the feature in the composite image.
6. A method for generating a composite image according to claim 1, wherein modifying the separate image includes modifying at least one of a position, an orientation, a size, and an appearance of the separate image.
7. A method for generating a composite image according to claim 1, wherein modifying includes displaying a toolbar.
8. A method for generating a composite image according to claim 7, wherein displaying a toolbar includes displaying buttons for modifying the position of the separate image by shifting the separate image left, right, up, and down.
9. A method for generating a composite image according to claim 7, wherein displaying a toolbar includes displaying buttons for modifying the orientation of the separate image by rotating the separate image clockwise and counterclockwise.
10. A method for generating a composite image according to claim 9, wherein displaying buttons for modifying the orientation of the separate image includes displaying a button for rotating the separate image clockwise.
11. A method for generating a composite image according to claim 9, wherein displaying buttons for modifying the orientation of the separate image includes displaying a button for rotating the separate image counterclockwise.
12. A method for generating a composite image according to claim 7, wherein displaying a toolbar includes displaying buttons for modifying the size of the separate image by enlargement and by reduction.
13. A method for generating a composite image according to claim 7, wherein displaying a toolbar includes displaying buttons for modifying the appearance of the separate image by selecting to display the separate image in color and in grayscale.
14. A method for generating a composite image according to claim 7, wherein displaying a toolbar includes displaying buttons for increasing the intensity of the composite image and decreasing the intensity of the separate image and for decreasing the intensity of the composite image and increasing the intensity of the separate image.
15. A method for generating a composite image according to claim 1, further including displaying an overlay image containing the modified composite image and the modified separate image.
16. A method for generating a composite image according to claim 15, further including individually displaying the modified composite image and the modified separate image.
17. A method for generating a composite image according to claim 1, wherein comparing the separate image to the composite image includes treating blank white areas of the composite image as if the blank white areas were blank black areas.
18. A method according to claim 1, wherein comparing the separate image to the composite image includes calculating an average light level for the composite image and subtracting the average composite image light level from the composite image.
19. A method according to claim 1, wherein comparing the separate image to the composite image includes calculating an average light level for the separate image and subtracting the separate image average light level from the separate image.
20. A computer program product for use on a computer system for calculating a mismatch value between an overlaid composite image and an overlaid separate image, the computer program product comprising a computer usable medium having computer readable program code thereon, the computer readable program code comprising program code for:
calculating an overlaid composite image average pixel intensity;
calculating an adjusted overlaid composite image pixel intensity by subtracting the overlaid composite image average pixel intensity from an intensity of each pixel in the overlaid composite image;
calculating an overlaid separate image average pixel intensity;
calculating an adjusted overlaid separate image pixel intensity by subtracting the overlaid separate image average pixel intensity from an intensity of each pixel in the overlaid separate image;
calculating an overlaid pixel intensity difference by subtracting each adjusted overlaid composite image pixel intensity from the adjusted overlaid separate image pixel intensity for each pixel;
calculating an overlaid pixel intensity difference absolute value for each pixel;
calculating a total overlaid pixel intensity difference by adding the overlaid pixel intensity difference absolute value of all pixels;
calculating a total overlaid separate image pixel intensity by adding the adjusted overlaid separate image pixel intensities of all pixels; and
calculating a mismatch value by dividing the total overlaid pixel intensity difference by the total overlaid separate image pixel intensity.
21. A system for generating a composite image based on a separate image, the system comprising:
means for receiving the separate image;
means for comparing the separate image to the composite image pixel by pixel, producing a mismatch value; and
means for modifying at least one of the composite image and the separate image to reduce the mismatch value, thereby generating a final composite image.
22. A system for generating a composite image according to claim 21, wherein the means for receiving includes an antenna.
23. A system for generating a composite image according to claim 21, wherein the means for comparing the separate image to the composite image includes a computer and a monitor.
24. A system for generating a composite image according to claim 21, wherein the computer includes a memory storing a computer code for execution by the computer to compare the separate image to the composite image and produce a mismatch value.
25. A system for generating a composite image according to claim 21, wherein the means for modifying at least one of the composite image and the separate image to reduce the mismatch value includes a device for selecting a modification.
26. A system for generating a composite image according to claim 25, wherein the selection device includes a mouse.
27. A computer program product for use on a computer system for generating a composite image based on a separate image, the computer program product comprising a computer usable medium having computer readable program code thereon, the computer readable program code comprising:
program code for comparing the separate image to the composite image pixel by pixel, producing a mismatch value; and
program code for modifying at least one of the composite image and the separate image to reduce the mismatch value, thereby generating a final composite image.
28. A computer program product according to claim 27, wherein the program code for comparing the separate image to the composite image, producing a mismatch value, includes program code for calculating an overlaid composite image average pixel intensity.
29. A computer program product according to claim 28, wherein the program code for comparing includes program code for calculating an adjusted overlaid composite image pixel intensity by subtracting the overlaid composite image average pixel intensity from an intensity of each pixel in the overlaid composite image.
30. A computer program product according to claim 29, wherein the program code for comparing includes program code for calculating an overlaid separate image average pixel intensity.
31. A computer program product according to claim 30, wherein the program code for comparing includes program code for calculating an adjusted overlaid separate image pixel intensity by subtracting the overlaid separate image average pixel intensity from an intensity of each pixel in the overlaid separate image.
32. A computer program product according to claim 31, wherein the program code for comparing includes program code for calculating an overlaid pixel intensity difference by subtracting each adjusted overlaid composite image pixel intensity from the adjusted overlaid separate image pixel intensity for each pixel.
33. A computer program product according to claim 32, wherein the program code for comparing includes program code for calculating an overlaid pixel intensity difference absolute value for each pixel.
34. A computer program product according to claim 33, wherein the program code for comparing includes program code for calculating a total overlaid pixel intensity difference by adding the overlaid pixel intensity difference absolute value of all pixels.
35. A computer program product according to claim 34, wherein the program code for comparing includes program code for calculating a total overlaid separate image pixel intensity by adding the adjusted overlaid separate image pixel intensities of all pixels.
36. A computer program product according to claim 35, wherein the program code for comparing includes program code for calculating a mismatch value by dividing the total overlaid pixel intensity difference by the total overlaid separate image pixel intensity.
37. A computer program product according to claim 27 wherein the program code for modifying at least one of the composite image and the separate image to reduce the mismatch value includes program code for incorporating a feature into the composite image.
38. A computer program product according to claim 37 wherein the program code for modifying at least one of the composite image and the separate image to reduce the mismatch value includes program code for modifying the location of the feature in the composite image.
39. A computer program product according to claim 27 wherein the program code for modifying at least one of the composite image and the separate image to reduce the mismatch value includes program code for modifying at least one of a position, an orientation, a size, and an appearance of the separate image.
40. A computer program product according to claim 39 wherein the program code for modifying at least one of the composite image and the separate image to reduce the mismatch value includes program code for modifying the position of the separate image by shifting the separate image left, right, up, and down.
41. A computer program product according to claim 39 wherein the program code for modifying at least one of the composite image and the separate image to reduce the mismatch value includes program code for modifying the orientation of the separate image by rotating the separate image clockwise and counterclockwise.
42. A computer program product according to claim 39 wherein the program code for modifying at least one of the composite image and the separate image to reduce the mismatch value includes program code for modifying the size of the separate image by enlargement and by reduction.
43. A computer program product according to claim 39 wherein the program code for modifying at least one of the composite image and the separate image to reduce the mismatch value includes program code for modifying the appearance of the separate image by selecting to display the separate image in color and in grayscale.
44. A computer program product according to claim 39 wherein the program code for modifying at least one of the composite image and the separate image to reduce the mismatch value includes program code for increasing the intensity of the composite image and decreasing the intensity of the separate image and for decreasing the intensity of the composite image and increasing the intensity of the separate image.
45. A method for generating a composite image in a computer based on a separate image, the method comprising:
receiving the separate image into a computer system;
using a digital process, comparing the separate image to the composite image in the computer, producing a mismatch value;
modifying at least one of the composite image and the separate image to reduce the mismatch value, thereby generating a final composite image, wherein modifying the separate image includes modifying at least one of a position, an orientation, a size, and an appearance of the separate image; and
displaying a toolbar on a display coupled to the computer that includes buttons for increasing the intensity of the composite image and decreasing the intensity of the separate image and for decreasing the intensity of the composite image and increasing the intensity of the separate image.
46. A computer program product for use on a computer system for generating a composite image based on a separate image, the computer program product comprising a computer usable medium having computer readable program code thereon, the computer readable program code comprising:
program code for comparing the separate image to the composite image including program code for:
(a) calculating an overlaid composite image average pixel intensity,
(b) calculating an adjusted overlaid composite image pixel intensity by subtracting the overlaid composite image average pixel intensity from an intensity of each pixel in the overlaid composite image,
(c) calculating an overlaid separate image average pixel intensity,
(d) calculating an adjusted overlaid separate image pixel intensity by subtracting the overlaid separate image average pixel intensity from an intensity of each pixel in the overlaid separate image,
(e) calculating an overlaid pixel intensity difference by subtracting each adjusted overlaid composite image pixel intensity from the adjusted overlaid separate image pixel intensity for each pixel,
(f) calculating an overlaid pixel intensity difference absolute value for each pixel,
(g) calculating a total overlaid pixel intensity difference by adding the overlaid pixel intensity difference absolute value of all pixels,
(h) calculating a total overlaid separate image pixel intensity by adding the adjusted overlaid separate image pixel intensities of all pixels,
(i) calculating a mismatch value by dividing the total overlaid pixel intensity difference by the total overlaid separate image pixel intensity, producing a mismatch value; and
program code for modifying at least one of the composite image and the separate image to reduce the mismatch value, thereby generating a final composite image.
US10/982,386 2004-11-05 2004-11-05 Method for generating a composite image Expired - Fee Related US7668402B2 (en)

Priority Applications (13)

Application Number Priority Date Filing Date Title
US10/982,386 US7668402B2 (en) 2004-11-05 2004-11-05 Method for generating a composite image
JP2007540307A JP4767258B2 (en) 2004-11-05 2005-09-27 How to generate composite images
DE602005027884T DE602005027884D1 (en) 2004-11-05 2005-09-27 PROCESS FOR GENERATING A COMPOSITE IMAGE
RU2007120940/07A RU2475850C2 (en) 2004-11-05 2005-09-27 Method of forming composite image, system for realising said method, method of calculating mismatch value and data medium
AU2005305293A AU2005305293B2 (en) 2004-11-05 2005-09-27 A method for generating a composite image
CN2005800429980A CN101080744B (en) 2004-11-05 2005-09-27 Method for generating a composite image
PCT/US2005/035075 WO2006052335A2 (en) 2004-11-05 2005-09-27 A method for generating a composite image
CA002586242A CA2586242A1 (en) 2004-11-05 2005-09-27 A method for generating a composite image
EP05802850A EP1807804B1 (en) 2004-11-05 2005-09-27 A method for generating a composite image
AT05802850T ATE508437T1 (en) 2004-11-05 2005-09-27 METHOD FOR GENERATING A COMPOSITE IMAGE
BRPI0517669-7A BRPI0517669A (en) 2004-11-05 2005-09-27 method to generate a composite image
TW094134888A TWI464705B (en) 2004-11-05 2005-10-06 A method for generating a composite image
IL182962A IL182962A (en) 2004-11-05 2007-05-03 Method for generating a composite image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US10/982,386 US7668402B2 (en) 2004-11-05 2004-11-05 Method for generating a composite image

Publications (2)

Publication Number Publication Date
US20060097139A1 US20060097139A1 (en) 2006-05-11
US7668402B2 true US7668402B2 (en) 2010-02-23

Family

ID=35811667

Family Applications (1)

Application Number Title Priority Date Filing Date
US10/982,386 Expired - Fee Related US7668402B2 (en) 2004-11-05 2004-11-05 Method for generating a composite image

Country Status (13)

Country Link
US (1) US7668402B2 (en)
EP (1) EP1807804B1 (en)
JP (1) JP4767258B2 (en)
CN (1) CN101080744B (en)
AT (1) ATE508437T1 (en)
AU (1) AU2005305293B2 (en)
BR (1) BRPI0517669A (en)
CA (1) CA2586242A1 (en)
DE (1) DE602005027884D1 (en)
IL (1) IL182962A (en)
RU (1) RU2475850C2 (en)
TW (1) TWI464705B (en)
WO (1) WO2006052335A2 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090167840A1 (en) * 2007-12-28 2009-07-02 Hon Hai Precision Industry Co., Ltd. Video instant messaging system and method thereof
US20090256909A1 (en) * 2008-04-11 2009-10-15 Nixon Stuart Systems and methods of capturing large area images in detail including cascaded cameras and/or calibration features
US20100013927A1 (en) * 2008-04-11 2010-01-21 Nearmap Pty Ltd. Systems and Methods of Capturing Large Area Images in Detail Including Cascaded Cameras and/or Calibration Features
US20140198985A1 (en) * 2013-01-17 2014-07-17 Realtek Semiconductor Corp. Image processing apparatus for performing color interpolation upon captured images and related method thereof
US8861890B2 (en) 2010-11-24 2014-10-14 Douglas Alan Lefler System and method for assembling and displaying individual images as a continuous image
US20150278997A1 (en) * 2012-09-26 2015-10-01 Korea Institute Of Science And Technology Method and apparatus for inferring facial composite

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102194241B (en) * 2011-04-28 2013-07-10 西安交通大学 Internet-based design method of artistic picture splicing system
RU2636906C2 (en) * 2015-02-06 2017-11-28 Общество С Ограниченной Ответственностью "Яндекс" System and method of setting up clusters of points of interest using grids
CN106780658B (en) * 2016-11-16 2021-03-09 北京旷视科技有限公司 Face feature adding method, device and equipment

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4034485A (en) * 1975-02-24 1977-07-12 Savo, Inc. Composite face apparatus and method
US5450504A (en) 1992-05-19 1995-09-12 Calia; James Method for finding a most likely matching of a target facial image in a data base of facial images
EP0704822A2 (en) 1994-09-30 1996-04-03 Istituto Trentino Di Cultura A method of storing and retrieving images of people, for example, in photographic archives and for the construction of identikit images
US6642918B2 (en) 2001-04-23 2003-11-04 Canon Kabushiki Kaisha Control of digital projection system
US6690830B1 (en) * 1998-04-29 2004-02-10 I.Q. Bio Metrix, Inc. Method and apparatus for encoding/decoding image data
US6707454B1 (en) 1999-07-01 2004-03-16 Lucent Technologies Inc. Systems and methods for visualizing multi-dimensional data in spreadsheets and other data structures
US6731302B1 (en) 1998-04-29 2004-05-04 Iq Biometrix, Inc. Method and apparatus for creating facial images
US20050231416A1 (en) * 2004-04-14 2005-10-20 Rowe Richard L Relational millimeter-wave interrogating
US7003140B2 (en) * 2003-11-13 2006-02-21 Iq Biometrix System and method of searching for image data in a storage medium

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4862081A (en) * 1988-11-23 1989-08-29 Picker International, Inc. DC artifact removal in magnetic resonance imaging
JP2979717B2 (en) * 1991-06-06 1999-11-15 ブラザー工業株式会社 Caricature making device
US5276478A (en) * 1992-05-19 1994-01-04 Eastman Kodak Company Method and apparatus for optimizing depth images by adjusting print spacing
CA2100324C (en) * 1992-08-06 2004-09-28 Christoph Eisenbarth Method and apparatus for determining mis-registration
JP3477750B2 (en) * 1993-08-02 2003-12-10 カシオ計算機株式会社 Face image creation device and face image creation method
JPH08223424A (en) * 1995-02-10 1996-08-30 Nikon Corp Negative/positive converter and film image converter using it
JP2930006B2 (en) * 1996-05-29 1999-08-03 日本電気株式会社 Caricature making device
US6526157B2 (en) * 1997-08-01 2003-02-25 Sony Corporation Image processing apparatus, image processing method and transmission medium
JP3821558B2 (en) * 1997-10-27 2006-09-13 日本電信電話株式会社 Video information simultaneous addition method and recording medium recording the method
AU1613599A (en) * 1997-12-01 1999-06-16 Arsev H. Eraslan Three-dimensional face identification system
DK1138011T3 (en) * 1998-12-02 2004-03-08 Univ Manchester Determination of facial subspace
US6785739B1 (en) * 2000-02-23 2004-08-31 Eastman Kodak Company Data storage and retrieval playback apparatus for a still image receiver
US6807290B2 (en) * 2000-03-09 2004-10-19 Microsoft Corporation Rapid computer modeling of faces for animation
AU2001267718A1 (en) * 2000-07-03 2002-01-14 Neville Anthony Wylie Samuelson A means for applying images to other images
JP2002360567A (en) * 2001-05-30 2002-12-17 Ge Medical Systems Global Technology Co Llc Method and apparatus for ultrasonic photographing
US7111256B2 (en) * 2002-06-05 2006-09-19 Kla-Tencor Technologies Corporation Use of overlay diagnostics for enhanced automatic process control

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4034485A (en) * 1975-02-24 1977-07-12 Savo, Inc. Composite face apparatus and method
US5450504A (en) 1992-05-19 1995-09-12 Calia; James Method for finding a most likely matching of a target facial image in a data base of facial images
EP0704822A2 (en) 1994-09-30 1996-04-03 Istituto Trentino Di Cultura A method of storing and retrieving images of people, for example, in photographic archives and for the construction of identikit images
US6690830B1 (en) * 1998-04-29 2004-02-10 I.Q. Bio Metrix, Inc. Method and apparatus for encoding/decoding image data
US6731302B1 (en) 1998-04-29 2004-05-04 Iq Biometrix, Inc. Method and apparatus for creating facial images
US7289647B2 (en) * 1999-05-28 2007-10-30 I. Q. Biometrix, Inc. System and method for creating and displaying a composite facial image
US6707454B1 (en) 1999-07-01 2004-03-16 Lucent Technologies Inc. Systems and methods for visualizing multi-dimensional data in spreadsheets and other data structures
US6642918B2 (en) 2001-04-23 2003-11-04 Canon Kabushiki Kaisha Control of digital projection system
US7003140B2 (en) * 2003-11-13 2006-02-21 Iq Biometrix System and method of searching for image data in a storage medium
US20050231416A1 (en) * 2004-04-14 2005-10-20 Rowe Richard L Relational millimeter-wave interrogating

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
Brunelli et al., "SpotIt! An Interactive Identikit System," CVGIP Graphical Models and Image Processing, vol. 58, No. 5, pp. 399-404, Sep. 1996.
Brunelli, Roberto and Poggio, Tomaso,Face Recognition: Features versus Templates, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. No. 10, Oct. 15, 1993, pp. 1042-1052.
Communication pursuant to Article 94(3) EPC, European Patent Office, Dec. 22, 2008, 6 pages.
Faces 4.0 for Law Enforcement, www.iqbiometrix.com/products-faces-40.html, pp. 1-4, Jul. 15, 2003.
Gillenson et al., "A Heuristic Strategy for Developing Human Facial Images on a CRT," Pattern Recognitiion UK, vol. 7, No. 4, pp. 187-196, Dec. 1975.
International Search Report, PCT/US2005/035075, Sep. 27, 2005, 5 pages.
Sakai et al., "Computer Analysis and Classification of Photographs of Human Faces," USA-Japan Computer Conference Proceedings, vol. Conf. 1, pp. 2701-2708, Oct. 3-5, 1972.

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090167840A1 (en) * 2007-12-28 2009-07-02 Hon Hai Precision Industry Co., Ltd. Video instant messaging system and method thereof
US8295313B2 (en) 2007-12-28 2012-10-23 Hon Hai Precision Industry Co., Ltd. Video instant messaging system and method thereof
US20090256909A1 (en) * 2008-04-11 2009-10-15 Nixon Stuart Systems and methods of capturing large area images in detail including cascaded cameras and/or calibration features
US20100013927A1 (en) * 2008-04-11 2010-01-21 Nearmap Pty Ltd. Systems and Methods of Capturing Large Area Images in Detail Including Cascaded Cameras and/or Calibration Features
US8497905B2 (en) 2008-04-11 2013-07-30 nearmap australia pty ltd. Systems and methods of capturing large area images in detail including cascaded cameras and/or calibration features
US8675068B2 (en) 2008-04-11 2014-03-18 Nearmap Australia Pty Ltd Systems and methods of capturing large area images in detail including cascaded cameras and/or calibration features
US10358235B2 (en) 2008-04-11 2019-07-23 Nearmap Australia Pty Ltd Method and system for creating a photomap using a dual-resolution camera system
US10358234B2 (en) 2008-04-11 2019-07-23 Nearmap Australia Pty Ltd Systems and methods of capturing large area images in detail including cascaded cameras and/or calibration features
US8861890B2 (en) 2010-11-24 2014-10-14 Douglas Alan Lefler System and method for assembling and displaying individual images as a continuous image
US20150278997A1 (en) * 2012-09-26 2015-10-01 Korea Institute Of Science And Technology Method and apparatus for inferring facial composite
US9691132B2 (en) * 2012-09-26 2017-06-27 Korea Institute Of Science And Technology Method and apparatus for inferring facial composite
US9008421B2 (en) * 2013-01-17 2015-04-14 Realtek Semiconductor Corp. Image processing apparatus for performing color interpolation upon captured images and related method thereof
US20140198985A1 (en) * 2013-01-17 2014-07-17 Realtek Semiconductor Corp. Image processing apparatus for performing color interpolation upon captured images and related method thereof

Also Published As

Publication number Publication date
ATE508437T1 (en) 2011-05-15
IL182962A (en) 2012-02-29
EP1807804B1 (en) 2011-05-04
TW200629175A (en) 2006-08-16
DE602005027884D1 (en) 2011-06-16
IL182962A0 (en) 2007-08-19
BRPI0517669A (en) 2008-10-14
US20060097139A1 (en) 2006-05-11
CA2586242A1 (en) 2006-05-18
RU2007120940A (en) 2008-12-10
AU2005305293A1 (en) 2006-05-18
JP4767258B2 (en) 2011-09-07
WO2006052335A3 (en) 2006-08-10
CN101080744B (en) 2013-05-22
WO2006052335A2 (en) 2006-05-18
CN101080744A (en) 2007-11-28
RU2475850C2 (en) 2013-02-20
EP1807804A2 (en) 2007-07-18
TWI464705B (en) 2014-12-11
JP2008519547A (en) 2008-06-05
AU2005305293B2 (en) 2010-10-21

Similar Documents

Publication Publication Date Title
AU2005305293B2 (en) A method for generating a composite image
CN112884637B (en) Special effect generation method, device, equipment and storage medium
CN107771336B (en) Feature detection and masking in images based on color distribution
US8538144B2 (en) Methods and systems for color correction of 3D images
US8884980B2 (en) System and method for changing hair color in digital images
CN114025219B (en) Rendering method, device, medium and equipment for augmented reality special effects
CN114096986A (en) Automatically segmenting and adjusting an image
JP3810943B2 (en) Image processing apparatus, image processing method, and recording medium recording image processing program
US9697642B2 (en) Image processing device and image processing method
CN113301409B (en) Video synthesis method and device, electronic equipment and readable storage medium
CN110827204A (en) Image processing method and device and electronic equipment
KR20240089729A (en) Image processing methods, devices, storage media and electronic devices
CN113610723A (en) Image processing method and related device
CN115147261A (en) Image processing method, device, storage medium, equipment and product
CN111275648B (en) Face image processing method, device, equipment and computer readable storage medium
AU2023233092A1 (en) Artificial intelligence techniques for extrapolating hdr panoramas from ldr low fov images
CN115272534B (en) Face image protection method, protection device, electronic equipment and readable storage medium
KR100422470B1 (en) Method and apparatus for replacing a model face of moving image
US11250632B2 (en) High quality AR cosmetics simulation via image filtering techniques
EP3734554B1 (en) Background image processing method and apparatus
JP6464977B2 (en) Projection control apparatus and program
CN118822864A (en) Image processing method and device, electronic equipment and storage medium
CN118573834A (en) Display control method of intelligent glasses, computer equipment and storage medium
CN113191940A (en) Image processing method, apparatus, device and medium
CN117274141A (en) Chrominance matting method and device and video live broadcast system

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTERGRAPH HARDWARE TECHNOLOGIES COMPANY, NEVADA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GRINDSTAFF, GENE A.;WHITAKER, SHEILA G.;REEL/FRAME:015817/0883

Effective date: 20041203

Owner name: INTERGRAPH HARDWARE TECHNOLOGIES COMPANY,NEVADA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GRINDSTAFF, GENE A.;WHITAKER, SHEILA G.;REEL/FRAME:015817/0883

Effective date: 20041203

AS Assignment

Owner name: INTERGRAPH SOFTWARE TECHNOLOGIES,NEVADA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:INTERGRAPH HARDWARE TECHNOLOGIES;REEL/FRAME:017706/0687

Effective date: 20010601

Owner name: INTERGRAPH SOFTWARE TECHNOLOGIES, NEVADA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:INTERGRAPH HARDWARE TECHNOLOGIES;REEL/FRAME:017706/0687

Effective date: 20010601

AS Assignment

Owner name: INTERGRAPH SOFTWARE TECHNOLOGIES COMPANY,NEVADA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:INTERGRAPH HARDWARE TECHNOLOGIES COMPANY;REEL/FRAME:018552/0858

Effective date: 20061122

Owner name: INTERGRAPH SOFTWARE TECHNOLOGIES COMPANY, NEVADA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:INTERGRAPH HARDWARE TECHNOLOGIES COMPANY;REEL/FRAME:018552/0858

Effective date: 20061122

AS Assignment

Owner name: MORGAN STANLEY & CO. INCORPORATED,NEW YORK

Free format text: FIRST LIEN INTELLECTUAL PROPERTY SECURITY AGREEMENT;ASSIGNORS:COBALT HOLDING COMPANY;INTERGRAPH CORPORATION;COBALT MERGER CORP.;AND OTHERS;REEL/FRAME:018731/0501

Effective date: 20061129

Owner name: MORGAN STANLEY & CO. INCORPORATED, NEW YORK

Free format text: FIRST LIEN INTELLECTUAL PROPERTY SECURITY AGREEMENT;ASSIGNORS:COBALT HOLDING COMPANY;INTERGRAPH CORPORATION;COBALT MERGER CORP.;AND OTHERS;REEL/FRAME:018731/0501

Effective date: 20061129

AS Assignment

Owner name: MORGAN STANLEY & CO. INCORPORATED,NEW YORK

Free format text: SECOND LIEN INTELLECTUAL PROPERTY SECURITY AGREEMENT;ASSIGNORS:COBALT HOLDING COMPANY;INTERGRAPH CORPORATION;COBALT MERGER CORP.;AND OTHERS;REEL/FRAME:018746/0234

Effective date: 20061129

Owner name: MORGAN STANLEY & CO. INCORPORATED, NEW YORK

Free format text: SECOND LIEN INTELLECTUAL PROPERTY SECURITY AGREEMENT;ASSIGNORS:COBALT HOLDING COMPANY;INTERGRAPH CORPORATION;COBALT MERGER CORP.;AND OTHERS;REEL/FRAME:018746/0234

Effective date: 20061129

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: INTERGRAPH CORPORATION, ALABAMA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH DISC, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: INTERGRAPH EUROPEAN MANUFACTURING, LLC, NETHERLAND

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: COADE INTERMEDIATE HOLDINGS, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH TECHNOLOGIES COMPANY, NEVADA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH (ITALIA), LLC, ITALY

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: M&S COMPUTING INVESTMENTS, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: INTERGRAPH CHINA, INC., CHINA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: INTERGRAPH HOLDING COMPANY (F/K/A COBALT HOLDING C

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: Z/I IMAGING CORPORATION, ALABAMA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH HOLDING COMPANY (F/K/A COBALT HOLDING C

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: INTERGRAPH DC CORPORATION - SUBSIDIARY 3, ALABAMA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: INTERGRAPH PP&M US HOLDING, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: Z/I IMAGING CORPORATION, ALABAMA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: COADE HOLDINGS, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: INTERGRAPH CHINA, INC., CHINA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH PP&M US HOLDING, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: ENGINEERING PHYSICS SOFTWARE, INC., TEXAS

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: M&S COMPUTING INVESTMENTS, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH (ITALIA), LLC, ITALY

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH ASIA PACIFIC, INC., AUSTRALIA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH CORPORATION, ALABAMA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: ENGINEERING PHYSICS SOFTWARE, INC., TEXAS

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH DISC, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH TECHNOLOGIES COMPANY, NEVADA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: WORLDWIDE SERVICES, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH SERVICES COMPANY, ALABAMA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: INTERGRAPH EUROPEAN MANUFACTURING, LLC, NETHERLAND

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: COADE INTERMEDIATE HOLDINGS, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: INTERGRAPH DC CORPORATION - SUBSIDIARY 3, ALABAMA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: INTERGRAPH SERVICES COMPANY, ALABAMA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: COADE HOLDINGS, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF FIRST LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. INCORPORATED;REEL/FRAME:025892/0299

Effective date: 20101028

Owner name: WORLDWIDE SERVICES, INC., ALABAMA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

Owner name: INTERGRAPH ASIA PACIFIC, INC., AUSTRALIA

Free format text: TERMINATION AND RELEASE OF SECOND LIEN INTELLECTUAL PROPERTY SECURITY INTEREST;ASSIGNOR:WACHOVIA BANK, NATIONAL ASSOCIATION;REEL/FRAME:025892/0028

Effective date: 20101028

FPAY Fee payment

Year of fee payment: 4

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.)

LAPS Lapse for failure to pay maintenance fees

Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.)

STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20180223