US8363910B2 - Image processing device, image processing method, and program - Google Patents

Image processing device, image processing method, and program Download PDF

Info

Publication number
US8363910B2
US8363910B2 US12/074,667 US7466708A US8363910B2 US 8363910 B2 US8363910 B2 US 8363910B2 US 7466708 A US7466708 A US 7466708A US 8363910 B2 US8363910 B2 US 8363910B2
Authority
US
United States
Prior art keywords
image
selection
eligible
positivity
facial
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US12/074,667
Other languages
English (en)
Other versions
US20080273798A1 (en
Inventor
Masamichi Asukai
Masaaki Tsuruta
Nozomu Ozaki
Taiji Ito
Akinobu Sugino
Hidehiko Sekizawa
Akane Sano
Yoichiro Sako
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Assigned to SONY CORPORATION reassignment SONY CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: OZAKI, NOZOMU, SAKO, YOICHIRO, SUGINO, AKINOBU, ASUKAI, MASAMICHI, SANO, AKANE, ITO, TAIJI, SEKIZAWA, HIDEHIKO, TSURUTA, MASAAKI
Publication of US20080273798A1 publication Critical patent/US20080273798A1/en
Application granted granted Critical
Publication of US8363910B2 publication Critical patent/US8363910B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5838—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof

Definitions

  • the present invention contains subject matter related to Japanese Patent Application JP 2007-058704 filed in the Japan Patent Office on Mar. 8, 2007, the entire contents of which being incorporated herein by reference.
  • the present invention relates to an image processing device, an image processing method, and a program.
  • the users who use these devices can immerse themselves in memories of scenes from the times when the images were acquired.
  • the users can discover among the acquired still images and moving images specific images in which a positive feeling toward the person who acquired the image of an imaged subject can be felt, for example, images in which the subject is smiling at the device, images in which the subject's line of vision is directed toward the device, and the like. The users can then obtain those images.
  • JP-A-2000-132693 Technology that is concerned with detecting the subject's face has been disclosed in Japanese Patent Application Publication No. JP-A-2000-132693, for example.
  • the present invention addresses the problem described above to provide an image processing device, an image processing method, and a program that are new and improved and that are capable of selecting an image of each imaged subject from at least one image, based on a relationship between the imaged subject and the person who acquired the image.
  • an image processing device that includes a facial image extraction portion, a positivity computation portion, and a selected image update portion.
  • the facial image extraction portion specifies a facial region of an imaged subject within at least one sequentially input selection-eligible image and extracts from the selection-eligible image a facial image that corresponds to the facial region.
  • the positivity computation portion computes for the selection-eligible image, using the facial image, a positivity of the imaged subject toward a person who acquired the selection-eligible image.
  • the selected image update portion compares the selection-eligible image for which the positivity was computed in the positivity computation portion to a selection candidate image that has the greatest positivity among all the selection-eligible images for the same imaged subject for which the positivities have already been computed, and makes the image with the greater positivity the new selection candidate image.
  • the image processing device includes the facial image extraction portion, the positivity computation portion, and the selected image update portion.
  • the facial image extraction portion can extract the facial image that corresponds to the facial region of the selection-eligible imaged subject.
  • the positivity computation portion can compute the positivity for the selection-eligible image, the positivity being a value that quantifies the relationship between the imaged subject in the selection-eligible image and the person who acquired the selection-eligible image.
  • the selected image update portion can compare the selection-eligible image for which the positivity was computed in the positivity computation portion to the selection candidate image, which is the image that has the greatest positivity among all the selection-eligible images for the same imaged subject for which the positivities have already been computed.
  • the selected image update portion makes the image with the greater positivity the new selection candidate image. This configuration makes it possible to select, from the at least one selection-eligible image, the image that has the greatest positivity for the imaged subject, based on the positivity of the relationship between the imaged subject and the person who acquired the image.
  • the image processing device may also include an image selection portion that selects, as a selected image for the imaged subject, the selection candidate image for the imaged subject that was updated in the selected image update portion.
  • This configuration makes it possible to select, from the at least one selection-eligible image, the image that has the greatest positivity for the imaged subject, based on the positivity of the relationship between the imaged subject and the person who acquired the image.
  • the image processing device may also compute the positivity based on at least one of a size of the facial image, a facial expression of the imaged subject, a face orientation of the imaged subject, a line-of-vision orientation of the imaged subject, a detection of eyes and a mouth, a color saturation of the facial image, and a spatial frequency of the facial image.
  • This configuration makes it possible to compute quantitatively the positivity of the relationship between the imaged subject and the person who acquired the image.
  • a corresponding sound may be input for the sequentially input selection-eligible image, and the image processing device may compute the positivity based also on the intonation of the sound and the volume of the sound.
  • This configuration makes it possible to compute quantitatively the positivity of the relationship between the imaged subject and the person who acquired the image.
  • the image processing device may also include an eligibility determination portion that determines, for the sequentially input selection-eligible image, whether or not the imaged subject is eligible for a selected image selection.
  • This configuration makes it possible to control whether or not an image is selected for the imaged subject.
  • the eligibility determination portion may also make the eligibility determination using selection-ineligible information that specifies an imaged subject that is not eligible for the selected image selection.
  • This configuration makes it possible to control whether or not an image is selected for the imaged subject.
  • the image processing device may also include a first positivity editing portion that can increase and decrease the positivity that the positivity computation portion computed for the selection-eligible image.
  • This configuration makes it possible for a user to increase and decrease the positivity, so the image selection can be performed more flexibly.
  • the image processing device may also include a second positivity editing portion that can increase and decrease the positivity that was computed for the selection candidate image for the imaged subject.
  • This configuration makes it possible for the user to increase and decrease the positivity, so the image selection can be performed more flexibly.
  • the image processing device may also include an image acquisition portion that acquires an image, and the at least one selection-eligible image may be an image that is acquired and input in real time by the image acquisition portion.
  • the image processing device may also include a storage portion that stores an image, and the at least one selection-eligible image may be an image that is stored in the storage portion.
  • the at least one selection-eligible image may be an image that is input from an external device.
  • an image processing method includes a step of specifying the facial region of the imaged subject within the at least one sequentially input selection-eligible image and extracting from the selection-eligible image the facial image that corresponds to the facial region.
  • the image processing method also includes a step of computing for the selection-eligible image, using the facial image, the positivity of the imaged subject toward the person who acquired the selection-eligible image.
  • the image processing method also includes a step of comparing the selection-eligible image for which the positivity was computed in the computing step to the selection candidate image that has the greatest positivity among all the selection-eligible images for the same imaged subject for which the positivities have already been computed, and making the image with the greater positivity the new selection candidate image.
  • this method makes it possible to select, from the at least one selection-eligible image, the image that has the greatest positivity for the imaged subject, based on the positivity of the relationship between the imaged subject and the person who acquired the image.
  • the image processing method may also include a step of selecting, as a selected image for the imaged subject, the selection candidate image for the imaged subject that is updated when the image with the greater positivity is made the new selection candidate image.
  • this method makes it possible to select, from the at least one selection-eligible image, the image that has the greatest positivity for the imaged subject, based on the positivity of the relationship between the imaged subject and the person who acquired the image.
  • a program that causes a computer to function as an extracting portion, a computing portion, and a comparing-updating portion.
  • the extracting portion specifies the facial region of the imaged subject within the at least one sequentially input selection-eligible image and extracts from the selection-eligible image the facial image that corresponds to the facial region.
  • the computing portion computes for the selection-eligible image, using the facial image, the positivity of the imaged subject toward the person who acquired the selection-eligible image.
  • the comparison-update portion compares the selection-eligible image for which the positivity was computed in the computing portion to a selection candidate image that has the greatest positivity among all the selection-eligible images for the same imaged subject for which the positivities have already been computed, and makes the image with the greater positivity the new selection candidate image.
  • This program makes it possible to select, from the at least one selection-eligible image, the image that has the greatest positivity for the imaged subject, based on the positivity of the relationship between the imaged subject and the person who acquired the image.
  • the program may also cause the computer to function as a selection portion that selects, as the selected image for the imaged subject, the selection candidate image for the imaged subject that was updated in the comparing-updating portion that makes the image with the greater positivity the new selection candidate image.
  • This program makes it possible to select, from the at least one selection-eligible image, the image that has the greatest positivity for the imaged subject, based on the positivity of the relationship between the imaged subject and the person who acquired the image.
  • FIG. 1 is an explanatory figure that shows a first example of an image processing device according to embodiments of the present invention
  • FIG. 2 is an explanatory figure that shows a second example of the image processing device according to the embodiments of the present invention
  • FIG. 3 is a block diagram that shows an image processing device according to a first embodiment of the present invention
  • FIG. 4 is a block diagram for explaining an image selection portion according to the embodiments of the present invention.
  • FIG. 5 is an explanatory figure for explaining an extraction of a facial image according to the embodiments of the present invention.
  • FIG. 6 is an explanatory figure for explaining a determination of whether or not an imaged subject is eligible for a selection of an image according to the embodiments of the present invention
  • FIG. 7 is an explanatory figure for explaining an example of a positivity computation portion according to the embodiments of the present invention.
  • FIG. 8 is an explanatory figure for explaining a first update portion for a selection candidate image according to the embodiments of the present invention.
  • FIG. 9 is an explanatory figure for explaining a second update portion for the selection candidate image according to the embodiments of the present invention.
  • FIG. 10 is an explanatory figure for explaining a third update portion for the selection candidate image according to the embodiments of the present invention.
  • FIG. 11 is an explanatory figure that shows an example of a selected image list according to the embodiments of the present invention.
  • FIG. 12 is a flowchart that shows a flow of an image processing method according to the embodiments of the present invention.
  • FIG. 13 is a block diagram that shows an example of a system that uses an image processing device according to a second embodiment of the present invention.
  • FIG. 14 is an explanatory figure that shows an example of a positivity ranking according to the embodiments of the present invention.
  • FIGS. 1 and 2 are explanatory figures that show the examples of the image processing device according to the embodiments of the present invention.
  • the image processing device may be a device that has an image acquisition function that is capable of acquiring still images and moving images.
  • the image processing device may be a device that is attached to a user's body, such as a wearable camera or the like, as shown in FIG. 1 (such that images can be acquired from a position that is close to the user's line of vision, for example).
  • the image processing device may also be a digital video camera like that shown in FIG. 2 , a digital camera (not shown in the drawings), a mobile telephone with a digital camera function (not shown in the drawings), or the like.
  • the image processing device may also be a device that has an image acquisition function that is capable of recording sound as well as acquiring still images and moving images, and may also be a device that has an audio recording function.
  • the image processing device is not limited to a device that has an image acquisition function and an audio recording function, like those shown in FIGS. 1 and 2 .
  • the image processing device can also be used in an information processing device that may have neither an image acquisition function nor an audio recording function, such as a personal computer (PC), a personal digital assistant (PDA), or the like, for example.
  • PC personal computer
  • PDA personal digital assistant
  • FIG. 3 is a block diagram that shows an image processing device 100 according to a first embodiment of the present invention.
  • the image processing device 100 includes an image processing portion 102 , an image acquisition portion 104 , a storage portion 106 , an operation portion 108 , and a display portion 110 .
  • the image processing device 100 may also include a control portion (not shown in the drawings) that is configured from a micro-processing unit (MPU) or the like and that controls the entire image processing device 100 .
  • the image processing device 100 may also include a communication portion (not shown in the drawings) that can receive images that are transmitted from an external device that is separate from the image processing device 100 .
  • the image processing portion 102 can process an image of an imaged subject from at least one image that is input sequentially to the image processing portion 102 (hereinafter called the “selection-eligible image”), as well as update and select the image.
  • the updating and the selection of the image in the image processing portion 102 are performed using a “positivity”, which is a value that quantifies a relationship between the imaged subject and the person who acquired the image.
  • positivity is a value that quantifies a relationship between the imaged subject and the person who acquired the image.
  • the selection-eligible image may be an image that is acquired by the image acquisition portion 104 (described below) and may be an image that is stored in the storage portion 106 (described below).
  • the selection-eligible image may also be an image that is transmitted from an external device that is separate from the image processing device 100 .
  • the selection-eligible image is not limited to being a still image and may also be a moving image (for example, a moving image can be handled frame-by-frame as a set of still images).
  • the still images may be images that are recorded in a still image format, for example, the Joint Photographic Experts Group (JPEG) format, a bitmap, or the like.
  • the moving images may be images that are recorded in a moving image format, for example, Windows Media Video (WMV), the H.264/Moving Picture Experts Group phase-4 Advanced Video Coding (H.264/MPEG-4 AVC) format, or the like.
  • WMV Windows Media Video
  • H.264/Moving Picture Experts Group phase-4 Advanced Video Coding H.264/MPEG-4 AVC
  • the image acquisition portion 104 is an image acquisition portion that has, for example, a lens and an image acquisition element such as a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like.
  • the image acquisition portion 104 can create the selection-eligible image by performing image acquisition.
  • the storage portion 106 is a storage portion that is included in the image processing device 100 and can store at least the selection-eligible image as an acquired image.
  • the storage portion 106 may be, for example, a magnetic storage medium such as a hard disk, a magnetic tape, or the like, a non-volatile memory such as a flash memory, a magnetoresistive random access memory (MRAM), a ferroelectric random access memory (FeRAM), a phase change random access memory (PRAM), or the like, a magneto-optical disk, or the like.
  • MRAM magnetoresistive random access memory
  • FeRAM ferroelectric random access memory
  • PRAM phase change random access memory
  • the storage portion 106 is obviously not limited to the examples described above.
  • the operation portion 108 is a device that a user who uses the image processing device 100 can operate. By performing specified operations with the operation portion 108 , the user can convey various commands to various portions of the image processing device 100 , for example, commands related to the updating and the selection of the image by the image processing portion 102 , image acquisition commands to the image acquisition portion 104 , and the like.
  • the operation portion 108 may be, for example, a button, a direction key, or a rotary selector such as a jog dial or the like, or any combination of these, but the operation portion 108 is not limited to these examples.
  • the display portion 110 is a display portion that is included in the image processing device 100 and can display the image that is updated and selected in the image processing portion 102 .
  • the display portion 110 may be a liquid crystal display (LCD), an organic light emitting diode display (OLED; also called an organic electroluminescence display), a field emission display (FED), or the like, but the display portion 110 is not limited to these examples.
  • the image processing device 100 configured as described above can select an image of a given imaged subject from at least one selection-eligible image, based on the positivity that is the value that quantifies the relationship between the imaged subject and the person who acquired the image.
  • An image selection portion according to the embodiments of the present invention will be explained below in order to show in greater detail the updating and the selection of the image in the image processing device 100 .
  • FIG. 4 is a block diagram for explaining the image selection portion according to the embodiments of the present invention.
  • the image processing portion 102 includes a facial image extraction portion 112 , an eligibility determination portion 114 , a positivity computation portion 116 , a selected image update portion 118 , and a list creation portion 120 .
  • the facial image extraction portion 112 specifies a region (hereinafter called the “facial region”) within the sequentially input selection-eligible image that contains the face portion of the imaged subject, then extracts (creates) a selection-eligible facial image that corresponds to the facial region of the imaged subject.
  • FIG. 5 is an explanatory figure for explaining the extraction of the facial image according to the embodiments of the present invention.
  • FIG. 5A is a figure that shows an example of a selection-eligible image according to the embodiments of the present invention
  • FIG. 5B is a figure that shows that a facial image X is extracted from the selection-eligible image shown in FIG. 5A . Note that the facial image X shown in FIG.
  • FIG. 5B does not show the background that is shown in the selection-eligible image shown in FIG. 5A , but the background has been omitted for the sake of convenience and may also be included in the facial image X. (The background is also omitted from the image in the same manner in the drawings shown below.)
  • the facial image extraction portion 112 specifies, for example, the rectangular facial region within the selection-eligible image that contains the face portion of the imaged subject, as shown in FIG. 5B .
  • the specification of the facial region of the imaged subject can be done, for example, by detecting a region that resembles a brightness distribution and a structural pattern of a face.
  • data related to the specification of the facial region such as the data for the brightness distribution and the structural pattern of the face, can be stored in the storage portion 106 , for example.
  • the image processing device 100 can be provided with the communication portion (not shown in the drawings) such that the data related to the specification of the facial region can be obtained from an external device.
  • the method of obtaining the data is not limited to these examples.
  • the specification of the facial region of the imaged subject can be done, for example, by using a difference between the current frame and the preceding frame to detect a moving body, then specifying a head portion of the detected moving body.
  • the size of the rectangular facial region can be determined, for example, by using a table in which the size of the face portion and the size of the rectangular facial region are correlated (the table being stored in the storage portion 106 , for example).
  • the size of the rectangular facial region can also be determined by defining the outer boundaries of the facial region to be a set distance from the face portion.
  • the size of the rectangular facial region can also be specified through the operation portion 108 (not shown in FIG. 4 ).
  • the facial image extraction portion 112 extracts the facial image X by, for example, performing a cutting-out (trimming) process that cuts out the facial region.
  • the facial image extraction portion 112 may also perform a distortion correction process.
  • the distortion correction process may, for example, correct the facial image that was extracted by the cutting-out process in to a rectangular image by expanding, in a horizontal direction and/or a vertical direction, the facial image that was extracted by the cutting-out process, thus equalizing the facial image with respect to the horizontal direction and the vertical direction according to the degree of expansion.
  • the distortion correction process is not limited to this example, and it is obvious that any method can be used that is capable of correcting distortion in the image.
  • the facial image extraction portion 112 can also append index information, for example, to the extracted facial image, the index information correlating the facial image to the selection-eligible image from which the facial image was extracted.
  • the index information can be appended to the facial image by tagging, digital watermarking, or the like, but the method of appending the index information is not limited to these examples. Appending the index information to the facial image makes it possible to correlate the facial image uniquely to the selection-eligible image from which the facial image was extracted.
  • the eligibility determination portion 114 can use the facial image that was extracted by the facial image extraction portion 112 to determine whether or not the imaged subject is eligible for image selection.
  • FIG. 6 is an explanatory figure for explaining the determination of whether or not the imaged subject is eligible for image selection according to the embodiments of the present invention.
  • the eligibility determination portion 114 determines whether the facial image X extracted by the facial image extraction portion 112 is eligible for selection by, for example, comparing the facial image X to each of selection-eligible images A to C that are included in selection-eligible information 122 for specifying images that are eligible for selection.
  • the determination of whether the facial image X is eligible for image selection can be performed by, for example, a face recognition process or the like in which data on feature points is extracted from the facial image and compared to feature point data from the pre-registered selection-eligible images A to C.
  • the determination method is not limited to this example.
  • the imaged subject shown in the facial image X is included in the selection-eligible information 122 as the selection-eligible image C, so the eligibility determination portion 114 can determine that the imaged subject shown in the facial image X is selection-eligible.
  • a determination result as to whether or not the imaged subject is eligible for image selection can be expressed by one bit of data (that is, the determination result is expressed by a value of “0” or “1”), for example, but the determination result is not limited to one bit.
  • the eligibility determination portion 114 determines whether the facial image X extracted by the facial image extraction portion 112 is eligible for selection by, for example, comparing the facial image X to each of selection-ineligible images D and E that are included in selection-ineligible information 124 for specifying images that are not eligible for selection. In this case, the determination of whether the facial image X is eligible for image selection can be performed by using face recognition technology, for example, in the same manner as in the first determination portion described above.
  • the imaged subject shown in the facial image X is not included in the selection-ineligible information 124 , so the eligibility determination portion 114 can determine that the imaged subject shown in the facial image X is selection-eligible.
  • the determination result as to whether or not the imaged subject is eligible for image selection can be expressed by one bit of data, for example, in the same manner as in the first determination portion described above.
  • the eligibility determination portion 114 can also determine whether the facial image X extracted by the facial image extraction portion 112 is eligible for selection by using the selection-eligible information 122 for specifying images that are eligible for selection and the selection-ineligible information 124 for specifying images that are not eligible for selection.
  • the imaged subject shown in the facial image X is included in the selection-eligible information 122 and is not included in the selection-ineligible information 124 : The same pattern as in the first determination portion.
  • the imaged subject shown in the facial image X is not included in the selection-eligible information 122 and is included in the selection-ineligible information 124 : The same pattern as in the second determination portion.
  • the imaged subject shown in the facial image X is included in the selection-eligible information 122 and is included in the selection-ineligible information 124 .
  • the imaged subject shown in the facial image X is not included in the selection-eligible information 122 and is not included in the selection-ineligible information 124 .
  • the third determination portion it is possible for the third determination portion to change the determination of whether or not the imaged subject shown in the facial image X is eligible for image selection depending on how the determination result is set. A more flexible determination can thus be made.
  • the eligibility determination portion 114 can determine whether the facial image X is eligible for image selection by using the selection-eligible information 122 and/or the selection-ineligible information 124 , as shown in the first to third determination portions, for example.
  • the information that is set in the selection-eligible information 122 and the selection-ineligible information 124 (for example, the selection-eligible images, the selection-ineligible images, and the like) can be stored in the storage portion 106 , for example.
  • the information that is set in the selection-eligible information 122 and the selection-ineligible information 124 can also be registered by the user using the operation portion 108 , for example, and can also be transmitted from an external device.
  • the setting of the information that is set in the selection-eligible information 122 and the selection-ineligible information 124 is not limited to these examples.
  • the positivity computation portion 116 uses the facial image that was extracted by the facial image extraction portion 112 to compute the positivity that is the value that quantifies the relationship between the imaged subject in the selection-eligible image and the person who acquired the selection-eligible image.
  • the determination result determined in the eligibility determination portion 114 is also input to the positivity computation portion 116 .
  • the positivity computation portion 116 can be set such that it does not compute the positivity in a case where it is determined in the eligibility determination portion 114 that the imaged subject shown in the facial image is not eligible for image selection.
  • a positivity computation portion according to the embodiments of the present invention will be explained below with reference to FIG. 7 .
  • FIG. 7 is an explanatory figure for explaining an example of the positivity computation portion according to the embodiments of the present invention.
  • the positivity computation portion 116 can compute the positivity using Equation 1, for example.
  • Positivity (Standard value)+(Variable value derived from the facial image) Equation 1
  • the standard value in Equation 1 is a value that serves as a standard for the positivity. It may be set to any value, for example, ⁇ 100, 0, 50.5, 100, 200, or the like.
  • the variable value derived from the facial image in Equation 1 can, for example, be set to a size of the facial image, a facial expression of the imaged subject, a face orientation of the imaged subject, a line-of-vision orientation of the imaged subject, a detection of eyes and a mouth, a color saturation of the facial image, a spatial frequency of the facial image, or the like.
  • the variable value derived from the facial image in Equation 1 is not limited to these examples.
  • FIG. 7 shows, as one example, an example in which the variable value derived from the facial image in Equation 1 is computed using a table that contains values for the three criteria of the facial expression of the imaged subject, the face orientation of the imaged subject, and the line-of-vision orientation of the imaged subject.
  • the positivity computation portion 116 can, for example, detect the positions of the eyes, the nose, and both ends of the lips (the feature points). Based on the detected positions of the eyes, the nose, and both ends of the lips, the positivity computation portion 116 can determine the facial expression, the face orientation, and the line-of-vision orientation of the imaged subject. Note that a determination portion for determining the facial expression, the face orientation, and the line-of-vision orientation according to the embodiments of the present invention is obviously not limited to the example described above.
  • information for determining the facial expression, the face orientation, the line-of-vision orientation, and the like of the imaged subject can be stored in the storage portion 106 , for example.
  • the positivity computation portion 116 may also include a storage portion, and the information for determining the facial expression, the face orientation, the line-of-vision orientation, and the like of the imaged subject in the positivity computation portion 116 may also be stored in the storage portion.
  • the storage portion that is included in the positivity computation portion 116 may be, for example, a magnetic storage medium such as a hard disk or the like, or a non-volatile memory such as a flash memory or the like, but is not limited to these examples.
  • the facial expression of the imaged subject is a smiling expression
  • the face orientation is toward the front
  • the positivity computation portion 116 can compute the positivity using Equation 1, for example, as shown above.
  • the positivity is computed using the three criteria of the facial expression, the face orientation, and the line-of-vision orientation of the imaged subject.
  • the value derived for the selection-eligible image can set the positivity over a wider range of values if the number of criteria is increased.
  • the positivity computation portion In the first example of the positivity computation portion, an example was shown in which the positivity is computed using the variable value derived from the facial image, but the positivity computation portion according to the embodiments of the present invention is not limited to the example described above.
  • the positivity computation portion 116 can compute the positivity using the sound that corresponds to the selection-eligible image that is input to the image processing device 100 .
  • the positivity computation portion 116 can compute the positivity using Equation 2, for example.
  • Positivity (Standard value)+(Variable value derived from the facial image)+(Variable value derived from the sound) Equation 2
  • Equation 2 the standard value and the variable value derived from the facial image are the same as in Equation 1 above (the first example).
  • the variable value derived from the sound in Equation 2 can be set based on the intonation of the sound, the volume of the sound, or the like, but is not limited to these examples.
  • the setting of the variable value derived from the sound in the positivity computation portion 116 can be done, for example, by having the positivity computation portion 116 refer to a table in which the intonation of the sound, the volume of the sound, or the like, is uniquely correlated to the variable value.
  • the information that is used to set the variable value derived from the sound can be stored in the storage portion 106 , for example.
  • the positivity computation portion 116 may also include a storage portion, and the information that is used to set the variable value derived from the sound may also be stored in the storage portion.
  • the eligibility determination portion 114 can determine whether or not the image is eligible for selection by using sound pattern information that is set in the selection-eligible information 122 and the selection-ineligible information 124 . Therefore, even in a case where the sound is used to compute the positivity, the positivity computation portion 116 can be set such that it does not compute the positivity in a case where the eligibility determination portion 114 determines that the imaged subject shown in the facial image is not eligible for selection.
  • the positivity computation portion In the first example of the positivity computation portion, the variable value derived from the facial image was used to compute the positivity. In the second example of the positivity computation portion, the variable value derived from the sound was used to compute the positivity.
  • the positivity computation portion according to the embodiments of the present invention is not limited to these examples.
  • the positivity can be computed by using the information that is appended to the selection-eligible image, as shown in Equation 3.
  • the information can be appended to the selection-eligible image by tagging, digital watermarking, or the like, for example.
  • Positivity (Standard value)+(Variable value derived from the facial image)+(Variable value derived from the sound)+(Variable value derived from the information that is appended to the selection-eligible image) Equation 3
  • Equation 3 the standard value, the variable value derived from the facial image, and the variable value derived from the sound are the same as in Equation 2 above (the second example).
  • the variable value derived from the information that is appended to the selection-eligible image in Equation 3 can be set according to whether or not the information that is appended to the selection-eligible image matches a selection condition, for example, a time, a location, or the like, that the user transmits to the positivity computation portion 116 by using the operation portion 108 .
  • the information is appended to the selection-eligible image, but the information is obviously not limited to information that indicates the date and time of image acquisition and information that indicates the location of image acquisition.
  • the positivity computation portion 116 can use Equations 1 to 3 to compute the positivity. Note that the positivity computation portion according to the embodiments of the present invention is not limited to Equations 1 to 3. The positivity may also be computed based on the variable value derived from the facial image and the variable value derived from the information that is appended to the selection-eligible image.
  • the user can also increase and decrease the value of the positivity computed by the positivity computation portion 116 by operating the operation portion 108 as a first positivity editing portion (not shown in FIG. 4 ).
  • the user can adjust the value of the positivity even for a selection-eligible image for which the positivity computation portion 116 computed a low positivity value.
  • the image processing device 100 can thus treat any image that the user desires as a selected image.
  • the image processing device 100 can have a checking portion (not shown in the drawings) that checks the selection-eligible image for which the positivity computation portion 116 computed the positivity, so that the user can use the operation portion 108 to adjust the positivity value.
  • the checking portion may be a display portion for checking that uses an LCD, an OLED, or the like, but the checking portion is not limited to these examples.
  • the selected image update portion 118 uses the positivity that the positivity computation portion 116 computed for the facial image to update an image for each imaged subject (hereinafter called the “selection candidate image”) that will become a candidate to be the image that is selected for the imaged subject (hereinafter called the “selected image”).
  • the selection candidate image that the selected image update portion 118 updates can be stored in a storage portion that is provided in the selected image update portion 118 , for example, but this example is not limiting, and the selection candidate image may also be stored in the storage portion 106 .
  • the storage portion that is provided in the selected image update portion 118 may be, for example, a magnetic storage medium such as a hard disk or the like, or a non-volatile memory such as a flash memory or the like, but is not limited to these examples.
  • Update portions for the selection candidate image according to the embodiments of the present invention will be explained below with reference to FIGS. 8 to 10 .
  • FIG. 8 is an explanatory figure for explaining a first update portion for the selection candidate image according to the embodiments of the present invention.
  • FIG. 8A is a figure that shows a pre-update state
  • FIG. 8B is a figure that shows a post-update state.
  • the first update portion is an update portion that is used in a case where the positivity has not been computed for the imaged subject shown in the facial image X (the selection-eligible image) prior to the extraction of the facial image X, that is, a case where the positivity is computed for the first time for the imaged subject shown in the facial image X.
  • a determination of whether or not the positivity has already been computed for the imaged subject shown in the facial image X can be made, for example, by performing face recognition that compares the selection candidate image for each imaged subject that the selected image update portion 118 has updated to the facial image X (the selection-eligible image) for which the positivity computation portion 116 has computed the positivity.
  • the selection candidate image updated by the selected image update portion 118 is stored as the selection-eligible information 122 shown in FIG. 6
  • the eligibility determination portion 114 determines whether or not the positivity has already been computed for the imaged subject shown in the facial image X.
  • the selected image update portion 118 can then use the result of the determination by the eligibility determination portion 114 to determine whether or not the positivity has already been computed for the imaged subject shown in the facial image X.
  • the selected image update portion 118 determines that this is the first time that the positivity has been computed for the imaged subject shown in the facial image X
  • the selected image update portion 118 performs an update such that the facial image X (the selection-eligible image) becomes the facial image for the new selection candidate image, as shown in FIG. 8B .
  • the index information that correlates the facial image X to the selection-eligible image from which the facial image X was extracted can be appended to the facial image X. Therefore, by referring to the index information that is appended to the facial image X, the selected image update portion 118 can perform an update such that the selection-eligible image from which the facial image X was extracted becomes the selection candidate image.
  • the selection candidate image that the selected image update portion 118 updates is obviously not limited to the selection-eligible image from which the facial image X was extracted, and the facial image X can also be used for the selection candidate image.
  • the facial image X is used for the selection candidate image for the sake of convenience.
  • FIG. 9 is an explanatory figure for explaining the second update portion for the selection candidate image according to the embodiments of the present invention.
  • FIG. 9A is a figure that shows a pre-update state
  • FIG. 9B is a figure that shows a post-update state.
  • the second update portion is an update portion that is used in a case where a facial image Xold (the selection candidate image) exists for which the positivity has already been computed for the imaged subject shown in the facial image X (the selection-eligible image).
  • the selected image update portion 118 determines that the positivity has already been computed for the imaged subject shown in the facial image X
  • the selected image update portion 118 compares the positivity 125 that was computed for the facial image X to the positivity 115 that was set for the selection candidate image (the facial image Xold), for which the positivity has already been computed.
  • the selected image update portion 118 can perform an update such that the image with the greater positivity value becomes the new selection candidate image. Therefore, as shown in FIG. 9B , the selected image update portion 118 performs an update such that the facial image X (the selection-eligible image), for which the positivity value is greater than that for the selection candidate image, becomes the new selection candidate image.
  • FIG. 10 is an explanatory figure for explaining the third update portion for the selection candidate image according to the embodiments of the present invention.
  • FIG. 10A is a figure that shows a pre-update state
  • FIG. 10B is a figure that shows a post-update state.
  • the third update portion is an update portion that is used in a case where, when the facial image X has been made the selection candidate image, the positivity is computed for a facial image Xnew (the selection-eligible image) of the same imaged subject as the imaged subject shown in the facial image X.
  • the selected image update portion 118 determines that the positivity has already been computed for the imaged subject shown in the facial image Xnew (the selection-eligible image)
  • the selected image update portion 118 compares the positivity 105 that was computed for the facial image Xnew (the selection-eligible image) to the positivity 125 that was set for the facial image X (the selection candidate image), for which the positivity has already been computed.
  • the selected image update portion 118 can determine that the image with the greater positivity value will serve as the selection candidate image.
  • the positivity is greater for the selection candidate image (the facial image X) than for the selection-eligible image (the facial image Xnew)
  • the selected image update portion 118 does not perform an update to make the facial image Xnew (the selection-eligible image) the new selection candidate image. Therefore, the facial image X continues to be stored as the selection candidate image.
  • the selected image update portion 118 performs the processing in the first to third update portions described above each time the positivity is computed by the positivity computation portion 116 . Therefore, for every selection-eligible image, the selection candidate image that is stored after the positivity is computed is the image with the greatest positivity for the imaged subject. Note that in a case where the positivity that is computed for the selection-eligible image is the same value as the positivity that is set for the selection candidate image, it is obviously possible for the selected image update portion 118 to perform the update such that the selection-eligible image becomes the new selection candidate image, and for the selected image update portion 118 not to perform the update of the selection candidate image.
  • the image processing portion 102 can select the image with the greatest positivity to be the selected image. In the cases described above, it is possible to prevent more than one image of an imaged subject from being selected as the selected image for the imaged subject.
  • the image processing device 100 according to the embodiments of the present invention is not limited to this example. It is also possible, for example, to select a plurality of the selected images for the same imaged subject.
  • the configuration described above can also be implemented in such a way that the selected image update portion 118 stores a plurality of the selection candidate images for the same imaged subject and then compares the positivity that the positivity computation portion 116 computes for the selection-eligible image to the positivity that is set for each of the plurality of the selection candidate images (that is, performs a one-to-many comparison).
  • the user can also increase and decrease the value of the positivity that is set for the selection candidate image updated by the selected image update portion 118 (or, if there is a plurality of the selection candidate images, the positivity that is set for each of the selection candidate images) by operating the operation portion 108 as a second positivity editing portion.
  • FIG. 4 shows a case in which the selection candidate images are stored in the storage portion 106 .
  • the user can adjust the value of the positivity, so the image processing device 100 can treat any image that the user desires as the selected image.
  • the image processing device 100 can have a checking portion (not shown in the drawings) that checks the selection candidate image updated by the selected image update portion 118 , so that the user can use the operation portion 108 to adjust the positivity value.
  • the list creation portion 120 is an example of an image selection portion according to the embodiments of the present invention.
  • the list creation portion 120 can use the selected image that the selected image update portion 118 outputs for each imaged subject (the selection candidate image that the selected image update portion 118 updates) to create a selected image list containing all the selected images.
  • FIG. 11 is an explanatory figure that shows an example of the selected image list according to the embodiments of the present invention.
  • FIG. 11 is an example of a list in which the selected images of the imaged subjects are arranged in descending order by the positivity, based on the positivity that is set for each selected image. Because the list creation portion 120 creates the selected image list as shown in FIG. 11 , the user can easily check the selected image that was selected by the image processing device 100 for each imaged subject.
  • additional information such as the names of the imaged subjects or the like
  • the list creation portion 120 creates, as shown in FIG. 11 .
  • the user can, for example, use the operation portion 108 to edit (add, change, delete, and the like) the additional information, such as the names of the imaged subjects or the like.
  • the additional information may also be added based on information that is appended to the selected image (for example, a tag, a digital watermark, or the like).
  • the user can, for example, cause the list creation portion 120 to create the selected image list selectively by transmitting commands through the operation portion 108 .
  • the list creation portion 120 does not create the selected image list
  • the selected image for each imaged subject can, for example, be output individually.
  • the image processing device 100 extracts the facial image for each imaged subject from the selection-eligible images that are sequentially input to the image processing portion 102 and uses the facial image to compute the positivity for each selection-eligible image.
  • the image processing device 100 can then compare the selection-eligible image for which the positivity has been computed to the selection candidate image, which is the image with the greatest positivity among the selection-eligible images for the same imaged subject for which the positivity has already been computed.
  • the image processing device 100 can then select the image with the greatest positivity to be the new selection candidate image. Therefore, for each imaged subject, the image processing device 100 can select the image with the greatest positivity to be the new selection candidate image.
  • the positivity that the image processing device 100 uses for selecting images is a quantified indicator that indicates the relationship between the imaged subject in the selection-eligible image and the person who acquired the selection-eligible image.
  • the positivity is computed based on the selection-eligible image. Therefore, the user who uses the image processing device 100 can obtain, for the imaged subject, a specific image in which a positive feeling is felt toward the person who acquired the image of the imaged subject.
  • the image may be an image in which the imaged subject is smiling, an image in which the imaged subject's line of vision is directed toward the image acquisition device, or the like.
  • the image processing device 100 can select a selected image for each imaged subject, the user can easily check which imaged subjects are included in the selection-eligible images. Therefore, the image processing device 100 can assist the user's recollection of memories. Moreover, because the image processing device 100 selects the selected image for each imaged subject based on the positivity, it is possible for the image processing device 100 to select an image of an imaged subject the user did not consider. It is thus possible for the image processing device 100 to act in a positive manner on the user's mental faculties. Therefore, the image processing device 100 can exhibit an effect of helping to maintain the user's mental health.
  • the image processing device 100 can also be provided with the positivity editing portions that make it possible for the user to edit the positivity. Therefore, the user can make any desired selection-eligible image the image that is selected as the selected image.
  • the image processing device 100 can also have the selection-eligible information that specifies the imaged subjects for which image selection is permitted and the selection-ineligible information that specifies the imaged subjects for which image selection is not permitted. Therefore, the image processing device 100 can use the selection-eligible information and the selection-ineligible information to control whether or not an image is selected for each imaged subject.
  • the user can also set the selection-eligible information and the selection-ineligible information as desired. Therefore, the user can obtain only an image of a specific imaged subject or can intentionally exclude an image of an undesired imaged subject.
  • a program that causes the image processing device 100 according to the first embodiment to function as a computer can update the selection candidate image for each imaged subject and can select an image for each imaged subject from at least one image.
  • FIG. 12 is a flowchart that shows a flow of the image processing method according to the embodiments of the present invention. Note that the image processing method shown in FIG. 12 shows one method for image processing and is repeated only for a number of cycles equal to the number of the selection-eligible images that are sequentially input.
  • the facial region of the imaged subject within one selection-eligible image among the sequentially input selection-eligible images is specified, and the facial image is extracted (Step S 100 ).
  • the facial image extracted at Step S 100 is used to determine whether or not the imaged subject shown in the facial image is eligible for image selection (Step S 102 ).
  • the determination at Step S 102 can be performed by, for example, face recognition of the facial image extracted at Step S 100 , using the selection-eligible information for specifying the images that are eligible for selection and the selection-ineligible information for specifying the images that are not eligible for selection.
  • Step S 102 If it is determined at Step S 102 that the imaged subject shown in the facial image is not eligible for image selection, the processing of the selection-eligible image from which the facial image was extracted at Step S 100 terminates.
  • the facial image extracted at Step S 100 is used to compute the positivity that is the quantified indicator that indicates the relationship between the imaged subject in the selection-eligible image and the person who acquired the selection-eligible image (Step S 104 ).
  • Step S 106 a determination is made as to whether or not there exists a selection candidate image for which the positivity has already been computed and in which the imaged subject is the same as the imaged subject shown in the facial image for which the positivity was computed at Step S 104 (Step S 106 ).
  • Step S 106 If it is determined at Step S 106 that a selection candidate image showing the same imaged subject and for which the positivity has already been computed does not exist, the selection-eligible image from which the facial image was extracted at Step S 100 is made the selection candidate image that is the candidate to be the selected image (Step S 112 ).
  • Step S 106 if it is determined at Step S 106 that a selection candidate image showing the same imaged subject and for which the positivity has already been computed does exist, the positivity that was computed at Step S 104 (the positivity of the selection-eligible image) and the positivity that is set for the selection candidate image are compared (Step S 108 ). The result of the positivity comparison at Step S 108 is then used to make the image with the greater positivity the new selection candidate image (Step S 110 ).
  • the image processing method extracts the facial image of the imaged subject from the sequentially input selection-eligible image, then uses the facial image to compute the positivity of the selection-eligible image.
  • the image processing method can then compare the selection-eligible image for which the positivity has been computed to the selection candidate image, which is the image with the greatest positivity among the selection-eligible images for the same imaged subject for which the positivity has already been computed.
  • the image processing method can then select the image with the greater positivity to be the new selection candidate image. Therefore, the image processing device that uses the image processing method according to the embodiments of the present invention can select the image with the greatest positivity as the selected image for each imaged subject.
  • the image processing device that uses the image processing method according to the embodiments of the present invention can control whether or not an image is selected for each imaged subject.
  • the image processing device 100 according to the first embodiment of the present invention was described above as an individual image processing device, that is, as a device that functions in a stand-alone state.
  • the image processing device according to the embodiments of the present invention is not limited to the device that functions in a stand-alone state. Accordingly, an image processing device according to a second embodiment of the present invention, which is capable of performing communication with an external device, will be explained next, as will examples of the use of the image processing device according to a second embodiment.
  • FIG. 13 is a block diagram that shows an example of a system that uses the image processing device according to the second embodiment of the present invention.
  • the system that uses the image processing device according to the second embodiment of the present invention includes image processing devices 200 , 300 , and the like and an external device 500 .
  • the image processing devices 200 , 300 , and the like are connected to the external device 500 through a network circuit 600 .
  • the network circuit 600 may be a fixed line network such as a local area network (LAN), a wide area network (WAN), or the like, for example.
  • the network circuit 600 may also be a wireless network, such as a wireless local area network (WLAN) or the like that uses multiple-input multiple-output (MIMO), or the Internet, which uses a communication protocol such as the Transmission Control Protocol/Internet Protocol (TCP/IP) or the like.
  • the network circuit 600 is not limited to these examples.
  • the image processing device 200 is basically configured in the same manner as the image processing device 100 according to the first embodiment of the present invention. Unlike the image processing device 100 , the image processing device 200 has an image processing device communication portion (not shown in the drawings) for performing communication with the external device 500 through the network circuit 600 . Therefore, the image processing device 200 can select a selected image for each imaged subject in the same manner as the image processing device 100 .
  • the image processing device communication portion (not shown in the drawings) can transmit to the external device 500 the selected image that the image processing portion 102 selects for each imaged subject and the selected image list.
  • the image processing devices 300 and the like are configured in the same manner as the image processing device 200 .
  • the external device 500 can include an external device storage portion 502 and a positivity ranking creation portion 504 .
  • the external device 500 may also include a control portion (not shown in the drawings) that is configured from an MPU or the like and that controls the entire external device 500 , a communication portion (not shown in the drawings) for communicating with the image processing devices 200 , 300 , and the like, an operation portion, and the like.
  • the external device storage portion 502 is a storage portion that is included in the external device 500 .
  • the external device storage portion 502 may be a magnetic storage medium such as a hard disk or the like, a non-volatile memory such as a flash memory or the like, a magneto-optical disk, or the like, but the external device storage portion 502 is not limited to these examples.
  • the external device storage portion 502 can store the selected images for each imaged subject and the selected image lists that are transmitted from the image processing devices 200 , 300 , and the like.
  • a selected image list A 506 transmitted from the image processing device 200 and a selected image list B 508 transmitted from the image processing device 300 are stored in the external device storage portion 502 .
  • the positivity ranking creation portion 504 uses the selected image lists 506 , 508 that are stored in the external device storage portion 502 to tabulate the positivities for the imaged subjects that are included in each of the selected image lists and to create a ranking. Note that it is obvious that the positivity ranking creation portion 504 can also create the ranking by using the selected images that are transmitted from the image processing devices 200 , 300 , and the like.
  • FIG. 14 is an explanatory figure that shows an example of the positivity ranking according to the embodiments of the present invention.
  • FIG. 14 is a figure that shows the positivity ranking for a fixed period (one month).
  • the positivities that are shown in the positivity ranking that the positivity ranking creation portion 504 creates can be computed and set in a variety of ways.
  • the positivities can be, for example, the average value for each imaged subject, the total value for each imaged subject, the frequency with which each imaged subject appears in a plurality of the selected image lists, or the like.
  • the image of each imaged subject that is shown in the positivity ranking that the positivity ranking creation portion 504 creates can be, for example, the selected image for which the greatest positivity is set for the imaged subject, but the image of each imaged subject that is shown in the positivity ranking is not limited to this example.
  • the positivity ranking that the positivity ranking creation portion 504 creates can be expressed, for example, as an image file in the JPEG format, the Graphics Interchange Format (GIF), or the like, as a Hypertext Markup Language (HTML) document, or the like.
  • GIF Graphics Interchange Format
  • HTML Hypertext Markup Language
  • using the selected images for each imaged subject and the selected lists that are transmitted from the image processing devices 200 , 300 , and the like to create the positivity ranking like that shown in FIG. 14 makes it possible to implement a view of the imaged subjects such as a page view on the World Wide Web (WWW), for example.
  • WWW World Wide Web
  • FIG. 13 shows a configuration that includes a plurality of the image processing devices 200 , 300 , and the like and the external device 500 , but the second embodiment of the present invention is not limited to this configuration.
  • the second embodiment may also be configured to have the one image processing device 200 and the external device 500 . If the system that uses the image processing device according to the second embodiment of the present invention is configured in this manner, the positivity ranking creation portion 504 can function as the image selection portion according to the embodiments of the present invention.
  • the selected image list can be created by the positivity ranking creation portion 504 .
  • the first use example shown in FIG. 13 describes a configuration in which the positivity ranking is created by using the selected images for the imaged subjects and the selected image lists that are transmitted from the image processing devices 200 , 300 , and the like.
  • the system that uses the image processing device according to the second embodiment of the present invention is not limited to this example.
  • the external device may also be configured such that it merely stores the selected images for the imaged subjects and the selected image lists that are transmitted from the image processing devices 200 , 300 , and the like.
  • the image processing device can use the image processing method according to the embodiments of the present invention to perform the selection of the selected images for the images that the image processing device acquires (the selection-eligible images).
  • the image processing device can then transmit the selection results (the selected images for the imaged subjects and the selected image list) to a home PC owned by the person who acquired the images, the home PC being the external device. Therefore, after returning home, the person who acquired the images can easily check which imaged subjects are included in the selection-eligible images just by checking the selection results that are stored in the home PC.
  • first and second use examples described above are merely examples of the system that uses the image processing device according to the second embodiment of the present invention, and a system that uses the image processing device according to the embodiments of the present invention is obviously not limited to these examples.
  • the image processing device according to the second embodiment of the present invention has basically the same configuration as the image processing device 100 according to the first embodiment of the present invention. Therefore, the image processing device according to the second embodiment of the present invention exhibits the same sort of effect as the image processing device 100 according to the first embodiment of the present invention.
  • the image processing device can transmit the selected images for the imaged subjects and the selected image list to the external device. It is therefore possible to implement an application system such as a system that creates the positivity ranking, for example.
  • the external device has been explained as a configuring element of the system that uses the image processing device according to the second embodiment of the present invention, but the embodiments of the present invention are not limited to this configuration.
  • the present invention can also be applied to a computer such as a PC, a server, or the like.
  • the image processing portion 102 according to the first embodiment of the present invention includes the list creation portion 120 , but is not limited to this configuration.
  • the image processing portion 102 can also be configured such that it does not include the list creation portion 120 .
  • the image processing device according to the present invention can update the selection candidate image for the imaged subject based on the positivity of the relationship between the imaged subject and the person who acquired the image.
  • the image processing device according to the present invention can therefore select an image for the imaged subject from at least one image, based on the positivity of the relationship between the imaged subject and the person who acquired the image.
  • the image processing portion 102 according to the first embodiment of the present invention also includes the eligibility determination portion 114 , but is not limited to this configuration.
  • the image processing portion 102 can also be configured such that it does not include the eligibility determination portion 114 .
  • the image processing device according to the present invention can select an image for the imaged subject from at least one image, based on the positivity of the relationship between the imaged subject and the person who acquired the image.
  • the image processing devices have been explained in terms of a configuration that selects the image based on the positivity, but the image processing devices are not limited to this example.
  • the positivity can be computed using the variable value derived from the sound that is used in Equation 2, such that the input sound can be factored in to the image selection based on that positivity.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Library & Information Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Image Processing (AREA)
  • Processing Or Creating Images (AREA)
  • Image Analysis (AREA)
US12/074,667 2007-03-08 2008-03-05 Image processing device, image processing method, and program Active 2031-01-02 US8363910B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2007058704A JP2008225550A (ja) 2007-03-08 2007-03-08 画像処理装置、画像処理方法、およびプログラム
JPJP2007-058704 2007-03-08
JP2007-058704 2007-03-08

Publications (2)

Publication Number Publication Date
US20080273798A1 US20080273798A1 (en) 2008-11-06
US8363910B2 true US8363910B2 (en) 2013-01-29

Family

ID=39844147

Family Applications (1)

Application Number Title Priority Date Filing Date
US12/074,667 Active 2031-01-02 US8363910B2 (en) 2007-03-08 2008-03-05 Image processing device, image processing method, and program

Country Status (4)

Country Link
US (1) US8363910B2 (ru)
JP (1) JP2008225550A (ru)
KR (1) KR20080082526A (ru)
CN (1) CN101261680B (ru)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5017989B2 (ja) 2006-09-27 2012-09-05 ソニー株式会社 撮像装置、撮像方法
JP2008096868A (ja) 2006-10-16 2008-04-24 Sony Corp 撮像表示装置、撮像表示方法
JP4853320B2 (ja) * 2007-02-15 2012-01-11 ソニー株式会社 画像処理装置、画像処理方法
JP4424364B2 (ja) * 2007-03-19 2010-03-03 ソニー株式会社 画像処理装置、画像処理方法
JP4396720B2 (ja) * 2007-03-26 2010-01-13 ソニー株式会社 画像処理装置、画像処理方法、およびプログラム
JP2009003334A (ja) * 2007-06-25 2009-01-08 Sony Corp 画像撮像装置、撮像制御方法
US20130243270A1 (en) * 2012-03-16 2013-09-19 Gila Kamhi System and method for dynamic adaption of media based on implicit user input and behavior
KR102469694B1 (ko) * 2015-08-24 2022-11-23 삼성전자주식회사 카메라를 구비하는 장치에서 사진 촬영 지원 기법
CN106447758B (zh) * 2016-09-26 2019-09-17 上海传英信息技术有限公司 图像互换格式图片的生成方法和移动终端
US10408190B2 (en) 2016-10-07 2019-09-10 Robert B. Deioma Wind turbine with open back blade
US10776610B2 (en) * 2017-09-15 2020-09-15 Ruth Ellen Cashion LLC System for monitoring facial presentation of users

Citations (58)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS6353531A (ja) 1986-08-25 1988-03-07 Canon Inc 像ブレ防止カメラ
US4901096A (en) 1988-12-30 1990-02-13 Lemelson Jerome H Camera apparatus and method
JPH0392830A (ja) 1989-09-06 1991-04-18 Asahi Optical Co Ltd 像ブレ防止カメラ
JPH05142616A (ja) 1991-11-15 1993-06-11 Nikon Corp ブレ防止カメラ
JPH0630374A (ja) 1992-07-10 1994-02-04 Kyocera Corp 不要撮影を防止する電子スチルカメラ
JPH0743803A (ja) 1993-07-29 1995-02-14 Olympus Optical Co Ltd カメラの手ぶれ防止装置
JPH07218970A (ja) 1994-02-03 1995-08-18 Nikon Corp カメラ
US5463443A (en) 1992-03-06 1995-10-31 Nikon Corporation Camera for preventing camera shake
JPH08110540A (ja) 1995-04-24 1996-04-30 Asahi Optical Co Ltd 像ブレ防止カメラ
JPH1048681A (ja) 1996-05-10 1998-02-20 Olympus Optical Co Ltd ぶれ防止カメラ
US5905848A (en) 1995-03-15 1999-05-18 Canon Kabushiki Kaisha Image pickup apparatus with vibration compensation
JPH11174520A (ja) 1997-12-16 1999-07-02 Canon Inc 撮像装置
US5923908A (en) 1997-10-30 1999-07-13 Eastman Kodak Company Camera with touch sensitive control
JPH11205761A (ja) 1998-01-14 1999-07-30 Mitsubishi Electric Corp カメラ機能付携帯電話装置
JP2000132693A (ja) 1998-10-27 2000-05-12 Sony Corp 画像処理装置および方法、並びに提供媒体
JP2001197296A (ja) 1999-10-29 2001-07-19 Fuji Photo Film Co Ltd 画像処理方法および装置
US20010010544A1 (en) 2000-02-01 2001-08-02 Asahi Kogaku Kogyo Kabushiki Digital camera
JP2002023716A (ja) 2000-07-05 2002-01-25 Pfu Ltd プレゼンテーションシステムおよび記録媒体
JP2002049912A (ja) 2000-08-04 2002-02-15 Nri & Ncc Co Ltd 人物画像取得システム
JP2003110999A (ja) 2001-07-27 2003-04-11 Hewlett Packard Co <Hp> デジタル記録を共有する方法
US6677969B1 (en) 1998-09-25 2004-01-13 Sanyo Electric Co., Ltd. Instruction recognition system having gesture recognition function
JP2004062560A (ja) 2002-07-30 2004-02-26 Omron Corp 顔照合装置および顔照合方法
JP2004120404A (ja) 2002-09-26 2004-04-15 Fuji Photo Film Co Ltd 画像配信装置および画像処理装置並びにプログラム
JP2004134950A (ja) 2002-10-09 2004-04-30 Sony Corp 画像合成方法および画像合成装置
US20040130628A1 (en) 2003-01-08 2004-07-08 Stavely Donald J. Apparatus and method for reducing image blur in a digital camera
JP2004242360A (ja) 2004-04-16 2004-08-26 Sony Corp 撮像装置
US20050046730A1 (en) 2003-08-25 2005-03-03 Fuji Photo Film Co., Ltd. Digital camera
JP2005110004A (ja) 2003-09-30 2005-04-21 Casio Comput Co Ltd 画像処理装置、利用者端末装置及びプログラム
JP2005182526A (ja) 2003-12-19 2005-07-07 Seiko Epson Corp 著作権保護システム、配置装置、保護装置及び著作権保護方法
JP2005303511A (ja) 2004-04-08 2005-10-27 Logic:Kk 画像蓄積システム、携帯端末、画像蓄積装置、画像蓄積方法、画像送信プログラムおよび画像蓄積プログラム
JP2005352239A (ja) 2004-06-11 2005-12-22 Mitsubishi Electric Corp カメラ付き携帯端末機器
JP3740351B2 (ja) 2000-06-23 2006-02-01 日本電信電話株式会社 画像加工装置および方法およびこの方法の実行プログラムを記録した記録媒体
JP2006050163A (ja) 2004-08-03 2006-02-16 Olympus Corp 撮像装置
US7035440B2 (en) * 2000-07-03 2006-04-25 Fuji Photo Film Co., Ltd. Image collecting system and method thereof
US20060165396A1 (en) 2005-01-24 2006-07-27 Canon Kabushiki Kaisha Image pickup apparatus and image pickup method therefor as well as image pickup program implementing the method and storage medium storing the program
JP2006197373A (ja) 2005-01-14 2006-07-27 Mitsubishi Electric Corp 視聴者情報測定装置
US20060171707A1 (en) 2005-01-28 2006-08-03 Canon Kabushiki Kaisha Camera body, interchangeable lens, and camera system
JP2006201531A (ja) 2005-01-21 2006-08-03 Konica Minolta Holdings Inc 撮像装置
JP2006202049A (ja) 2005-01-20 2006-08-03 Omron Corp 顔認識装置
JP2006202181A (ja) 2005-01-24 2006-08-03 Sony Corp 画像出力方法および装置
JP2006203600A (ja) 2005-01-21 2006-08-03 Konica Minolta Holdings Inc 撮像装置、画像処理装置、画像処理システム、画像処理方法及びプログラム
JP2006221378A (ja) 2005-02-09 2006-08-24 Smart Wireless Kk 顔認証を利用した警報システムおよび方法
US20060228029A1 (en) * 2005-03-29 2006-10-12 Microsoft Corporation Method and system for video clip compression
JP2006279291A (ja) 2005-03-28 2006-10-12 Toshiba Corp 撮影装置
US20060257026A1 (en) 2005-05-16 2006-11-16 Shiffer Katerina L Methods and apparatus for efficient, automated red eye detection
JP2006319610A (ja) 2005-05-12 2006-11-24 Matsushita Electric Ind Co Ltd 撮像装置
JP2006331271A (ja) 2005-05-30 2006-12-07 Nippon Hoso Kyokai <Nhk> 代表画像抽出装置及び代表画像抽出プログラム
JP2006330800A (ja) 2005-05-23 2006-12-07 Nippon Telegr & Teleph Corp <Ntt> 画像合成システム、画像合成方法、この方法のプログラム
US20070014543A1 (en) 2005-07-13 2007-01-18 Canon Kabushiki Kaisha Image processing apparatus and control method therefor
US20070076960A1 (en) * 2005-09-30 2007-04-05 Fuji Photo Film Co., Ltd. Image display apparatus and method, computer-readable recording medium on which the program is recorded, and photograph print order accepting apparatus
JP2007166542A (ja) 2005-12-16 2007-06-28 Canon Inc 画像処理装置及び画像処理方法
JP2007249527A (ja) 2006-03-15 2007-09-27 Omron Corp 画像編集装置、画像編集方法および画像編集プログラム
US20080231714A1 (en) 2007-03-22 2008-09-25 Texas Instruments Incorporated System and method for capturing images
US20080239092A1 (en) 2007-03-26 2008-10-02 Sony Corporation Image processing apparatus, image processing method and program
US20080316339A1 (en) 2007-06-25 2008-12-25 Sony Corporation Picture imaging apparatus and imaging control method
US20090016645A1 (en) 2007-03-19 2009-01-15 Sony Corporation Image processing apparatus and image processing method
US7596307B2 (en) 2002-12-25 2009-09-29 Nikon Corporation Blur correction camera system
US20100066840A1 (en) 2007-02-15 2010-03-18 Sony Corporation Image processing device and image processing method

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI221574B (en) * 2000-09-13 2004-10-01 Agi Inc Sentiment sensing method, perception generation method and device thereof and software
US6931147B2 (en) * 2001-12-11 2005-08-16 Koninklijke Philips Electronics N.V. Mood based virtual photo album
JP2004237022A (ja) * 2002-12-11 2004-08-26 Sony Corp 情報処理装置および方法、プログラム、並びに記録媒体
JP2006115406A (ja) * 2004-10-18 2006-04-27 Omron Corp 撮像装置

Patent Citations (60)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS6353531A (ja) 1986-08-25 1988-03-07 Canon Inc 像ブレ防止カメラ
US4901096A (en) 1988-12-30 1990-02-13 Lemelson Jerome H Camera apparatus and method
JPH0392830A (ja) 1989-09-06 1991-04-18 Asahi Optical Co Ltd 像ブレ防止カメラ
JPH05142616A (ja) 1991-11-15 1993-06-11 Nikon Corp ブレ防止カメラ
US5463443A (en) 1992-03-06 1995-10-31 Nikon Corporation Camera for preventing camera shake
JPH0630374A (ja) 1992-07-10 1994-02-04 Kyocera Corp 不要撮影を防止する電子スチルカメラ
JPH0743803A (ja) 1993-07-29 1995-02-14 Olympus Optical Co Ltd カメラの手ぶれ防止装置
JPH07218970A (ja) 1994-02-03 1995-08-18 Nikon Corp カメラ
US5905848A (en) 1995-03-15 1999-05-18 Canon Kabushiki Kaisha Image pickup apparatus with vibration compensation
JPH08110540A (ja) 1995-04-24 1996-04-30 Asahi Optical Co Ltd 像ブレ防止カメラ
JPH1048681A (ja) 1996-05-10 1998-02-20 Olympus Optical Co Ltd ぶれ防止カメラ
US5923908A (en) 1997-10-30 1999-07-13 Eastman Kodak Company Camera with touch sensitive control
JPH11174520A (ja) 1997-12-16 1999-07-02 Canon Inc 撮像装置
JPH11205761A (ja) 1998-01-14 1999-07-30 Mitsubishi Electric Corp カメラ機能付携帯電話装置
US6677969B1 (en) 1998-09-25 2004-01-13 Sanyo Electric Co., Ltd. Instruction recognition system having gesture recognition function
JP2000132693A (ja) 1998-10-27 2000-05-12 Sony Corp 画像処理装置および方法、並びに提供媒体
JP2001197296A (ja) 1999-10-29 2001-07-19 Fuji Photo Film Co Ltd 画像処理方法および装置
US20010010544A1 (en) 2000-02-01 2001-08-02 Asahi Kogaku Kogyo Kabushiki Digital camera
JP3740351B2 (ja) 2000-06-23 2006-02-01 日本電信電話株式会社 画像加工装置および方法およびこの方法の実行プログラムを記録した記録媒体
US7035440B2 (en) * 2000-07-03 2006-04-25 Fuji Photo Film Co., Ltd. Image collecting system and method thereof
JP2002023716A (ja) 2000-07-05 2002-01-25 Pfu Ltd プレゼンテーションシステムおよび記録媒体
JP2002049912A (ja) 2000-08-04 2002-02-15 Nri & Ncc Co Ltd 人物画像取得システム
JP2003110999A (ja) 2001-07-27 2003-04-11 Hewlett Packard Co <Hp> デジタル記録を共有する方法
JP2004062560A (ja) 2002-07-30 2004-02-26 Omron Corp 顔照合装置および顔照合方法
JP2004120404A (ja) 2002-09-26 2004-04-15 Fuji Photo Film Co Ltd 画像配信装置および画像処理装置並びにプログラム
JP2004134950A (ja) 2002-10-09 2004-04-30 Sony Corp 画像合成方法および画像合成装置
US7711253B2 (en) 2002-12-25 2010-05-04 Nikon Corporation Blur correction camera system
US7596307B2 (en) 2002-12-25 2009-09-29 Nikon Corporation Blur correction camera system
US20040130628A1 (en) 2003-01-08 2004-07-08 Stavely Donald J. Apparatus and method for reducing image blur in a digital camera
US20050046730A1 (en) 2003-08-25 2005-03-03 Fuji Photo Film Co., Ltd. Digital camera
JP2005110004A (ja) 2003-09-30 2005-04-21 Casio Comput Co Ltd 画像処理装置、利用者端末装置及びプログラム
JP2005182526A (ja) 2003-12-19 2005-07-07 Seiko Epson Corp 著作権保護システム、配置装置、保護装置及び著作権保護方法
JP2005303511A (ja) 2004-04-08 2005-10-27 Logic:Kk 画像蓄積システム、携帯端末、画像蓄積装置、画像蓄積方法、画像送信プログラムおよび画像蓄積プログラム
JP2004242360A (ja) 2004-04-16 2004-08-26 Sony Corp 撮像装置
JP2005352239A (ja) 2004-06-11 2005-12-22 Mitsubishi Electric Corp カメラ付き携帯端末機器
JP2006050163A (ja) 2004-08-03 2006-02-16 Olympus Corp 撮像装置
JP2006197373A (ja) 2005-01-14 2006-07-27 Mitsubishi Electric Corp 視聴者情報測定装置
JP2006202049A (ja) 2005-01-20 2006-08-03 Omron Corp 顔認識装置
JP2006203600A (ja) 2005-01-21 2006-08-03 Konica Minolta Holdings Inc 撮像装置、画像処理装置、画像処理システム、画像処理方法及びプログラム
JP2006201531A (ja) 2005-01-21 2006-08-03 Konica Minolta Holdings Inc 撮像装置
CN1812501A (zh) 2005-01-24 2006-08-02 佳能株式会社 摄像装置及其摄像方法
JP2006202181A (ja) 2005-01-24 2006-08-03 Sony Corp 画像出力方法および装置
US20060165396A1 (en) 2005-01-24 2006-07-27 Canon Kabushiki Kaisha Image pickup apparatus and image pickup method therefor as well as image pickup program implementing the method and storage medium storing the program
US20060171707A1 (en) 2005-01-28 2006-08-03 Canon Kabushiki Kaisha Camera body, interchangeable lens, and camera system
JP2006221378A (ja) 2005-02-09 2006-08-24 Smart Wireless Kk 顔認証を利用した警報システムおよび方法
JP2006279291A (ja) 2005-03-28 2006-10-12 Toshiba Corp 撮影装置
US20060228029A1 (en) * 2005-03-29 2006-10-12 Microsoft Corporation Method and system for video clip compression
JP2006319610A (ja) 2005-05-12 2006-11-24 Matsushita Electric Ind Co Ltd 撮像装置
US20060257026A1 (en) 2005-05-16 2006-11-16 Shiffer Katerina L Methods and apparatus for efficient, automated red eye detection
JP2006330800A (ja) 2005-05-23 2006-12-07 Nippon Telegr & Teleph Corp <Ntt> 画像合成システム、画像合成方法、この方法のプログラム
JP2006331271A (ja) 2005-05-30 2006-12-07 Nippon Hoso Kyokai <Nhk> 代表画像抽出装置及び代表画像抽出プログラム
US20070014543A1 (en) 2005-07-13 2007-01-18 Canon Kabushiki Kaisha Image processing apparatus and control method therefor
US20070076960A1 (en) * 2005-09-30 2007-04-05 Fuji Photo Film Co., Ltd. Image display apparatus and method, computer-readable recording medium on which the program is recorded, and photograph print order accepting apparatus
JP2007166542A (ja) 2005-12-16 2007-06-28 Canon Inc 画像処理装置及び画像処理方法
JP2007249527A (ja) 2006-03-15 2007-09-27 Omron Corp 画像編集装置、画像編集方法および画像編集プログラム
US20100066840A1 (en) 2007-02-15 2010-03-18 Sony Corporation Image processing device and image processing method
US20090016645A1 (en) 2007-03-19 2009-01-15 Sony Corporation Image processing apparatus and image processing method
US20080231714A1 (en) 2007-03-22 2008-09-25 Texas Instruments Incorporated System and method for capturing images
US20080239092A1 (en) 2007-03-26 2008-10-02 Sony Corporation Image processing apparatus, image processing method and program
US20080316339A1 (en) 2007-06-25 2008-12-25 Sony Corporation Picture imaging apparatus and imaging control method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Hirasawa et al., "Development of Face Image Capturing System for Best Appearance-The Improvement of the Face Detection Algorithm", Institute of Electronics, Information and Communication Engineers Technical Study Reports, Japan, Corporate Judicial Person Institute of Electronics, Information and Communication Engineers, Nov. 11, 2004, vol. 104, No. 449, p. 61-66.
Hirasawa et al., "Development of Face Image Capturing System for Best Appearance—The Improvement of the Face Detection Algorithm", Institute of Electronics, Information and Communication Engineers Technical Study Reports, Japan, Corporate Judicial Person Institute of Electronics, Information and Communication Engineers, Nov. 11, 2004, vol. 104, No. 449, p. 61-66.
Newton, "Philosophiae Naturalis Principia Mathematica" 1687, p. 12-13, vol. 1, Royal Society, London.

Also Published As

Publication number Publication date
JP2008225550A (ja) 2008-09-25
CN101261680B (zh) 2011-02-23
CN101261680A (zh) 2008-09-10
US20080273798A1 (en) 2008-11-06
KR20080082526A (ko) 2008-09-11

Similar Documents

Publication Publication Date Title
US8363910B2 (en) Image processing device, image processing method, and program
US11714523B2 (en) Digital image tagging apparatuses, systems, and methods
CN105981368B (zh) 在成像装置中的照片构图和位置引导
CN105787908B (zh) 用于高动态范围图像的图像处理方法、装置及存储介质
JP4902270B2 (ja) デジタル画像のコレクションの組み立て方法
US8650242B2 (en) Data processing apparatus and data processing method
DE112019001257T5 (de) Stabilisierung von video zur verringerung von kamera- und gesichtsbewegung
GB2522670A (en) Computing device
KR20120028491A (ko) 이미지 데이터 관리장치 및 방법
JP6108755B2 (ja) 撮影機器、撮影画像送信方法及び撮影画像送信プログラム
JP5440173B2 (ja) 画像補正方法及び画像補正装置並びにプログラム
US20150172669A1 (en) System and method for processing compressed images and video for improved data communication
CN105940673A (zh) 图像处理装置、摄像装置、图像处理方法、程序及记录介质
JP2011054075A (ja) 画像制御装置およびプログラム
CN110012350B (zh) 一种视频处理方法及装置、设备、存储介质
US20150066923A1 (en) Reference card for scene referred metadata capture
CN111800569B (zh) 拍照处理方法、装置、存储介质及电子设备
US20170053388A1 (en) Techniques for automatically correcting groups of images
CN114096994A (zh) 图像对齐方法及装置、电子设备、存储介质
CN109408652A (zh) 一种图片搜索方法、装置及设备
CN115063803B (zh) 图像处理方法、装置、存储介质及电子设备
WO2016088477A1 (ja) 情報処理装置、情報処理方法およびプログラム
JP4317003B2 (ja) ハイライト静止画生成・配信装置
WO2019242535A1 (zh) 基于时间、位置、随机数和条形码的拍照打卡或取证方法
JP4992664B2 (ja) 画像処理装置、画像処理方法、およびプログラム

Legal Events

Date Code Title Description
AS Assignment

Owner name: SONY CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ASUKAI, MASAMICHI;TSURUTA, MASAAKI;OZAKI, NOZOMU;AND OTHERS;REEL/FRAME:021233/0723;SIGNING DATES FROM 20080513 TO 20080704

Owner name: SONY CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ASUKAI, MASAMICHI;TSURUTA, MASAAKI;OZAKI, NOZOMU;AND OTHERS;SIGNING DATES FROM 20080513 TO 20080704;REEL/FRAME:021233/0723

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STCF Information on status: patent grant

Free format text: PATENTED CASE

FPAY Fee payment

Year of fee payment: 4

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY