KR101649322B1 - Systems and methods for image-to-text and text-to-image association - Google Patents

Abstract

The present invention relates to a computerized facial image attribute evaluation system for assigning evaluations representing facial images to a plurality of individual facial attributes of a facial image; And a computerized classification device for classifying facial images according to a plurality of individual facial attributes, the evaluations being represented by adjectives.

Description

TECHNICAL FIELD The present invention relates to a system and a method for associating an image with text and a text with an image,

The present invention relates to a system and method for associating image-text and text-images, filed on February 3, 2011, entitled "SYSTEMS AND METHODS FOR IMAGE-TO-TEXT AND TEXT- "Which is incorporated herein by reference and refers to United States Patent Application Serial No. 61 / 439,021, which claims priority under 37 CFR 1.78 (a) (4) and (5) (i).

The present invention also is related to the applicant's own U.S. Patent Application No. 12 / 922,984, incorporated herein by reference.

The present invention relates generally to associating images to text and from text to images.

Current state of the art is described in U.S. Patent No. 4,926,491; 5,164,992; 5,963,670; 6,292,575; 6,301, 370; 6,819,783; 6,944, 319; 6,990,217; 7,274, 822 and 7,295, 687; And U. S. Patent Application No. 2006/0253491; 2007/0237355 and 2009/0210491.

The present invention seeks to provide an improved system and methodology for associating text to images and text to images.

Thus, according to a preferred embodiment of the present invention, a computerized facial image attribute evaluation system for assigning evaluations representing facial images to a plurality of individual facial attributes of a facial image; And a computerized classification device for classifying facial images according to a plurality of individual facial attributes, wherein the evaluations are represented by an adjective.

According to a preferred embodiment of the present invention, a computerized facial image attribute evaluation system comprises a database containing a plurality of stored evaluations corresponding to a plurality of facial images, each of the facial images comprising a plurality of individual facial attributes And at least some of the individual face attributes have evaluations expressed as adjectives associated therewith.

Preferably, the system also includes a face attribute statistical reporting function that provides statistical information derived from a plurality of stored ratings.

Preferably, the computerized facial image attribute rating system comprises: a database comprising a plurality of stored face images and a plurality of stored ratings; And an adjective-based comparison device that compares a face image with a plurality of stored face images by comparing a plurality of individual face attributes of the face image attribute and the adjective manner with a plurality of stored face images, Having at least some of the plurality of individual face attributes, at least some of the individual face attributes have evaluations expressed in an adjective associated therewith. Preferably, the adjective-based comparison device queries the database in an adjective manner.

Preferably, the system also includes a computerized identification device operative to identify at least one stored face image corresponding to the output in response to the output from the computerized classification device. Preferably, the computerized identification device operates to generate a ranked list of stored face images corresponding to the output.

Preferably, the system also includes a social network interface that allows information to be made available to a computerized facial image attribute rating system in a social network. Preferably, the system also includes a face model generation functionality operative to generate a face model corresponding to the face image. Preferably, the computerized identification device uses a face model.

Assigning evaluations representing face images to a plurality of individual face attributes of a face image according to another preferred embodiment of the present invention; And classifying the face image according to a plurality of individual face attributes, the evaluations being further provided with a computerized method for classifying a face image of a person represented by an adjective.

According to a preferred embodiment of the present invention, each of the face images has at least some of the plurality of individual face attributes, and at least some of the individual face attributes are represented by adjectives associated therewith. Preferably, the method also includes providing statistical information derived from a plurality of stored evaluations.

Preferably, each of the stored face images has at least some of the plurality of individual face attributes, and at least some of the individual face attributes have evaluations expressed by an adjective associated therewith, Comparing the plurality of individual face attributes of the plurality of stored face images to the plurality of stored face images. Preferably, the comparing step queries the database in an adjective manner.

Preferably, the method further comprises identifying at least one stored face image corresponding to an output of the classifying step. Preferably, the identifying step operates to generate a ranked list of stored face images corresponding to the output. Preferably, the method also includes making the information available to a computerized facial image attribute method evaluation device in a social network. Preferably, the method also includes generating a face model that operates to generate a face model corresponding to the face image. Preferably, the identifying step uses a face model.

A face image / person identification acquisition subsystem for obtaining at least one face image and at least one person identification item of a person according to another preferred embodiment of the present invention; A face model generation function unit operable to generate a face model corresponding to at least one face image; And an image-attribute mapping function operable to assign ratings expressed as adjectives to a plurality of face attributes of a face image, wherein the image-attribute mapping function comprises computerized sub-functions for receiving at least one face image of a person and at least one person identification item system; And a database for storing estimates of face attributes and information for a plurality of persons.

Preferably, the system also includes an attribute-image mapping function operative to identify a particular individual using the face model by identifying the stored face image using a set of evaluations of face attributes, the computerized subsystem . Preferably, the computerized subsystem comprises an evaluation combination device operable to identify a particular individual by combining the set of assessments of the face model and face attributes into a combined set of assessments that can be matched with a set of corresponding stored assessments combiner < / RTI >

Preferably, the system also includes a continuous face image acquisition subsystem for acquiring and providing at least one facial image to the computerized subsystem, wherein the computerized subsystem comprises a face corresponding to the successive facial images And to identify consecutive face images as a specific individual by identifying the stored face images, assigning estimates expressed in adjectives to a plurality of face attributes of the successive face images, and identifying at least one The person identification item relates to the person stored in the database.

Preferably, an evaluation combining device is used to identify the particular individual by combining the face model and a set of evaluations corresponding to successive face images. Preferably, at least one person identification item of the person is obtained from the pre-registration data.

Preferably, the system also includes a social network interface that enables information to be available from the social network to the computerized subsystem. Preferably, the face image / personally identifying acquisition subsystem operates to obtain at least one face image and at least one person identification item of a person other than the person interacting with the subsystem. Additionally or alternatively, the facial image / personally identifiable acquisition subsystem operates to obtain at least one facial image of an unidentified person other than the person interacting with the subsystem.

Preferably, the system includes a computerized facial image attribute evaluation system that assigns evaluations representing facial images to a plurality of individual facial attributes of the facial image; And a computerized classifier for classifying facial images according to a plurality of individual facial attributes, wherein the evaluations are represented by adjectives.

A facial image generation / acquisition unit for acquiring at least one facial image of a person according to another preferred embodiment of the present invention, a facial model generation function for generating a facial model corresponding to at least one facial image, An image-attribute mapping function operable to assign evaluations expressed as adjectives to a plurality of face attributes of the face image, wherein the image-attribute mapping function comprises a computerized subsystem for receiving the at least one face image, There is further provided a system for recognizing a person who is present in a place repeatedly, comprising a database storing said evaluations of information and face attributes.

Preferably, the computerized subsystem also includes an attribute-image mapping function that is operative to identify a corresponding stored face image associated with a particular individual using the face model, using a collection of evaluations of face attributes. Preferably, the computerized subsystem also includes an evaluation combining device operative to combine the set of evaluations of the face model and face attributes into a combined set of evaluations that can be matched with a set of corresponding stored evaluations.

Preferably, the system also includes a continuous face image acquisition subsystem for acquiring and providing at least one facial image to the computerized subsystem, wherein the computerized subsystem preferably includes a processor To identify a specific person repeatedly by recognizing the stored face image by assigning evaluations represented by adjectives to a plurality of face attributes of the successive face images, As the face image of the user.

Preferably, the evaluating combination device is used to recognize a person who exists repeatedly by combining a set of evaluations corresponding to a face model and a successive facial image. Preferably, the system also includes a repeat presence statistics generator that uses sets of facial models and evaluations to generate attribute-based statistics for the repeatedly existing persons at the site. Preferably, the system also includes a social network interface that allows information to be available from the social network to the computerized subsystem.

Preferably, the face image / personally identifying acquisition subsystem operates to obtain at least one face image and at least one person identification item of a person other than the person interacting with the subsystem. Additionally or alternatively, the facial image / personally identification acquisition subsystem operates to acquire at least one face image of an unidentified person other than the person interacting with the subsystem.

Preferably, the system includes a computerized facial image attribute evaluation system that assigns evaluations representing facial images to a plurality of individual facial attributes of the facial image; And a computerized classifier for classifying facial images according to a plurality of individual facial attributes, wherein the evaluations are represented by adjectives.

According to another preferred embodiment of the present invention there is provided a method of generating a computerized facial image attribute system evaluation apparatus capable of assigning evaluations, each represented by an adjective, to a plurality of individual facial attributes of a facial image, Collecting a plurality of face images each having at least one face image attribute represented by an adjective, receiving the evaluated face image, and assigning evaluations represented by adjectives to a plurality of individual face attributes of the face image being evaluated And generating a function that is operative to use the collected results for the computerized facial image attribute evaluation method.

Preferably, the collecting comprises collecting a plurality of facial images each having at least one facial image attribute represented by an adjective associated with the facial image from a publicly available source, and using crowdsourcing to generate the plurality of facial images And improving the correspondence between the adjectives and facial attributes that appear in the face. Preferably, the crowd sourcing uses one or more of the plurality of face images and a plurality of people viewing the adjectives and displaying a human's view of the match between the adjectives and the face attributes in the one of the plurality of face images . Preferably, the evaluations are numerical values.

According to another preferred embodiment of the present invention, a computerized facial image attribute evaluation method and apparatus for assigning evaluations expressing a facial image obtained at a point of user's reaction to a stimulus to a plurality of individual facial attributes of the facial image A system for recognizing a user response to at least one stimulus, wherein the evaluations are represented by an adjective, is also provided.

Preferably, the system also includes a computerized attribute comparison device for comparing the plurality of individual face attributes before and after applying the at least one stimulus.

According to still another preferred embodiment of the present invention, there is provided a method for evaluating a facial image, comprising: assigning evaluations representing a facial image obtained at a user's reaction to a stimulus to a plurality of individual facial attributes of the facial image; Wherein the evaluations are represented by an adjective, wherein the user responses to at least one stimulus are identified.

Preferably, the method also includes comparing a plurality of individual face attributes before and after applying the at least one stimulus.

According to another preferred embodiment of the present invention, there are provided a relationship coefficient generating apparatus for generating relationship coefficients indicating a probability that a person is in a specific context at a specific time, and a computer- A computerized system for classifying persons is further provided, including a classifying device.

Preferably, the context is one of a geographic location and an event. Preferably, the relationship coefficient includes an evaluation and a decay function. Preferably, the damping function is a linear function. Alternatively, the damping function is an exponential function.

Preferably, the context is one of the hierarchical contexts of the hierarchy. Preferably, the relationship counts of the contexts of one layer are interdependent. Preferably, the relationship coefficient generation device is operative to generate an interdependent relationship coefficient between a plurality of persons in a second context, when a plurality of persons are at least in the first context.

Preferably, the system also includes a computerized classification apparatus for classifying facial images according to a plurality of individual facial attributes.

Are included in the scope of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The invention, together with the drawings, will be better understood and appreciated from the following detailed description:
Figures 1a, 1b and 1c are schematic diagrams of an identification system using an association of image-text and text-image in accordance with a preferred embodiment of the present invention.
Figures 2a and 2b are schematic diagrams of an identification system that utilizes an association of image-text and text-image in accordance with another preferred embodiment of the present invention.
Figures 3a and 3b are schematic diagrams of an identification system that utilizes an association of image-text and text-image in accordance with another preferred embodiment of the present invention.
4A, 4B, and 4C are schematic diagrams of an identification system that utilizes an association of image-text and text-image in accordance with another preferred embodiment of the present invention.
5A and 5B are schematic diagrams of an identification system that utilizes an association of image-text and text-image in accordance with another preferred embodiment of the present invention.
6 is a schematic diagram of a user satisfaction monitoring system that utilizes image-text association in accordance with another preferred embodiment of the present invention.
FIG. 7 is a schematic diagram of an image / text / image database generation methodology useful for building a database used in the system of FIGS. 1A-6.
Figure 8 is a schematic flow diagram illustrating a training process for associating an adjective with an image.
Figure 9 is a schematic flow diagram illustrating the process of training a visual classifier.
10 is a schematic flow diagram illustrating a process for retrieving an adjective associated with an image.
11 is a schematic flow diagram illustrating a process for retrieving images associated with one or more adjectives.
12 is a schematic flow diagram illustrating a process for retrieving a face image similar to the first image.

Reference is now made to Figs. 1A, 1B and 1C which are schematic diagrams of an identification system using an association of image-text and text-image in accordance with a preferred embodiment of the present invention. Preferably, the system of FIGS. 1A-1C includes a computerized facial image attribute system evaluation apparatus for assigning values representing facial images to a plurality of individual facial attributes of a facial image, and a plurality of individual facial attributes And a computerized classification apparatus for classifying facial images according to a plurality of images, wherein the evaluations are represented by adjectives.

As shown in FIG. 1A, Mr. Jones, a customer of AAA department store on Jan. 1, enters the department store and registers as a valued customer of the department store at the registration stand 100 do. Preferably, the registry includes a computer 102 connected to a department store computer network and a digital camera 104 connected to the computer 102. The evaluation customer registration process includes inputting customer ' s personal identification details, such as the customer's name, and capturing the customer's face image 108 with the digital camera 104. [ Alternatively, the customer ' s personal identification may be retrieved, for example, from a customer ' s existing personal social network account. Alternatively, the customer may register as an evaluation location on the Internet from a remote location.

The personal identification and face image 108 preferably includes a face model generation functionality 112, an image-to-attributes mapping functionality 114, To a computerized personal identification system 110 that includes an attribute-to-images mapping functionality 116 and a value combiner 117. The computer- Preferably, the computerized person identification system 110 also includes an assessment customer database 118 that stores assessments of registration details and all registered customer face attributes. It is recognized that the database 118 may be any suitable computerized information store.

The face model generation function unit 112 operates to generate a face model 120 corresponding to the face image 108. [ It is recognized that the facial model generation function 112 may use any suitable method of generating a facial model known in the art. 1A, the face model 120 generated by the face model generation function 112 and corresponding to the face image 108 is stored in the database 118 as one of the attributes of Jones.

According to a preferred embodiment of the present invention, the image-attribute mapping function 114 operates to assign the evaluations represented by the adjectives 122 to a plurality of face attributes of the face image 108. For example, adjectives 122 representing facial attributes may include adjectives describing hair color, nose shape, skin color, face type, face beard type and presence. As shown in FIG. 1A, the adjectives generated by the attribute mapping function 114 corresponding to the face image 108 are stored in the database 118 as estimates of Jones's attributes.

Further, according to a preferred embodiment of the present invention, the attribute-image mapping function 116 operates to identify a particular individual by identifying a corresponding stored face image using a set of evaluations of face attributes.

Also, in accordance with a preferred embodiment of the present invention, the evaluator combiner 117 preferably combines a set of evaluations of face models and facial attributes with a set of corresponding evaluations to be combined with a combined collection of evaluations, To identify a particular individual.

Now, referring to FIG. 1B, a customer enters the AAA department store on a later date, such as January 17, and the digital camera 150 installed at the entrance of the department store captures the customer's face image 152. The face image 152 is transmitted to the computerized personal identification system 110 where the face model 160 corresponding to the face image 152 is preferably generated by the face model generation function 112. In addition, evaluations 162, represented by adjectives, are preferably assigned to a plurality of face attributes of face image 152 by image-attribute mapping function 114.

1B, the face model 160 and the adjectives 162 are preferably combined into a combined set of ratings by the rating combination device 117, Is compared with the set of stored assessments and identified as matching Jones 'face model and adjectives to identify the person depicted in the face image 152 captured by the camera 150 as Jones' do. It is recognized that the set of evaluations that are combined by the evaluator combiner 117 and that are compared to the sets of evaluations stored in the database 118 can be any subset of the face model 160 and adjectives 162.

Referring now to FIG. 1C, when identifying a customer who entered the department store as a registered valuation customer, for example Jones, the manager is notified by the system 110 that the valuation customer has entered the department store, And gives him a discount on the new product.

Reference is now made to Figs. 2A and 2B which are schematic diagrams of an identification system utilizing an association of image-text and text-image in accordance with another preferred embodiment of the present invention. 2A, a customer of the AAA department store enters a department store and a digital camera 200 installed at the entrance to the department store captures a customer's face image 202 on a particular day such as January 1, for example . The face image 202 is preferably a computerized image that includes a face model generation function 212, an image-attribute mapping function 214, an attribute-image mapping function 216, And transmitted to the personal identification system 210. Preferably, the computerized identification system 210 also includes a customer database 218 that stores evaluations of all customer face attributes that have preferably entered the department store and, preferably, And a visit counter 219 for tracking the accumulated number of visits. It is recognized that the database 218 may be any suitable computerized information store.

The face model generation function unit 212 operates to generate a face model 220 corresponding to the face image 202. It is recognized that the facial model generation function 212 may be any suitable method of facial model generation known in the art. 2A, the face model 220 generated by the face model generation function 212 and corresponding to the face image 202 is stored in the database 218 as one of the attributes of the customer of the face image 202 ).

According to a preferred embodiment of the present invention, the image-attribute mapping function 214 operates to assign the evaluations represented by the adjectives 222 to a plurality of face attributes of the face image 202. For example, adjectives 222 representing facial attributes may include adjectives describing the age group, sex, race, face type, mood, and overall appearance.

Also, according to a preferred embodiment of the present invention, the attribute-image mapping function 216 operates to identify a particular individual by identifying a corresponding stored face image using a set of evaluations of face attributes. In addition, the set of evaluations may be used to identify non-physical features of the customer's appearance, such as clothing type and color, which may be used to identify an individual within a short period of time if current assessments of face attributes are not available do.

Also, in accordance with a preferred embodiment of the present invention, the evaluator combiner 217 preferably combines the set of evaluations of facial models and facial attributes into a combined set of evaluations that can be matched with a corresponding set of stored evaluations, To identify a particular individual.

2A, the face model 220 and the adjectives 222 are preferably combined into a combined set of evaluations by the evaluator combiner 217, which in turn is coupled to the database 218 Is compared with the sets of stored evaluations, and it is found that the face model and the adjectives match the face model and the adjectives corresponding to the returning customer. Thus, the customer's visit counter 219 increases. It is recognized that the set of evaluations that are combined by the evaluator combiner 217 and that are compared to the sets of evaluations stored in the database 218 can be any subset of the face model 220 and adjectives 222.

Alternatively, if it is not determined that the combined set of evaluations generated by the evaluator combiner 217 matches any of the sets of evaluations stored in the database 218, The combined set and face image 202 is preferably stored in database 218 as representing a new customer and the new customer's counter 219 is initialized to one.

Now, referring to FIG. 2B, at the end time, such as 5:00 PM on January 1, the manager of the department store preferably includes a system 210 that includes segmentation of customers who entered the department store during January 1 The first report 230 is received. Such segmentation may be stored in the database 218, such as by gender, age group, race and mood, depending on the adjectives. In addition, the report 230 preferably includes information on previous visits made to the department store by customers on January 1.

In addition, the manager of the department store may also receive a second report 234 from the system 210 that includes the reclassification of returning customers who entered the department store during January 1. Such segmentation may be stored in the database 218, such as by gender, age group, race and mood, depending on the adjectives. Reports 230 and 234 are perceived to be useful, for example, in planning targeted marketing campaigns or in evaluating the success of previously executed marketing campaigns.

 Reference is now made to Figs. 3A and 3B which are schematic diagrams of an identification system utilizing an association of image-text and text-image in accordance with another preferred embodiment of the present invention. As shown in FIG. 3A, on a particular day, such as January 1, a customer of the AAA department store enters the department store and watches the merchandise at the toy store of the department store. The digital camera 250 installed in the toy store captures the customer's face image 252. As shown in FIG. 3A, additional digital cameras are preferably installed throughout the various stores of the department store.

The face image 252 includes a computerized personal identification system 264 including a face model generation function 262, an image-attribute mapping function 264, an attribute-image mapping function 266, Lt; / RTI > Preferably, the computerized identification system 260 also includes a customer database 268 that stores assessments of all customer face attributes that have entered the department store for preferably one day, As shown in FIG. It is recognized that the database 268 may be any suitable computerized information store.

The face model generation function unit 262 operates to generate a face model 270 corresponding to the face image 252. [ It is recognized that the facial model generation function 262 can utilize any suitable method of generating face models known in the art. 3A, the face model 270 generated by the face model generation function section 262 and corresponding to the face image 252 is stored in the database 268 as one of the attributes of the customer of the face image 252 ).

According to a preferred embodiment of the present invention, the image-attribute mapping function 264 operates to assign the evaluations represented by the adjectives 272 to a plurality of face attributes of the face image 252. [ For example, adjectives 272 representing facial attributes may include adjectives describing the age group, sex, race, face type, mood, and overall appearance. The adjectives generated by the attribute mapping function 264, corresponding to the face image 252, are stored in the database 268 as an evaluation of the attributes of the customer of the face image 252, as shown in FIG.

Also, according to a preferred embodiment of the present invention, the attribute-image mapping function 266 operates to identify a particular individual by identifying a corresponding stored face image using a set of evaluations of face attributes. In addition, the set of evaluations may be used to identify non-physical features of the customer's appearance, such as clothing type and color, which may be used to identify an individual within a short period of time if current assessments of face attributes are not available do.

Also, in accordance with a preferred embodiment of the present invention, the evaluator combiner 267 preferably combines the set of evaluations of the face model and face attributes into a combined set of evaluations that can be matched with a corresponding set of stored evaluations, To identify a particular individual.

In addition, the system 260 records the store the customer visited in the database 268 as a toy store.

Now, referring to FIG. 3B, at the end time, such as 5:00 PM on January 1, the manager of the department store preferably has a system 260 including segmentation of customers who entered the toy store of the department store during Jan. 1 A report 280 is received. This segmentation may be stored in database 268, such as by gender, age group, race and mood, depending on the adjectives. For example, it is recognized that the report 280 may be useful in planning a targeted marketing campaign or in evaluating the success of a previously executed marketing campaign.

Reference is now made to Figs. 4A, 4B and 4C which are schematic diagrams of an identification system utilizing an association of image-text and text-image in accordance with another preferred embodiment of the present invention. As shown in FIG. 4A, on January 1, a potential participant registers preferably to attend the annual meeting of florists via computer 300. [ As part of the registration process, a potential participant is preferably required to enter a personal identification, such as his / her name, and upload his / her at least one face image 302. Alternatively, a potential participant may choose to retrieve personal identification and, for example, one or more face images from an existing personal social network account.

The personal identification and face image 302 preferably includes a face model generation function 312, an image-attribute mapping function 314, an attribute-image mapping function 316 and an evaluation combination device 317 Lt; RTI ID = 0.0 > 310 < / RTI > Preferably, the computerized conference registration system 310 also includes a database 318 that stores the registration details and evaluations of the face attributes of all registered attendees. It is recognized that the database 318 may be any suitable computerized information store.

The face model generation function unit 312 operates to generate a face model 320 corresponding to the face image 302. It is recognized that the face model generation function 312 can use any suitable method of generating a face model known in the art. 4A, the face model 320 generated by the face model generation function 312 and corresponding to the face image 302 is stored in the database 318 as one of the attributes of Jones, a potential participant, / RTI >

According to a preferred embodiment of the present invention, the image-attribute mapping function 314 operates to assign the evaluations represented by the adjectives 322 to a plurality of face attributes of the face image 308. For example, adjectives representing facial properties may include adjectives describing hair color, nose shape, skin color, face type, face beard type and presence or absence. 4A, the adjectives generated by the attribute mapping function 314, corresponding to the face image 302, are stored in the database 318 as evaluations of the attributes of Jones, a potential participant.

Also, according to a preferred embodiment of the present invention, the attribute-image mapping function 316 operates to identify a particular individual by identifying a corresponding stored face image using a set of evaluations of face attributes.

In addition, according to a preferred embodiment of the present invention, the evaluator combiner 317 preferably combines the set of evaluations of the face model and face attributes into a combined set of evaluations that can be matched with a corresponding set of stored evaluations, To identify a particular individual.

Now, referring to FIG. 4B, it is shown that at a later date, such as January 17, the attendee enters the annual meeting of the florists and approaches the registration booth 330 at the conference hall. The registration booth 330 includes a digital camera 332 that captures the face image 334 of the attendee. The face image 334 is transmitted to the computerized conference registration system 310 where the face model 340 corresponding to the face image 334 is preferably generated by the face model generation function 312. In addition, evaluations 342, which are expressed as adjectives, are preferably assigned by the image-attribute mapping function 314 to a plurality of face attributes of the face image 334.

4B, the face model 340 and the evaluations 342 are preferably combined into a combined set of evaluations by the evaluator combiner 317, which is then stored in the database 318 And identifies the person depicted in the face image 334 captured by the camera 332 as being Jones by proving that the assessments assigned to the face model and Jones are matched. It is recognized that the set of evaluations that are combined by the evaluator combiner 317 and that are compared to the sets of evaluations stored in the database 318 can be any subset of the face model 340 and adjectives 342. When it is identified as Mr. Jones, the registration of the attendee is completed, and the meeting staff welcomes the attendee.

4C, an attendee who wishes to be introduced to another participant while attending a meeting may be able to capture their face image 350 using a digital camera embedded in the mobile communication device 352, for example, Lt; / RTI > The conference attendee's mobile communication device 352 is allowed access to a computerized conference registration system 310 via a computer network. It is recognized that the computer network may be, for example, a local computer network or the Internet.

Additionally or alternatively, the attendee may access the computerized conference registration system 310 by capturing a face image of the new attendee and preferably sending the face image with the associated person identification information to the registration system 310 Register new, currently unregistered attendees as meetings.

The mobile communication device 352 sends the image 350 via the computer network to the computerized conference registration system 310 where the image corresponding to the face image 350 The face model 360 is preferably generated by the face model generation function 312. In addition, the evaluations 362 represented by the adjectives are preferably assigned to a plurality of face attributes of the face image 350 by the image-attribute mapping function 314.

4C, the face model 360 and the evaluations 362 are preferably combined into a combined set of evaluations by the evaluator combiner 317, which in turn is coupled to the database 318 Is compared to the set of stored assessments and the person depicted in the face image 350 captured by the mobile communication device 352 is identified by Jones as being proved to match the face models and assessments assigned to Jones & . It is recognized that the set of evaluations that are combined by the rating combination device 317 and that are compared to the sets of ratings stored in the database 318 can be any subset of the face model 360 and adjectives 362. A notification identifying the attendee depicted in image 350 as Jones is retransmitted to the mobile communication device 352 by the computerized conference registration system 310 and the notification is sent by the operator of the mobile communication device 352 to Jones Let him know that he is approaching him.

Reference is now made to Figs. 5A and 5B which are schematic diagrams of an identification system utilizing an association of image-text and text-image in accordance with another preferred embodiment of the present invention. In the embodiment of Figures 5A and 5B, a relationship coefficient is used to measure the relationship between a person and a context. For example, the context may be a geographic location or an event, and the relational factor includes an evaluated and predetermined decay function. One person can have relationship counts at the same time as multiple contexts. For example, the relationship coefficient may be used to predict the probability that a person will be in a predetermined position at a particular time.

The damping function may be any mathematical function. For example, the damping function for a geographic location may be a linear function, which indicates a tendency for a person to gradually and linearly separate from its location over time. For example, the damping function for a one-time event may be an exponential damping function.

While the person is in a particular context, the current value of the generated relationship coefficient between the person and the context is set to be high. Whenever the person appears repeatedly in its context, the value of the relationship coefficient potentially increases in an exponential manner.

It is recognized that the context can be hierarchical. For example, a geographic location may be within a larger geographic area, such as a city or a country. Thus, a person having a particular geographic location and a relationship coefficient may also have a low relationship coefficient with that particular geographic location and all other geographic locations that are hierarchical, such that the distance coefficient between the particular geographic location and the hierarchical geographic location of the relational form .

It is also recognized that the relationship coefficients of others can be at least partially interdependent. For example, a first person who has been found with a second person at multiple locations at multiple times may be assigned a relatively higher relationship count for a new location where the second person was found.

As shown in FIG. 5A, on a particular day, such as January 1, 2011, dinner is served at the nearby Cafe Jaques in the Eiffel Tower, Paris, France. A friend of the meal guest captures a facial image 400 of a meal guest using a digital camera that is part of the portable mobile device 402 and sends the facial image 400 with computerized personal identification < RTI ID = 0.0 > System 410 to register the scene of a meal guest. For example, the location may be provided by a GPS module included in the device 402. Alternatively, the location may be retrieved, for example, from a social network. Using the associated time and location, a relationship coefficient that relates the location to the meal guest is generated as described above.

The computerized personal identification system 410 includes a face model generation function 412, an image-attribute mapping function 414, an attribute-image mapping function 416, and an evaluation combination device 417. Preferably, the computerized personality identification system 410 also includes a sight database 418 that stores, together with the associated time and location, estimates of the face attributes of all persons preferably discovered and registered. It is recognized that the database 418 may be any suitable computerized information store.

The face model generation function unit 412 operates to generate a face model 420 corresponding to the face image 400. [ It is recognized that the facial model generation function 422 can utilize any suitable method of face model generation known in the art. 5A, the face model 420 generated by the face model generation function 412 and corresponding to the face image 400 is stored in the database 418 as one of the personal attributes of the face image 400 ).

According to a preferred embodiment of the present invention, the image-attribute mapping function 414 is operative to assign the evaluations represented by the adjectives 422 to a plurality of face attributes of the face image 400. For example, adjectives 422 representing facial attributes may include adjectives describing age groups, gender, race, facial form, mood, and overall appearance. 5A, the adjectives generated by the attribute mapping function 414, corresponding to the face image 400, are stored in the database 418 as evaluations of the individual attributes of the face image 400. As shown in FIG. Additionally, the time and location associated with the face image 400 are also stored in the database 418. [

Also, according to a preferred embodiment of the present invention, the attribute-image mapping function 416 operates to identify a particular individual by identifying a corresponding stored face image using a set of evaluations of face attributes. In addition, the set of evaluations may be used to identify non-physical features of the customer's appearance, such as clothing type and color, which may be used to identify an individual within a short period of time if current assessments of face attributes are not available do.

Furthermore, in accordance with a preferred embodiment of the present invention, the evaluation combining device 417 preferably combines the set of evaluations of face models and face attributes into a combined set of evaluations that can be matched with a corresponding set of stored evaluations, To identify a particular individual.

Now, looking at FIG. 5B, we will have dinner at the nearby Café Jacques at the Eiffel Tower in Paris, France, for example on February 1, 2011. A passenger captures a facial image 450 of a meal guest using a digital camera that is part of the portable mobile device 452 and transmits the facial image 450 with relevant time and location over the Internet to a computerized personal identification system 410 The face model 460 corresponding to the face image 450 in the computerized personal identification system is preferably created by the face model generation function 412. [ In addition, the evaluations 462 represented by the adjectives are preferably assigned to a plurality of face attributes of the face image 450 by the image-attribute mapping function 414.

5B, the time and position associated with the face model 460, the evaluations 462, and the face image 450 are preferably combined into a combined set of evaluations by the evaluation combination device 417 , This combined set is compared to the set of ratings stored in the database 418 and the time and locations associated with the face model, assessments, and facial images are compared to the most recently discovered meal guest at the Eiffel Tower on January 1, It is found to match the assigned combined evaluations. It is recognized that the set of evaluations that are combined by the rating combination device 417 and that are compared to the sets of ratings stored in the database 418 may be any subset of the face model 460 and adjectives 462. A notification identifying the meal guest depicted in the image 450 is retransmitted to the mobile communication device 452 by the computerized personal identification system 410. [

A particular feature of this embodiment of the invention is that the relationship coefficient relating the location to a meal guest can also be used as an attribute value to increase the reliability of the identification of a meal guest.

A particular feature of this embodiment of the present invention is that a combination of evaluations of facial attributes associated with a facial image together with additional information, such as a particular location, for example an individual's frequent location, And operate to identify individuals more efficiently.

Another specific feature of the present embodiment of the present invention is that the identification of individuals according to this embodiment of the present invention is not limited to accurate identification of a specific individual based on individual identification information such as name and performance, Attributes and identification of individuals according to comprehensive behavioral information.

Reference is now made to Fig. 6, which is a schematic diagram of a user satisfaction monitoring system that utilizes an association of image-text and text-image in accordance with another preferred embodiment of the present invention. As shown in FIG. 6, the reader browses the computerized content 482 using a multimedia viewing device 480. For example, it is recognized that the device 480 may be a television device or a computer. For example, the content 482 may be a video clip, a movie, or an advertisement.

Preferably, the digital camera 484 connected to the multimedia viewing device 480 captures the face image 486 of the viewer at regular intervals, such as every few seconds, and displays the online computerized content satisfaction monitoring And transmits the image 486 to the system 490. Alternatively, the image 486 may be monitored, stored, and analyzed with appropriate functionality embedded in the device 480. [

Preferably, the system 490 includes an image-attribute mapping function 492 and a viewer lookup database 494. It is recognized that the database 494 may be any suitable computerized information store.

According to a preferred embodiment of the present invention, the image-attribute mapping function 492 assigns the rating represented by the adjective 496 to the look of the viewer captured in the face image 486, 496 < / RTI > For example, the adjective 496 may include "happy", "sad", "angry", "satisfied", and "indifferent". For example, it is recognized that adjectives 496 stored in the database 494 may be useful in evaluating the effectiveness of the content 482.

Reference is now made to Fig. 7, which is a schematic diagram of an image / text / image database creation methodology useful for building databases used in the systems of Figs. 1A-6. As shown in FIG. 7, a plurality of images 500 are collected from the publicly available image store 502 on the Internet by a computerized personal identification training system 510. For example, image store 502 may be a publicly available social network or text search engine that associates text with the image on the same page as the image or with the image displayed on one or more proximity pages. Preferably, one or more associated features are provided by the image repository having respective images 500. For example, a feature may include an individual who has published an image, may include a name, age or age range, gender, overall appearance and mood, is generally subjective, or an individual who has tagged the published image with comments that may include such features Lt; / RTI > associated with the image.

First, the computerized personal identification training system 510 analyzes each feature associated with each image 500 and converts each such appropriate feature to an attribute value. For each of these values, the system 510 then sends each image 500 and associated property values to a crow sourcing provider, such as, for example, Amazon Mechanical Turk, And indicate their opinion on the degree of agreement of the relevant attribute values. Upon receipt of the crow sourcing results for each image-attribute value pair, the system 510 stores in the database 520 attribute values, which generally receive a high degree of match with the associated image.

Reference is now made to Fig. 8, which is a schematic flow diagram illustrating the training process of associating an adjective with an image. As shown in FIG. 8, the adjectives defining the face attributes are selected by the system from the list of adjectives to be trained, and one or more of the publicly available character search engines are preferably used to retrieve the images associated with the adjectives. In addition, one or more publicly available character search engines are preferably used to retrieve images associated with one or more translations of adjectives in various languages. For example, a list of adjectives can be edited by collecting adjectives from the dictionary.

A visual face detector is used to identify the searched image including the face image. Thereafter, crowd sourcing is preferably used to determine which of the facial images corresponds to an adjective by majority vote. The adjective and corresponding face image are then used to train the visual classifier as described below with respect to FIG. The visual classifier is then used to associate an adjective with an additional set of facial images, and crowd sourcing is further used to confirm the match of each additional set of adjectives and facial images, and the result is further used to train visual classifiers do. It is recognized that the training of additional cycles and visual classifiers of crowd sourcing can be used to further improve the accuracy of the visual classifier until the desired accuracy is reached. After training of the visual classifier, the classifier is added to a bank of attribute functions that can be used later by the system to classify the face images into adjectives that define facial attributes.

Reference is now made to Fig. 9, which is a schematic flow diagram illustrating the process of training a visual classifier. As shown in FIG. 9, for each adjective, the result of the crowd sourcing process described above with respect to FIG. 8 is used to generate two sets of images. The first "positive" collection includes images identified as corresponding to the adjectives, and the second "negative" collection includes images not identified as corresponding to the adjectives.

The images of both positive and negative sets are then normalized by compensating for the conversion of each image into a canonical image by varying the two-dimensional and three-dimensional alignment and varying illumination. Then, the normal image is transformed into canonical numerical vectors, and the classifier uses a supervised-classifier such as a support vector machine (SVM) to generate positive and negative numerical vectors From a training set that includes pairs of < RTI ID = 0.0 >

Reference is now made to FIG. 10, which is a schematic flow diagram illustrating a process for searching for adjectives associated with an image. As shown in FIG. 10, the image is first analyzed to detect and cut a face image that is part of the image. Then, the face image is transformed into a regular numerical vector by changing the two-dimensional and three-dimensional alignment and compensating the illuminance differently and normalizing the image. Thereafter, the bank of attribute functions described above with respect to FIG. 8 is applied as a numeric vector, and the value returned from each attribute function is recorded as a numeric vector representing the adjective associated with the face image.

Reference is now made to Fig. 11, which is a schematic flow diagram illustrating a process for retrieving an image from a pre-indexed database of images associated with one or more adjectives. As shown in FIG. 11, a textual query for an image having an adjective associated with the image is constructed first. Using Natural Language Processing (NLP), adjectives are extracted from character queries. The system then preferably retrieves the image from a preprocessed database of facial images that best matches the adjectives extracted from the query by using Latent Dirichlet Allocation (LDA). The retrieved facial image is ordered according to the correlation of the related numerical vectors and the adjectives extracted from the query, and the result of the ordered facial images is provided as a result of the system.

Reference is now made to Fig. 12, which is a schematic flow diagram illustrating a process for searching a face image similar to a first image. As shown in Fig. 12, the first image is first analyzed to detect and cut a face image that is part of the image. The face image is then transformed into a normalized numeric vector by varying the two-dimensional and three-dimensional pose-alignment and normalizing the image by compensating the illumination differently. Thereafter, the bank of attribute functions described above with respect to FIG. 8 is applied as a numeric vector, and the value returned from each attribute function is recorded as a numeric vector representing the adjective associated with the face image.

A pre-indexed database containing a numerical vector of images, such as a KD tree, may be a numerical vector representing an image closely matching a numerical vector of the first image, for example using a similarity-function, such as Euclidean distance Lt; / RTI >

It will be appreciated by those skilled in the art that the present invention is not limited by what has been particularly shown or described hereinabove. Rather, the scope of the invention includes both combinations and sub-combinations of the various features described above, as well as variations that can be made by those skilled in the art without departing from the prior art.

Claims (57)

An evaluation unit for evaluating a computerized facial image attribute face that assigns values representing facial images to a plurality of individual facial attributes of the facial image; And
And a computerized classification apparatus for classifying the face images according to the plurality of individual face attributes,
The evaluations are expressed as adjectives,
The evaluation apparatus comprises:
Wherein the evaluation device is a database that includes a plurality of types of storage face images and a plurality of types of storage estimates, each of the storage face images having at least some of the plurality of individual face attributes, The database having the evaluation expressed as an adjective associated with a face attribute; And
And an adjective-based comparison device for comparing the face image and the plurality of types of stored face images by comparing the plurality of individual face attributes of the face image and the plurality of types of stored face images with respect to the attribute face and the adjective face, ,
In response to the comparison, a face image that does not match the plurality of types of stored face images is stored in the database as one of the plurality of types of stored face images,
Further comprising a face model generation functionality operative to generate a face model corresponding to the face image. ≪ Desc / Clms Page number 21 > The method according to claim 1,
Further comprising facial attribute statistical reporting functionality for providing statistical information derived from the plurality of types of storage evaluations. ≪ RTI ID = 0.0 > 11. < / RTI > The method according to claim 1,
Wherein the adjective-based comparison device queries the database for an adjective face. 4. The method according to any one of claims 1 to 3,
Further comprising a computerized identification device operable to identify at least one storage face image corresponding to the output in response to an output from the computerized classification device, system. 5. The method of claim 4,
Wherein the computerized identification device is operative to generate a ranked list of stored face images corresponding to the output. 4. The method according to any one of claims 1 to 3,
Further comprising a social network interface for making information from the social network available to the evaluation device of the computerized facial image attribute facet. delete 5. The method of claim 4,
Wherein the computerized identification device uses the face model. A computerized facial image evaluation apparatus comprising: a step of assigning, to a plurality of individual facial attributes of a facial image, evaluations expressing a facial image;
Wherein the computerized classification apparatus classifies the face image according to the plurality of the individual face attributes;
Comparing the face image and the plurality of kinds of stored face images by comparing the plurality of individual face attributes of the face image and the stored face image with respect to the attribute face and the adjective face; And
And in response to the comparison, the computerized facial image attribute plane evaluation device includes a step of storing, in the database, one of the plurality of types of stored face images, the face image not matching the plurality of kinds of stored face images ,
Wherein the evaluations are represented by an adjective, wherein the evaluation device is a database comprising a plurality of types of stored face images and a plurality of types of storage evaluations, each of the stored face images having at least some of the plurality of individual face attributes, Wherein at least some of the individual face attributes have the database having the evaluation expressed as an adjective associated with the face attribute,
A computerized method for classifying a face image of a person, the computerized facial model generation apparatus further comprising a step of generating a face model corresponding to the face image. 10. The method of claim 9,
A computerized method for classifying a face image of a person, the computerized providing device also including a process of providing statistical information derived from a plurality of kinds of storage evaluations. 10. The method of claim 9,
Wherein the adjective-based comparison device queries the database for an adjective face. 12. The method according to any one of claims 9 to 11,
A computerized method for classifying a face image of a person, the computerized identification device further comprising the step of identifying at least one storage face image corresponding to an output of the sorting process. 13. The method of claim 12,
Wherein the identifying step is operable to generate a ranked list of stored face images corresponding to the output. 12. The method according to any one of claims 9 to 11,
A computerized method for classifying a face image of a person, the method further comprising the step of allowing the social network interface to make information from the social network available for the evaluation device of the computerized facial image attribute facet. delete 13. The method of claim 12,
Wherein the identifying step uses the face model to classify a human face image. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images