AU2011358100A1 - Systems and methods for image-to-text and text-to-image association - Google Patents

Systems and methods for image-to-text and text-to-image association Download PDF

Info

Publication number
AU2011358100A1
AU2011358100A1 AU2011358100A AU2011358100A AU2011358100A1 AU 2011358100 A1 AU2011358100 A1 AU 2011358100A1 AU 2011358100 A AU2011358100 A AU 2011358100A AU 2011358100 A AU2011358100 A AU 2011358100A AU 2011358100 A1 AU2011358100 A1 AU 2011358100A1
Authority
AU
Australia
Prior art keywords
facial
computerized
facial image
persons
attributes
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
AU2011358100A
Other versions
AU2011358100B2 (en
Inventor
Gil Hirsch
Eden Shochat
Yaniv Taigman
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Meta Platforms Inc
Original Assignee
Facebook Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Facebook Inc filed Critical Facebook Inc
Publication of AU2011358100A1 publication Critical patent/AU2011358100A1/en
Application granted granted Critical
Publication of AU2011358100B2 publication Critical patent/AU2011358100B2/en
Ceased legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5854—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using shape and object relationship
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5838—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Library & Information Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

A computerized system for classifying facial images of persons including a computerized facial image attribute-wise evaluator, assigning values representing a facial image to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives and a computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes.

Description

WO 2012/104830 PCT/IL2011/000287 SYSTEMS AND METHODS FOR IMAGE-TO-TEXT AND TEXT-TO-IMAGE ASSOCIATION 5 REFERENCE TO RELATED APPLICATIONS Reference is made to U.S. Provisional Patent Application Serial No. 61/439,021, filed February 3, 2011 and entitled "SYSTEMS AND METHODS FOR 10 IMAGE-TO-TEXT AND TEXT-TO-IMAGE ASSOCIATION", the disclosure of which is hereby incorporated by reference and priority of which is hereby claimed pursuant to 37 CFR 1.78(a) (4) and (5)(i). Reference is also made to the following patent application, owned by assignee, the disclosure of which is hereby incorporated by reference: 15 U.S. Patent Application Serial No.: 12/922,984. FIELD OF THE INVENTION 20 The present invention relates generally to image-to-text and text-to image association. BACKGROUND OF THE INVENTION 25 The following patents and patent publications are believed to represent the current state of the art: US Patent Nos.: 4,926,491; 5,164,992; 5,963,670; 6,292,575; 6,301,370; 6,819,783; 6,944,319; 6,990,217; 7,274,822 and 7,295,687; and US Published Patent Application Nos.: 2006/0253491; 2007/0237355 30 and 2009/0210491. 1 WO 2012/104830 PCT/IL2011/000287 SUMMARY OF THE INVENTION The present invention seeks to provide improved systems and 5 methodologies for image-to-text and text-to-image association. There is thus provided in accordance with a preferred embodiment of the present invention a computerized system for classifying facial images of persons including a computerized facial image attribute-wise evaluator, assigning values representing a facial image to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives 10 and a computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes. In accordance with a preferred embodiment of the present invention, the computerized facial attribute-wise evaluator includes a database including a multiplicity of stored values corresponding to a plurality of facial images, each of the facial images 15 having at least some of the plurality of discrete facial attributes, at least some of the discrete facial attributes having the values, represented by adjectives, associated therewith. Preferably, the system also includes facial attribute statistic reporting functionality providing statistical information derived from the multiplicity of stored 20 values. Preferably, the computerized facial attribute-wise evaluator includes a database including a multiplicity of stored facial images, and a multiplicity of stored values, each of the stored facial images having at least some of the plurality of discrete facial attributes, at least some of the discrete facial attributes having the values, 25 represented by adjectives, associated therewith, and an adjective-based comparator, comparing a facial image with the multiplicity of stored facial images by comparing the plurality of discrete facial attributes of the facial image, attribute- and adjective- wise with the multiplicity of stored facial images. Preferably, the adjective-based comparator queries the database in an adjective-wise manner. 30 Preferably, the system also includes a computerized identifier operative in response to an output from the computerized classifier for identifying at least one stored facial image corresponding to the output. Preferably, the computerized identifier 2 WO 2012/104830 PCT/IL2011/000287 is operative for generating a ranked list of stored facial images corresponding to said output. Preferably, the system also includes a social network interface for making available information from a social network to the computerized facial image 5 attribute-wise evaluator. Preferably, the system also includes face model generation functionality operative to generate a face model corresponding to the facial image. Preferably, the computerized identifier employs the face model. There is also provided in accordance with another preferred embodiment of the present invention a computerized method for classifying facial images of persons 10 including assigning values representing a facial image to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives, and classifying the facial image in accordance with the plural ones of the discrete facial attributes. In accordance with a preferred embodiment of the present invention, each of the facial images has at least some of the plurality of discrete facial attributes 15 and at least some of the discrete facial attributes have the values, represented by adjectives, associated therewith. Preferably, the method also includes providing statistical information derived from the multiplicity of stored values. Preferably, each of the stored facial images has at least some of the plurality of discrete facial attributes, and at least some of the discrete facial attributes 20 have the values, represented by adjectives, associated therewith, and the method preferably also includes comparing a facial image with a multiplicity of stored facial images by comparing the plurality of discrete facial attributes of the facial image, attribute- and adjective- wise with the multiplicity of stored facial images. Preferably, the comparing queries a database in an adjective-wise manner. 25 Preferably, the method also includes identifying at least one stored facial image corresponding to an output of the classifying. Preferably, the identifying is operative for generating a ranked list of stored facial images corresponding to the output. Preferably, the method also includes making available information from a social network to the computerized facial image attribute-wise evaluator. Preferably, the 30 method also includes face model generation operative to generate a face model. corresponding to the facial image. Preferably, the identifying employs the face model. 3 WO 2012/104830 PCT/IL2011/000287 There is further provided in accordance with yet another preferred embodiment of the present invention a system for registration of persons in a place including a facial image/person identification acquisition subsystem acquiring at least one facial image and at least one item of personal identification of a person, and a 5 computerized subsystem receiving the at least one facial image and the at least one item of personal identification of the person, the computerized subsystem including face model generation functionality operative to generate a face model corresponding to the at least one facial image and image-to-attributes mapping functionality operative to assign values represented by adjectives to a plurality of facial attributes of the facial 10 image, and a database which stores information and the values of facial attributes for a plurality of the persons. Preferably, the system also includes attributes-to-image mapping functionality operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image and thereby to identify a particular individual 15 utilizing the face model. Preferably, the computerized subsystem also includes a value combiner is operative to combine the face model and the collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a particular individual. Preferably, the system also includes a subsequent facial image 20 acquisition subsystem acquiring at least one facial image and supplying it to the computerized subsystem, and the computerized subsystem is preferably operative to create a face model corresponding to the subsequent facial image, assign values represented by adjectives to a plurality of facial attributes of the subsequent facial image, and identify a corresponding stored facial image and thereby the subsequent 25 facial image as a particular individual, at least one item of personal identification relating to whom is stored in the database. Preferably, the value combiner is employed to combine the face model and the collection of values corresponding to the subsequent facial image and thereby to identify the particular individual. Preferably, the at least one item of personal 30 identification of the person is obtained from pre-registration data. Preferably, the system also includes a social network interface for making available information from a social network to the computerized subsystem. 4 WO 2012/104830 PCT/IL2011/000287 Preferably, the facial image/person identification acquisition subsystem is operative for acquiring at least one facial image and at least one item of personal identification of a person other than a person interacting with the subsystem. Additionally or alternatively, the facial image/person identification acquisition subsystem is operative for acquiring at 5 least one facial image of an otherwise unidentified person other than a person interacting with the subsystem. Preferably, the system is embodied in a computerized facial image attribute-wise evaluator, assigning values representing a facial image to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives 10 and a computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes. There is further provided in accordance with yet another preferred embodiment of the present invention a system for recognizing repeated presence of persons in a place including a facial image/person identification acquisition subsystem 15 acquiring at least one facial image of a person, and a computerized subsystem receiving the at least one facial image, the computerized subsystem including face model generation functionality operative to generate a face model corresponding to the at least one facial image, and image-to-attributes mapping functionality operative to assign values represented by adjectives to a plurality of facial attributes of the facial image, 20 and a database which stores information and the values of facial attributes for a plurality of the persons. Preferably, the computerized subsystem also includes attributes-to-image mapping functionality operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image associated with a particular individual, 25 utilizing the face model. Preferably, the computerized subsystem also includes a value combiner is operative to combine the face model and the collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values. Preferably, the system also includes a subsequent facial image 30 acquisition subsystem acquiring at least one facial image and supplying it to the computerized subsystem, and the computerized subsystem is preferably operative to create a face model corresponding to the subsequent facial image, assign values 5 WO 2012/104830 PCT/IL2011/000287 represented by adjectives to a plurality of facial attributes of the subsequent facial image, and identify a corresponding stored facial image and thereby the subsequent facial image as being that of a particular individual, for recognizing repeated presence of that particular person. 5 Preferably, the value combiner is employed to combine the face model and the collection of values corresponding to the subsequent facial image thereby to recognize repeated presence of a person. Preferably, the system also includes a repeat presence statistics generator employing the face models and the collections of values for generate attribute-wise statistics regarding persons repeatedly present at a place. 10 Preferably, the system also includes a social network interface for making available information from a social network to the computerized subsystem. Preferably, the facial image/person identification acquisition subsystem is operative for acquiring at least one facial image and at least one item of personal identification of a person other than a person interacting with the subsystem. 15 Additionally or alternatively, the facial image/person identification acquisition subsystem is operative for acquiring at least one facial image of an otherwise unidentified person other than a person interacting with the subsystem. Preferably, the system is embodied in a computerized facial image attribute-wise evaluator, assigning values representing a facial image to plural ones of 20 discrete facial attributes of the facial image, the values being represented by adjectives, and a computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes. There is yet further provided in accordance with yet still another preferred embodiment of the present invention a method for generating a computerized 25 facial image attribute-wise evaluator, capable of assigning values, each represented by an adjective, to plural ones of discrete facial attributes of a facial image, the method including gathering a multiplicity of facial images, each having at least one facial image attribute, characterized by an adjective, associated therewith, and generating a function operative to receive a facial image to be evaluated and to utilize results of the gathering 30 for assigning values to plural ones of discrete facial attributes of the facial image to be evaluated, the values being represented by adjectives. 6 WO 2012/104830 PCT/IL2011/000287 Preferably, the gathering includes collecting a multiplicity of facial images, each having at least one facial image attribute, characterized by an adjective, associated therewith from publicly available sources, and employing crowdsourcing to enhance correspondence between adjectives and facial attributes appearing in the 5 multiplicity of facial images. Preferably, the crowdsourcing includes employing multiple persons who view ones of the multiplicity of facial images and the adjectives and indicate their views as to the degree of correspondence between the adjectives and the facial attributes in the ones of the multiplicity of images. Preferably, the values are numerical values. 10 There is also provided in accordance with another preferred embodiment of the present invention a system for recognizing user reaction to at least one stimulus including a computerized facial image attribute-wise evaluator, assigning values representing a facial image obtained at a time corresponding to user reaction to a stimulus to plural ones of discrete facial attributes of the facial image, the values being 15 represented by adjectives, and a computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes. Preferably, the system also includes a computerized attribute comparator comparing the plural ones of the discrete facial attributes prior to and following application of the at least one stimulus. 20 There is further provided in accordance with yet another preferred embodiment of the present invention a method for recognizing user reaction to at least one stimulus including assigning values representing a facial image obtained at a time corresponding to user reaction to a stimulus to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives, and classifying the facial 25 image in accordance with the plural ones of the discrete facial attributes. Preferably, the method also includes comparing the plural ones of the discrete facial attributes prior to and following application of the at least one stimulus. There is further provided in accordance with yet another preferred embodiment of the present invention a computerized system for classifying persons 30 including a relationship coefficient generator which generates relationship coefficients representing the probability of a person to be in a particular context at a particular time, 7 WO 2012/104830 PCT/IL2011/000287 and a computerized classifier which classifies the person in accordance with the plural ones of the relationship coefficients. Preferably, the context is one of a geographic location and an event. Preferably, the relationship coefficients include a value and a decay function. 5 Preferably, the decay function is a linear function. Alternatively, the decay function is an exponential function. Preferably, the context is one of a hierarchy of hierarchical contexts. Preferably, relationship coefficients of contexts of a hierarchy of contexts are interdependent. Preferably, the relationship coefficient generator is operative in a case 10 where multiple persons have been together in at least a first context to generate interdependent relationship coefficients between the multiple persons in a second context. Preferably, the system also includes a computerized classifier which classifies facial images in accordance with plural ones of discrete facial attributes. 15 8 WO 2012/104830 PCT/IL2011/000287 BRIEF DESCRIPTION OF THE DRAWINGS The present invention will be understood and appreciated more fully 5 from the following detailed description, taken in conjunction with the drawings in which: Figs. 1A, 1B and IC are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with a preferred embodiment of the present invention; 10 Figs. 2A and 2B are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with another preferred embodiment of the present invention; Figs. 3A and 3B are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with yet another 15 preferred embodiment of the present invention; Figs. 4A, 4B and 4C are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with yet another preferred embodiment of the present invention; Figs. 5A and 5B are simplified illustrations of an identification system 20 employing image-to-text and text-to-image association in accordance with yet another preferred embodiment of the present invention; Fig. 6 is a simplified illustration of a user satisfaction monitoring system employing image-to-text association in accordance with yet another preferred embodiment of the present invention; 25 Fig. 7 is a simplified illustration of an image/text/image database generation methodology useful in building a database employed in the systems of Figs. 1A -6; Fig. 8 is a simplified flow chart illustrating a training process for associating adjectives with images; 30 Fig. 9 is a simplified flow chart illustrating the process of training a visual classifier; 9 WO 2012/104830 PCT/IL2011/000287 Fig. 10 is a simplified flow chart illustrating a process for retrieving adjectives associated with an image; Fig. 11 is a simplified flow chart illustrating a process for retrieving images associated with one or more adjectives; and 5 Fig. 12 is a simplified flow chart illustrating a process for retrieving facial images similar to a first image. 10 10 WO 2012/104830 PCT/IL2011/000287 DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS Reference is now made to Figs. 1A, 1B and IC, which are simplified 5 illustrations of an identification system employing image-to-text and text-to-image association in accordance with a preferred embodiment of the present invention. The system of Figs. 1 A - 1 C preferably includes a computerized facial image attribute-wise evaluator, assigning values representing a facial image to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives, and a 10 computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes. As seen in Fig. 1A, on January 1, Mr. Jones, a customer of the AAA Department Store, enters the store and registers as a valued customer of the store at a registration stand 100. The registration stand preferably includes a computer 102 15 connected to a store computer network, and a digital camera 104 connected to computer 102. The valued customer registration process includes entering personal identification details of the customer, such as his full name, and capturing a facial image 108 of the customer by digital camera 104. Alternatively, personal identification details of the customer may be retrieved, for example, from a pre-existing personal social network 20 account of the customer. Alternatively, the customer may register as a valued location over the internet from a remote location. The personal identification details and facial image 108 are transmitted to a computerized person identification system 110 which preferably includes face model generation functionality 112, image-to-attributes mapping functionality 114, 25 attributes-to-image mapping functionality 116 and a value combiner 117. Computerized person identification system 110 also preferably includes a valued customer database 118 which stores registration details and values of facial attributes of all registered customers. It is appreciated that database 118 may be any suitable computerized information store. 30 Face model generation functionality 112 is operative to generate a face model 120 which corresponds to facial image 108. It is appreciated that face model generation functionality 112 may employ any suitable method of face model generation 11 WO 2012/104830 PCT/IL2011/000287 known in the art. As seen in Fig. 1 A, face model 120 generated by face model generation functionality 112 and corresponding to facial image 108 is stored in database 118 as one of the attributes of Mr. Jones. In accordance with a preferred embodiment of the present invention, 5 image-to-attributes mapping functionality 114 is operative to assign values represented by adjectives 122 to a plurality of facial attributes of facial image 108. The adjectives 122 representing the facial attributes may include, for example, adjectives describing hair color, nose shape, skin color, face shape, type and presence of or absence of facial hair. As seen in Fig. 1A, adjectives generated by attributes mapping functionality 114 10 which correspond to facial image 108 are stored in database 118 as values of attributes of Mr. Jones. Further in accordance with a preferred embodiment of the present invention, attributes-to-image mapping functionality 116 is operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image, 15 and thereby to identify a particular individual. Yet further in accordance with a preferred embodiment of the present invention, value combiner 117 preferably is operative to combine a face model and a collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a 20 particular individual. Turning now to Fig. IB, it is seen that on a later date, such as on January 17, a customer enters the AAA Department Store and a digital camera 150, mounted at the entrance to the store, captures a facial image 152 of the customer. Facial image 152 is transmitted to computerized person identification system 110 where a face model 160 25 corresponding to facial image 152 is preferably generated by face model generation functionality 112. Additionally, values 162 represented by adjectives are preferably assigned to a plurality of facial attributes of facial image 152 by image-to-attributes mapping functionality 114. As shown in Fig. 1B, face model 160 and adjectives 162 are preferably 30 combined by value combiner 117 into a combined collection of values, which is compared to the collections of values stored in database 118, and are found to match the face model and adjectives assigned to Mr. Jones, thereby identifying the person 12 WO 2012/104830 PCT/IL2011/000287 portrayed in facial image 152 captured by camera 150 as being Mr. Jones. It is appreciated that the collection of values combined by value combiner 117 and which are compared to the collections of values stored in database 118 may be any subset of face model 160 and adjectives 162. 5 Turning now to Fig. 1C, it is shown that for example, upon identifying the customer who has entered the store as Mr. Jones, who is a registered valued customer, the manager is notified by system 110 that a valued customer has entered the store, and the manager therefore approaches Mr. Jones to offer him a new product at a discount. 10 Reference is now made to Figs. 2A and 2B, which are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with another preferred embodiment of the present invention. As seen in Fig. 2A, on a particular day such as January 1, a customer of the AAA Department Store enters the store and a digital camera 200 mounted at the entrance to 15 the store captures a facial image 202 of the customer. Facial image 202 is transmitted to a computerized person identification system 210 which preferably, includes face model generation functionality 212, image-to-attributes mapping functionality 214, attributes to-image mapping functionality 216 and a value combiner 217. Computerized person identification system 210 also preferably includes a customer database 218, which 20 preferably stores values of facial attributes of all customers who have ever entered the store, and a visit counter 219 which preferably tracks the number of accumulated visits that each particular customer has made to the store. It is appreciated that database 218 may be any suitable computerized information store. Face model generation functionality 212 is operative to generate a face 25 model 220, which corresponds to facial image 202. It is appreciated that face model generation functionality 212 may employ any suitable method of face model generation known in the art. As seen in Fig. 2A, face model 220 generated by face model generation functionality 212 and corresponding to facial image 202 is stored in database 218 as one of the attributes of the customer of facial image 202. 30 In accordance with a preferred embodiment of the present invention, image-to-attributes mapping functionality 214 is operative to assign values represented by adjectives 222 to a plurality of facial attributes of facial image 202. The adjectives 13 WO 2012/104830 PCT/IL2011/000287 222 representing the facial attributes may include, for example, adjectives describing age group, gender, ethnicity, face shape, mood and general appearance. Further in accordance with a preferred embodiment of the present invention, attributes-to-image mapping functionality 216 is operative to utilize a 5 collection of values of facial attributes to identify a corresponding stored facial image, and thereby to identify a particular individual. It is appreciated that the collection of values may also include non-physical characteristics of the customer's appearance such as clothing type and color which may be used to identify an individual within a short period of time in a case where current values of facial attributes are not available. 10 Yet further in accordance with a preferred embodiment of the present invention, value combiner 217 preferably is operative to combine a face model and a collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a particular individual. 15 As seen in Fig. 2A, face model 220 and adjectives 222 are preferably combined by value combiner 217 into a combined collection of values, which is compared to the collections of values stored in database 218, and are found to match the face model and adjectives corresponding to a returning customer. Therefore, the visit counter 219 of the customer is incremented. It is appreciated that the collection of .20 values combined by value combiner 217 and which are compared to the collections of values stored in database 218 may be any subset of face model 220 and adjectives 222. Alternatively, if the combined collection of values generated by value combiner 217 is not found to match any of the collections of values stored in database 218, the combined collection of values generated by value combiner 217 and facial 25, image 202 are preferably stored in database 218 as representing a new customer, and the counter 219 of the new customer is initialized to 1. Turning now to Fig. 2B, it is shown that at closing time, such as at 5:00 PM on January 1, the manager of the store preferably receives a first report 230 from system 210 which includes a segmentation of customers who have entered the store over 30 the course of the January 1. The segmentation may be according to any of the adjectives stored in database 218, such as gender, age group, ethnicity and mood. Report 230 also 14 WO 2012/104830 PCT/IL2011/000287 preferably includes information regarding the number of previous visits that were made to the store by the customers of January 1. Additionally, the manager of the store may also receive a second report 234 from system 210 which includes a segmentation of returning customers who have 5 entered the store over the course of the January 1. The segmentation may be according to any of the adjectives stored in database 218, such as gender, age group, ethnicity and mood. It is appreciated that reports 230 and 234 may be useful, for example, for planning targeted marketing campaigns, or for evaluating the success of previously executed marketing campaigns. 10 Reference is now made to Figs. 3A and 3B, which are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with yet another preferred embodiment of the present invention. As seen in Fig. 3A, on a particular day such as January 1, a customer of the AAA Department Store enters the store and browses merchandise in the store's toy 15 department. A digital camera 250 mounted in the toy department captures a facial image 252 of the customer. As shown in Fig. 3A, additional digital cameras are preferably mounted throughout the various departments of the store. Facial image 252 is transmitted to a computerized person identification system 260 which includes face model generation functionality 262, image-to-attributes 20 mapping functionality 264, attributes-to-image mapping functionality 266 and a value combiner 267. Computerized person identification system 260 also preferably includes a customer database 268, which preferably stores values of facial attributes of all customers who have entered the store during the day, and information indicating which of the store's departments each customer visited. It is appreciated that database 268 may 25 be any suitable computerized information store. Face model generation functionality 262 is operative to generate a face model 270, which corresponds to facial image 252. It is appreciated that face model generation functionality 262 may employ any suitable method of face model generation known in the art. As seen in Fig. 3A, face model 270 generated by face model 30 generation functionality 262 and corresponding to facial image 252 is stored in database 268 as one of the attributes of the customer of facial image 25.2. 15 WO 2012/104830 PCT/IL2011/000287 In accordance with a preferred embodiment of the present invention, image-to-attributes mapping functionality 264 is operative to assign values represented by adjectives 272 to a plurality of facial attributes of facial image 252. The adjectives 272 representing the facial attributes may include, for example, adjectives describing 5 age group, gender, ethnicity, face shape, mood and general appearance. As seen in Fig. 3A, adjectives generated by attributes mapping functionality 264 which correspond to facial image 252 are stored in database .268 as values of attributes of the customer of facial image 252. Further in accordance with a preferred embodiment of the present 10 invention, attributes-to-image mapping functionality 266 is operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image, and thereby to identify a particular individual. It is appreciated that the collection of values may also include non-physical characteristics of the customer's appearance such as clothing type and color which may be used to identify an individual within a short 15 period of time in a case where current values of facial attributes are not available. Yet further in accordance with a preferred embodiment of the present invention, value combiner 267 preferably is operative to combine a face model and a collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a 20 particular individual. Additionally, system 260 records the department which the customer has visited in database 268 as being the toys department. Turning now to Fig. 3B, it is shown that at closing time, such as at 5:00 PM on January 1, the manager of the store preferably receives a report 280 from system 25 260 which includes a segmentation of customers who have entered the store's toy department over the course of the January 1. The segmentation may be according to any of the adjectives stored in database 268, such as gender, age group, ethnicity and mood. It is appreciated that report 280 may be useful, for example, for planning targeted marketing campaigns, or for evaluating the success of previously executed marketing 30 campaigns. Reference is now made to Figs. 4A, 4B and 4C, which are simplified illustrations of an identification system employing image-to-text and text-to-image 16 WO 2012/104830 PCT/IL2011/000287 association in accordance with yet another preferred embodiment of the present invention. As shown in Fig. 4A, on January 1, a potential attendee registers to attend the florists' annual conference, preferably via a computer 300. As part of the registration process, the potential attendee is preferably prompted to enter personal identification 5 details, such as his full name, and to upload at least one facial image 302 of himself. Alternatively, the potential attendee may choose to import personal identification details and one or more facial images, for example, from a pre-existing personal social network account. The personal identification details and facial image 302 are transmitted 10 to a computerized conference registration system 310 which preferably includes face model generation functionality 312, image-to-attributes mapping functionality 314, attributes-to-image mapping functionality 316 and a value combiner 317. Computerized conference registration system 310 also preferably includes a database 318 which stores registration details and values of facial attributes of all registered attendees. It is 15 appreciated that database 318 may be any suitable computerized information store. Face model generation functionality 312 is operative to generate a face model 320, which corresponds to facial image 302. It is appreciated that face model generation functionality 312 may employ any suitable method of face model generation known in the art. As seen in Fig. 4A, face model 320 generated by face model 20 generation functionality 312 and corresponding to facial image 302 is stored in database 318 as one of the attributes of potential attendee Mr. Jones. In accordance with a preferred embodiment of the present invention, image-to-attributes mapping functionality 314 is operative to assign values represented by adjectives 322 to a plurality of facial attributes of facial image 308. The adjectives 25 representing the facial attributes may include, for example, adjectives describing hair color, nose shape, skin color, face shape, type and presence of or absence of facial hair. As seen in Fig. 4A, adjectives generated by attributes mapping functionality 314, which correspond to facial image 302 are stored in database 318 as values of attributes of potential attendee Mr. Jones. 30 Further in accordance with a preferred embodiment of the present invention, attributes-to-image mapping functionality 316 is operative to utilize a 17 WO 2012/104830 PCT/IL2011/000287 collection of values of facial attributes to identify a corresponding stored facial image, and thereby to identify a particular individual. Yet further in accordance with a preferred embodiment of the present invention, value combiner 317 preferably is operative to combine a face model and a 5 collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a particular individual. Turning now to Fig. 4B, it is seen that on .a later date, such as on January 17, an attendee enters the florists' annual conference and approaches a registration 10 booth 330 on the conference floor. Registration booth 330 includes a digital camera 332 which captures a facial image 334 of the attendee. Facial image 334 is transmitted computerized conference registration system 310 where a face model 340 corresponding to facial image 334 is preferably generated by face model generation functionality 312. Additionally, values 342, represented by adjectives, are preferably assigned to a 15 plurality of facial attributes of facial image 334 by image-to-attributes mapping functionality 314. As shown in Fig. 4B, face model 340 and values 342 are preferably combined by value combiner 317 into a combined collection of values, which is compared to the collections of values stored in database 318, and are found to match the 20 face model and values assigned to Mr. Jones, thereby identifying the person portrayed in facial image 334 captured by camera 332 as being Mr. Jones. It is appreciated that the collection of values combined by value combiner 317 and which are compared to the collections of values stored in database 318 may be any subset of face model 340 and adjectives 342. Upon being identified as Mr. Jones, the attendee's registration is 25 completed and the attendee is welcomed by the conference staff. Turning now to Fig. 4C, it is shown that while attending the conference, attendees who wish to be introduced to other attendees, allow other attendees to capture a facial image 350 of them, using, for example, a digital camera embedded in a mobile communicator device 352. Mobile communicator devices 352 of conference attendees 30 are granted access to computerized conference registration system 310 via a computer network. It is appreciated that the computer network may be, for example, a local computer network or the internet. 18 WO 2012/104830 PCT/IL2011/000287 Additionally or alternatively, an attendee may access computerized conference registration system 310 to register new, currently unregistered attendees to the conference, by capturing a facial image of the new attendee and transmitting the facial image, preferably together with associated personal identification information, to 5 registration system 310. Upon capturing image 350 of a conference attendee, mobile communicator device 352 transmits image 350 over the computer network to computerized conference registration system 310 where a face model 360 corresponding to facial image 350 is preferably generated by face model generation functionality 312. 10 Additionally, values 362 represented by adjectives are preferably assigned to a plurality of facial attributes of facial image 350 by image-to-attributes mapping functionality 314. As shown in Fig. 4C, face model 360 and values 362 are combined by value combiner 317 into a combined collection of values, which is compared to the 15 collections of values stored in database 318, and are found to match the face model and values assigned to Mr. Jones, thereby identifying the person portrayed in facial image 350 captured by mobile communicator device 352 as being Mr. Jones. It is appreciated that the collection of values combined by value combiner 317 and which are compared to the collections of values stored in database 318 may be any subset of face model 360 20 and adjectives 362. Notification of the identification of the attendee portrayed in image 350 as Mr. Jones is transmitted by computerized conference registration system 310 back to mobile communicator device 352, which notification enables the operator of mobile communicator device 352 to know that he is approaching Mr. Jones. Reference is now made to Figs. 5A and 5B, which are simplified 25 illustrations of an identification system employing image-to-text and text-to-image association in accordance with yet another preferred embodiment of the present invention. In the embodiment of Figs. 5A and 5B, a relationship coefficient which measures the relationship between a person and a context is employed. The context may be, for example, a geographic location or an event, and the relationship coefficient 30 comprises a value and a predefined decay function. A single person may have a relationship coefficient with multiple contexts simultaneously. The relationship 19 WO 2012/104830 PCT/IL2011/000287 coefficient can be used, for example, to predict the probability of a person being at a given location at a particular time. The decay function may be any mathematical function. For example, the decay function for a geographical location may be a linear function, representing the 5 tendency of a person to gradually and linearly distance himself from the location over time. The decay function for a one-time event may be, for example, an exponential decay function. While a person is within a particular context, the current value of the generated relationship coefficient between the person and the context is set to be high. 10 Each time the person is repeatedly sighted within the context, the value of the relationship coefficient is increased, potentially in an exponential manner. It is appreciated that contexts may be hierarchical. For example, a geographic location may be within a larger geographical area such as a city or a country. Therefore, a person who has a relationship coefficient with a particular geographic 15 location will also have a lower relationship coefficient with all other geographical locations hierarchical thereto, which decreases as a function of the distance between the particular geographic location and the related hierarchical geographic locations. It is also appreciated that relationship coefficient of different people may be at least partially interdependent. For example, a first person who has been sighted 20 together with a second person at multiple locations at multiple times will be assigned a relatively high relationship coefficient to a new location where the second person has been sighted. As seen in Fig. 5A, on a particular day such as January 1, 2011, a diner dines at Caf6 Jaques which is close proximity to the Eiffel Tower in Paris, France. A 25 friend of the diner captures a facial image 400 of the diner using a digital camera which is part of a handheld mobile device 402 and registers the sighting of the diner by transmitting facial image 400 together with an associated time and location over the internet to a computerized person identification system 410. The location may be provided, for example, by a GPS module provided with device 402. Alternatively, the 30 location may be retrieved, for example, from a social network. Using the associated time and location, a relationship coefficient which relates the diner to the location is generated as described hereinabove. 20 WO 2012/104830 PCT/IL2011/000287 Computerized person identification system 410 includes face model generation functionality 412, image-to-attributes mapping functionality 414, attributes to-image mapping functionality 416 and a value combiner 417. Computerized person identification system 410 also preferably includes a sightings database 418 which 5 preferably stores values of facial attributes of all persons who have been sighted and registered, together with an associated time and location. It is appreciated that database 418 may be any suitable computerized information store. Face model generation functionality 412 is operative to generate a face model 420, which corresponds to facial image 400. It is appreciated that face model 10 generation functionality 422 may employ any suitable method of face model generation known in the art. As seen in Fig. 5A, face model 420 generated by face model generation functionality 412 and corresponding to facial image 400 is stored in database 418 as one of the attributes of the individual of facial image 400. In accordance with a preferred embodiment of the present invention, 15 image-to-attributes mapping functionality 414 is operative to assign values represented by adjectives 422 to a plurality of facial attributes of facial image 400. The adjectives 422 representing the facial attributes may include, for example, adjectives describing age group, gender, ethnicity, face shape, mood and general appearance. As seen in Fig. 5A, adjectives generated by attributes mapping functionality 414 which correspond to 20 facial image 400 are stored in database 418 as values of attributes of the individual of facial image 400. Additionally, the time and location associated with facial image 400 are also stored in database 418. Further in accordance with a preferred embodiment of the present invention, attributes-to-image mapping functionality 416 is operative to utilize a 25 collection of values of facial attributes to identify a corresponding stored facial image, and thereby to identify a particular individual. It is appreciated that the collection of values may also include non-physical characteristics of the customer's appearance such as clothing type and color which may be used to identify an individual within a short period of time in a case where current values of facial attributes are not available. 30 Yet further in accordance with a preferred embodiment of the present invention, value combiner 417 preferably is operative to combine a face model and a collection of values of facial attributes into a combined collection of values which can 21 WO 2012/104830 PCT/IL2011/000287 be matched to a corresponding stored collection of values, and thereby to identify a particular individual. Turning now to Fig. 5B, it is shown that on a later date, such as on February 1, 2011, a diner dines at Caf6 Jaques which is in close proximity to the Eiffel 5 Tower in Paris, France. A bystander captures a facial image 450 of the diner using a digital camera which is part of a handheld mobile device 452 and registers the sighting of the diner by transmitting facial image 450 together with an associated time and location over the internet to a computerized person identification system 410 where a face model 460, corresponding to facial image 450, is preferably generated by face 10 model generation functionality 412. Additionally, values 462 represented by adjectives are preferably assigned to a plurality of facial attributes of facial image 450 by image to-attributes mapping functionality 414. As shown in Fig. 5B, face model 460, values 462 'and the time and location associated with facial image 450 are preferably combined by value combiner 15 417 into a combined collection of values, which is compared to the collections of values stored in database 418, and are found to match the combined values assigned to the diner who was last seen at the Eiffel Tower on January 1, 2011. It is appreciated that the collection of values combined by value combiner 417 and which are compared to the collections of values stored in database 418 may be any subset of face model 460 and 20 adjectives 462. Notification of the identification of the diner portrayed in image 450 is transmitted over the internet by computerized person identification system 410 back to mobile communicator device 452. It is a particular feature of this embodiment of the present invention that the relationship coefficient which relates the diner to the location may also be used as an 25 attribute value which increases the reliability of the identification of the diner. It is a particular feature of the present embodiment of the current invention that the combination of the values of the facial attributes associated with a facial image together with additional information such as a particular location frequented by an individual is operative to more effectively identify individuals at the 30 particular location or at related locations, such as at other locations which are in close proximity to the particular location. 22 WO 2012/104830 PCT/IL2011/000287 It is another particular feature of the present embodiment of the current invention that identification of individuals according to the present embodiment of the current invention is not limited to precise identification of particular individuals based on personal identification information such as first and last name, but rather also 5 includes identification of individuals according by facial attributes and aggregating behavioral information pertaining to the individuals. Reference is now made to Fig. 6, which is a simplified illustration of a user satisfaction monitoring system employing image-to-text association in accordance with yet another preferred embodiment of the present invention. As shown in Fig. 6, a 10 viewer uses a multimedia viewing device 480 to view computerized content 482. It is appreciated that device 480 may be, for example, a television device or a computer. Content 482 may be, for example, a video clip, a movie or an advertisement. A digital camera 484 connected to multimedia viewing device 480 preferably captures a facial image 486 of the viewer at predefined intervals such as, for 15 example, every few seconds, and preferably transmits images 486 over the internet to an online computerized content satisfaction monitoring system 490. Alternatively, images 486 may be monitored, stored and analyzed by suitable functionality embedded in device 480. Preferably, system 490 includes image-to-attributes mapping 20 functionality 492 and a viewer expressions database 494. It is appreciated that database 494 may be any suitable computerized information store. In accordance with a preferred embodiment of the present invention, image-to-attributes mapping functionality 492 is operative to assign a value represented by an adjective 496 to the expression of the viewer as captured in facial images 486, and 25 to store adjectives 496 in database 494. Adjectives 496 may include, for example, "happy", "sad", "angry", "content" and "indifferent". It is appreciated that adjectives 496 stored in database 494 may be useful, for example, for evaluating the effectiveness of content 482. Reference is now made to Fig. 7, which is a simplified illustration of an 30 image/text/image database generation methodology useful in building a database employed in the systems of Figs. IA - 6. As shown in Fig. 7, a plurality of images 500 are collected from an image repository 502 which is publicly available on the internet, 23 WO 2012/104830 PCT/IL2011/000287 by a computerized person identification training system 510. Image repository 502 may be, for example, a publicly available social network or textual search engine which associates text with images appearing on the same page as the images or on one or more nearby pages. Preferably, one or more associated characteristics are provided by the 5 image repository with each of images 500. The characteristics may include, for example, a name, age or age group, gender, general appearance and mood, and are generally subjective and are associated with the images by the individuals who have publicized the images or by individuals who have tagged the publicized images with comments which may include such characteristics. 10 Computerized person identification training system 510 first analyzes each of the characteristics associated with each of images 500 and translates each such suitable characteristic to an attribute value. For each such value, system 510 then sends each of images 500 and its associated attribute value to a crowdsourcing provider, such as Amazon Mechanical Turk, where a plurality of individuals voice their opinion as to 15 the level of correspondence of each image with its associated attribute value. Upon receiving the crowdsourcing results for each image-attribute value pair, system 510 stores those attribute values which received a generally high correspondence level with their associated image in a database 520. Reference is now made to Fig. 8, which is a simplified flow chart 20 illustrating a training process for associating adjectives with images. As seen in Fig. 8, an adjective defining a facial attribute is chosen by the system from a list of adjectives to be trained, and one or more publicly available textual search engines are preferably employed to retrieve images which are associated with the adjective. Additionally, one or more publicly available textual search engines are preferably employed to retrieve 25 images which are associated with one or more translations of the adjective in various languages. The list of adjectives may be compiled, for example, by collecting adjectives from a dictionary. A visual face detector is employed to identify those retrieved images which include a facial image. Crowdsourcing is then preferably employed to ascertain, 30 based on a majority vote, which of the facial images correspond to the adjective. The adjective and corresponding facial images are then used to train a visual classifier, as described hereinbelow with regard to Fig. 9. The visual classifier is then employed to 24 WO 2012/104830 PCT/IL2011/000287 associate the adjective with an additional set of facial images, and crowdsourcing is further employed to ascertain the level of correspondence of each of the additional set of facial images with the adjective, the results of which are used to further train the visual classifier. It is appreciated that additional cycles of crowdsourcing and training of the 5 visual classifier may be employed to further refine the accuracy of the visual classifier, until a desired level of accuracy is reached. After the training of the visual classifier, the classifier is added to a bank of attribute functions which can later be used by the system to classify facial images by adjectives defining facial attributes. Reference is now made to Fig. 9, which is a simplified flow chart 10 illustrating the process of training a visual classifier. As shown in Fig. 9, for each adjective, the results of the crowdsourcing process described hereinabove with regard to Fig. 8 are employed to generate two collections of images. A first, "positive" collection includes images which have been ascertained to correspond to the adjective, and a second, "negative" collection includes images which have not been ascertained to 15 correspond to the adjective. The images of both the positive and the negative collection are then normalized to compensate for varying 2- dimensional and 3-dimensional alignment and differing illumination, thereby transforming each of the images into a canonical image. The canonical images are then converted into canonical numerical vectors, and a 20 classifier is learned from a training set comprising of pairs of positive and negative numerical vectors using a supervised-classifier, such as a Support Vector Machine (SVM). Reference is now made to Fig. 10, which is a simplified flow chart illustrating a process for retrieving adjectives associated with an image. As shown in 25 Fig. 10, the image is first analyzed to detect and crop a facial image which is a part of the image. The facial image is then converted to a canonical numerical vector by normalizing the image to compensate for varying 2-dimensional and 3-dimensional pose-alignment and differing illumination. The bank of attribute functions described hereinabove with regard to Fig. 8 is then applied to the numerical vector, and the value 30 returned from each attribute function is recorded in a numerical vector which represents the adjectives associated with the facial image. 25 WO 2012/104830 PCT/IL2011/000287 Reference is now made to Fig. 11, which is a simplified flow chart illustrating a process for retrieving images from a pre-indexed database of images, which are associated with one or more adjectives. As shown in Fig. 11, a textual query for images having adjectives associated therewith is first composed. Using Natural 5 Language Processing (NLP), adjectives are extracted from the textual query. The system then retrieves images from a previously processed database of facial images which are best-matched to the adjectives extracted from the query, preferably by using Latent Dirichlet Allocation (LDA). The retrieved facial images are ordered by the level of correlation of their associated numerical vectors to the adjectives extracted from the 10 query, and the resulting ordered facial images are provided as output of the system. Reference is now made to Fig. 12, which is a simplified flow chart illustrating a process for retrieving facial images which are similar to a first image. As shown in Fig. 12, the first image is first analyzed to detect and crop a facial image which is a part of the image. The facial image is then converted to a canonical numerical 15 vector by normalizing the image to compensate for varying 2-dimensional and 3 dimensional pose-alignment and differing illumination. The bank of attribute functions described hereinabove with regard to Fig. 8 is then applied to the numerical vector, and the value returned from each attribute function is recorded in a numerical vector which represents the adjectives associated with the facial image. 20 A previously indexed database comprising numerical vectors of images, such as a KD tree, is searched using a similarity-function, such as Euclidian distance, to find a collection of numerical vectors which represent images which closely match the numerical vector of the first image. It will be appreciated by persons skilled in the art that the present 25 invention is not limited by what has been particularly shown and described hereinabove. Rather the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove as well as modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not in the prior art. 30 26

Claims (57)

1. A computerized system for classifying facial images of persons 5 comprising: a computerized facial image attribute-wise evaluator, assigning values representing a facial image to plural ones of discrete facial attributes of said facial image, said values being represented by adjectives; and a computerized classifier which classifies said facial image in accordance 10 with said plural ones of said discrete facial attributes.
2. A computerized system for classifying facial images of persons according to claim 1 and wherein said computerized facial attribute-wise evaluator comprises: 15 a database comprising a multiplicity of stored values corresponding to a plurality of facial images, each of said facial images having at least some of said plurality of discrete facial attributes, at least some of said discrete facial attributes having said values, represented by adjectives, associated therewith. 20
3. A computerized system for classifying facial images of persons according to claim 2 and also comprising: facial attribute statistic reporting functionality providing statistical information derived from said multiplicity of stored values. 25
4. A computerized system for classifying facial images of persons according to claim 1 and wherein said computerized facial attribute-wise evaluator comprises: a database comprising a multiplicity of stored facial images, and a multiplicity of stored values, each of said stored facial images having at least some of 30 said plurality of discrete facial attributes, at least some of said discrete facial attributes having said values, represented by adjectives, associated therewith; and 27 WO 2012/104830 PCT/IL2011/000287 an adjective-based comparator, comparing a facial image with said multiplicity of stored facial images by comparing said plurality of discrete facial attributes of said facial image, attribute- and adjective- wise with said multiplicity of stored facial images. 5
5. A computerized system for classifying facial images of persons according to claim 4 and wherein said adjective-based comparator queries said database in an adjective-wise manner. 10
6. A computerized system for classifying facial images of persons according to any of claims 1 - 5 and also comprising a computerized identifier operative in response to an output from said computerized classifier for identifying at least one stored facial image corresponding to said output. 15
7. A computerized system for classifying facial images of persons according to claim 6 and wherein said computerized identifier is operative for generating a ranked list of stored facial images corresponding to said output.
8. A computerized system for classifying facial images of persons 20 according to any of claims 1 - 7 and also comprising a social network interface for making available information from a social network to said computerized facial image attribute-wise evaluator.
9. A computerized system for classifying facial images of persons 25 according to any of claims 1 - 8 and also comprising face model generation functionality operative to generate a face model corresponding to said facial image.
10. A computerized system for classifying facial images of persons according to claims 6 and 9 and where said computerized identifier employs said face 30 model. 28 WO 2012/104830 PCT/IL2011/000287
11. A computerized method for classifying facial images of persons comprising: assigning values representing a facial image to plural ones of discrete facial attributes of said facial image, said values being represented by adjectives; and 5 classifying said facial image in accordance with said plural ones of said discrete facial attributes.
12. A computerized method for classifying facial images of persons according to claim 11 and wherein each of said facial images has at least some of said 10 plurality of discrete facial attributes and at least some of said discrete facial attributes have said values, represented by adjectives, associated therewith.
13. A computerized method for classifying facial images of persons according to claim 12 and also comprising: 15 providing statistical information derived from said multiplicity of stored values.
14. A computerized method for classifying facial images of persons according to claim 11 and wherein: 20 each of said stored facial images has at least some of said plurality of discrete facial attributes, and at least some of said discrete facial attributes have said values, represented by adjectives, associated therewith; and also comprising: comparing a facial image with a multiplicity of stored facial images by comparing said plurality of discrete facial attributes of said facial image, attribute- and 25 adjective- wise with said multiplicity of stored facial images.
15. A computerized method for classifying facial images of persons according to claim 14 and wherein said comparing queries a database in an adjective wise manner. 30 29 WO 2012/104830 PCT/IL2011/000287
16. A computerized method for classifying facial images of persons according to any of claims 11 - 15 and also comprising identifying at least one stored facial image corresponding to an output of said classifying. 5
17. A computerized method for classifying facial images of persons according to claim 16 and wherein said identifying is operative for generating a ranked list of stored facial images corresponding to said output.
18. A computerized method for classifying facial images of persons 10 according to any of claims 11 - 17 and also comprising making available information from a social network to said computerized facial image attribute-wise evaluator.
19. A computerized method for classifying facial images of persons according to any of claims 11 - 18 and also comprising face model generation operative 15 to generate a face model corresponding to said facial image.
20. A computerized method for classifying facial images of persons according to claims 16 and 19 and where said identifying employs said face model. 20
21. A system for registration of persons in a place comprising: a facial image/person identification acquisition subsystem acquiring at least one facial image and at least one item of personal identification of a person; and a computerized subsystem receiving said at least one facial image and said at least one item of personal identification of said person, said computerized 25 subsystem comprising: face model generation functionality operative to generate a face model corresponding to said at least one facial image; and image-to-attributes mapping functionality operative to assign values represented by adjectives to a plurality of facial attributes 30 of said facial image; and a database which stores information and said values of facial attributes for a plurality of said persons. 30 WO 2012/104830 PCT/IL2011/000287
22. A system for registration of persons in a place according to claim 21 and wherein said computerized subsystem also comprises: attributes-to-image mapping functionality operative to utilize a 5 collection of values of facial attributes to identify a corresponding stored facial image and thereby to identify a particular individual utilizing said face model.
23. A system for registration of persons in a place according to claim 21 and 10 wherein said computerized subsystem also comprises: a value combiner is operative to -combine said face model and said collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a particular individual. 15
24. A system for registration of persons in a place according to either of claims 22 and 23 and also comprising: a subsequent facial image acquisition subsystem acquiring at least one facial image and supplying it to said computerized subsystem; and 20 wherein said computerized subsystem is operative to: create a face model corresponding to said subsequent facial image; assign values represented by adjectives to a plurality of facial 25 attributes of said subsequent facial image; and identify a corresponding stored facial image and thereby said subsequent facial image as a particular individual, at least one item of personal identification relating to whom is stored in said database. 30
25. A system for registration of persons in a place according to claims 23 and 24 wherein said value combiner is employed to combine said face model and said 31 WO 2012/104830 PCT/IL2011/000287 collection of values corresponding to said subsequent facial image and thereby to identify said particular individual.
26. A system for registration of persons in a place according to any of claims 5 21 - 25 and wherein said at least one item of personal identification of said person is obtained from pre-registration data.
27. A system for registration of persons in a place according to any of claims 21 - 26 and also comprising a social network interface for making available information 10 from a social network to said computerized subsystem.
28. A system for registration of persons in a place according to any of claims 24 - 27 and wherein said facial image/person identification acquisition subsystem is operative for acquiring at least one facial image and at least one item of personal 15 identification of a person other than a person interacting with said subsystem.
29. A system for registration of persons in a place according to any of claims 21 - 27 and wherein said facial image/person identification acquisition subsystem is operative for acquiring at least one facial image of an otherwise unidentified person 20 other than a person interacting with said subsystem.
30. A system for registration of persons in a place according to any of claims 21 - 29 which is embodied in: a computerized facial image attribute-wise evaluator, assigning values 25 representing a facial image to plural ones of discrete facial attributes of said facial image, said values being represented by adjectives; and a computerized classifier which classifies said facial image in accordance with said plural ones of said discrete facial attributes. 30
31. A system for recognizing repeated presence of persons in a place comprising: 32 WO 2012/104830 PCT/IL2011/000287 a facial image/person identification acquisition subsystem acquiring at least one facial image of a person; and a computerized subsystem receiving said at least one facial image, said computerized subsystem comprising: 5 face model generation functionality operative to generate a face model corresponding to said at least one facial image; and image-to-attributes mapping functionality operative to assign values represented by adjectives to a plurality of facial attributes of said facial image; and 10 a database which stores information and said values of facial attributes for a plurality of said persons.
32. A system for recognizing repeated presence of persons in a place according to claim 31 and wherein said computerized subsystem also comprises: 15 attributes-to-image mapping functionality operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image associated with a particular individual, utilizing said face model. 20
33. A system for recognizing repeated presence of persons in a place according to claim 31 and wherein said computerized subsystem also comprises: a value combiner is operative to combine said face model and said collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of 25 values.
34. A system for recognizing repeated presence of persons in a place according to either of claims 32 and 33 and also comprising: a subsequent facial image acquisition subsystem acquiring at least one 30 facial image and supplying it to said computerized subsystem; and wherein said computerized subsystem is operative to: 33 WO 2012/104830 PCT/IL2011/000287 create a face model corresponding to said subsequent facial image; assign values represented by adjectives to a plurality of facial attributes of said subsequent facial image; and 5 identify a corresponding stored facial image and thereby said subsequent facial image as being that of a particular individual, for recognizing repeated presence of that particular person.
35. A system for recognizing repeated presence of persons in a place 10 according to claims 33 and 34 wherein said value combiner is employed to combine said face model and said collection of values corresponding to said subsequent facial image thereby to recognize repeated presence of a person.
36. A system for recognizing repeated presence of persons in a place 15 according to any of claims 31 - 36 and also comprising: a repeat presence statistics generator employing said face models and said collections of values for generate attribute-wise statistics regarding persons repeatedly present at a place. 20
37. A system for recognizing repeated presence of persons in a place according to any of claims 31 - 36 and also comprising a social network interface for making available information from a social network to said computerized subsystem.
38. A system for recognizing repeated presence of persons in a place 25 according to any of claims 31 - 37 and wherein said facial image/person identification acquisition subsystem is operative for acquiring at least one facial image and at least one item of personal identification of a person other than a person interacting with said subsystem. 30
39. A system for recognizing repeated presence of persons in a place according to any of claims 31 - 37 and wherein said facial image/person identification 34 WO 2012/104830 PCT/IL2011/000287 acquisition subsystem is operative for acquiring at least one facial image of an otherwise unidentified person other than a person interacting with said subsystem.
40. A system for recognizing repeated presence of persons in a place 5 according to any of claims 31 - 39 which is embodied in: a computerized facial image attribute-wise evaluator, assigning values representing a facial image to plural ones of discrete facial attributes of said facial image, said values being represented by adjectives; and a computerized classifier which classifies said facial image in accordance 10 with said plural ones of said discrete facial attributes.
41. A method for generating a computerized facial image attribute-wise evaluator, capable of assigning values, each represented by an adjective, to plural ones of discrete facial attributes of a facial image, the method comprising: 15 gathering a multiplicity of facial images, each having at least one facial image attribute, characterized by an adjective, associated therewith; and generating a function operative to receive a facial image to be evaluated and to utilize results of said gathering for assigning values to plural ones of discrete facial attributes of said facial image to be evaluated, said values being represented by 20 adjectives.
42. A method for generating a computerized facial image attribute-wise evaluator according to claim 41 and wherein said gathering comprises: collecting a multiplicity of facial images, each having at least one facial 25 image attribute, characterized by an adjective, associated therewith from publicly available sources; and employing crowdsourcing to enhance correspondence between adjectives and facial attributes appearing in said multiplicity of facial images. 30
43. A method for generating a computerized facial image attribute-wise evaluator according to claim 42 and wherein said crowdsourcing comprises: 35 WO 2012/104830 PCT/IL2011/000287 employing multiple persons who view ones of said multiplicity of facial images and said adjectives and indicate their views as to the degree of correspondence between said adjectives and said facial attributes in said ones of said multiplicity of images. 5
44. A method for generating a computerized facial image attribute-wise evaluator according to any of claims 41 - 43 and wherein said values are numerical values. 10
45. A system for recognizing user reaction to at least one stimulus comprising: a computerized facial image attribute-wise evaluator, assigning values representing a facial image obtained at a time corresponding to user reaction to a stimulus to plural ones of discrete facial attributes of said facial image, said values 15 being represented by adjectives; and a computerized classifier which classifies said facial image in accordance with said plural ones of said discrete facial attributes.
46. A system for recognizing user reaction to at least one stimulus according 20 to claim 45 and also comprising a computerized attribute comparator comparing said plural ones of said discrete facial attributes prior to and following application of said at least one stimulus.
47. A method for recognizing user reaction to at least one stimulus 25 comprising: assigning values representing a facial image obtained at a time corresponding to user reaction to a stimulus to plural ones of discrete facial attributes of said facial image, said values being represented by adjectives; and classifying said facial image in accordance with said plural ones of said 30 discrete facial attributes. 36 WO 2012/104830 PCT/IL2011/000287
48. A method for recognizing user reaction to at least one stimulus according to claim 45 and also comprising comparing said plural ones of said discrete facial attributes prior to and following application of said at least one stimulus. 5
49. A computerized system for classifying persons comprising: a relationship coefficient generator which generates relationship coefficients representing the probability of a person to be in a particular context at a particular time; and a computerized classifier which classifies said person in accordance with 10 said plural ones of said relationship coefficients.
50. A computerized system for classifying persons according to claim 49 and wherein said context is one of a geographic location and an event. 15
51. A computerized system for classifying persons according to either of claims 49 and 50 and wherein said relationship coefficients comprise a value and a decay function.
52. A computerized system for classifying persons according to claim 51 and 20 wherein said decay function is a linear function.
53. A computerized system for classifying persons according to claim 51 and wherein said decay function is an exponential function. 25
54. A computerized system for classifying persons according to any of claims 49 - 53 and wherein said context is one of a hierarchy of hierarchical contexts.
55. A computerized system for classifying persons according to claim 51 and wherein relationship coefficients of contexts of a hierarchy of contexts are 30 interdependent. 37 WO 2012/104830 PCT/IL2011/000287
56. A computerized system for classifying persons according to any of claims 49 - 55 and wherein said relationship coefficient generator is operative in a case where multiple persons have been together in at least a first context to generate interdependent relationship coefficients between said multiple persons in a second 5 context.
57. A computerized system for classifying persons according to claim 49 and also comprising: a computerized classifier which classifies facial images in accordance 10 with plural ones of discrete facial attributes. 38
AU2011358100A 2011-02-03 2011-03-31 Systems and methods for image-to-text and text-to-image association Ceased AU2011358100B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201161439021P 2011-02-03 2011-02-03
US61/439,021 2011-02-03
PCT/IL2011/000287 WO2012104830A1 (en) 2011-02-03 2011-03-31 Systems and methods for image-to-text and text-to-image association

Publications (2)

Publication Number Publication Date
AU2011358100A1 true AU2011358100A1 (en) 2013-09-05
AU2011358100B2 AU2011358100B2 (en) 2016-07-07

Family

ID=46602131

Family Applications (1)

Application Number Title Priority Date Filing Date
AU2011358100A Ceased AU2011358100B2 (en) 2011-02-03 2011-03-31 Systems and methods for image-to-text and text-to-image association

Country Status (8)

Country Link
JP (1) JP5857073B2 (en)
KR (1) KR101649322B1 (en)
CN (1) CN103620590B (en)
AU (1) AU2011358100B2 (en)
BR (1) BR112013019907A2 (en)
CA (1) CA2826177C (en)
MX (1) MX345437B (en)
WO (1) WO2012104830A1 (en)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9158970B2 (en) 2012-11-16 2015-10-13 Canon Kabushiki Kaisha Devices, systems, and methods for visual-attribute refinement
US9626597B2 (en) 2013-05-09 2017-04-18 Tencent Technology (Shenzhen) Company Limited Systems and methods for facial age identification
CN104143079B (en) 2013-05-10 2016-08-17 腾讯科技（深圳）有限公司 The method and system of face character identification
US20170177927A1 (en) * 2014-02-17 2017-06-22 Nec Solution Innovators, Ltd. Impression analysis device, game device, health management device, advertising support device, impression analysis system, impression analysis method, and program recording medium
US10635672B2 (en) * 2015-09-02 2020-04-28 Oath Inc. Method and system for merging data
US10909779B2 (en) 2016-08-11 2021-02-02 Tekion Corp 3D vehicle model data capturing and retrieving for vehicle inspection, service and maintenance
CN106951825B (en) * 2017-02-13 2021-06-29 苏州飞搜科技有限公司 Face image quality evaluation system and implementation method
CN108009280B (en) * 2017-12-21 2021-01-01 Oppo广东移动通信有限公司 Picture processing method, device, terminal and storage medium
US11941044B2 (en) 2018-01-26 2024-03-26 Walmart Apollo, Llc Automatic personalized image-based search
KR102585358B1 (en) * 2021-08-09 2023-10-05 양주섭 Method and apparatus for analyzing behavior pattern of customer in unmanned shop using ai-based image processing model
CN113641857A (en) * 2021-08-13 2021-11-12 三星电子（中国）研发中心 Visual media personalized search method and device
KR102477338B1 (en) * 2022-02-23 2022-12-14 주식회사 블랙탠저린 Method, apparatus and program for providing facial mood diagnosis platform using image
CN117095083B (en) * 2023-10-17 2024-03-15 华南理工大学 Text-image generation method, system, device and storage medium

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7298931B2 (en) * 2002-10-14 2007-11-20 Samsung Electronics Co., Ltd. Image retrieval method and apparatus using iterative matching
EP1573658A1 (en) * 2002-12-13 2005-09-14 Koninklijke Philips Electronics N.V. Expression invariant face recognition
US7680330B2 (en) * 2003-11-14 2010-03-16 Fujifilm Corporation Methods and apparatus for object recognition using textons
JP2006221355A (en) * 2005-02-09 2006-08-24 Hitachi Ltd Monitoring device and monitoring system
US7809192B2 (en) * 2005-05-09 2010-10-05 Like.Com System and method for recognizing objects from images and identifying relevancy amongst images and information
JP4668680B2 (en) * 2005-05-17 2011-04-13 ヤマハ発動機株式会社 Attribute identification system and attribute identification dictionary generator
US7864989B2 (en) * 2006-03-31 2011-01-04 Fujifilm Corporation Method and apparatus for adaptive context-aided human classification
US7684651B2 (en) * 2006-08-23 2010-03-23 Microsoft Corporation Image-based face search
JP2011013732A (en) * 2009-06-30 2011-01-20 Sony Corp Information processing apparatus, information processing method, and program
US8670597B2 (en) * 2009-08-07 2014-03-11 Google Inc. Facial recognition with social network aiding

Also Published As

Publication number Publication date
KR20140058409A (en) 2014-05-14
CA2826177C (en) 2017-08-08
CA2826177A1 (en) 2012-08-09
JP2014511520A (en) 2014-05-15
KR101649322B1 (en) 2016-08-18
CN103620590B (en) 2018-10-09
WO2012104830A1 (en) 2012-08-09
MX345437B (en) 2017-01-31
CN103620590A (en) 2014-03-05
JP5857073B2 (en) 2016-02-10
MX2013008985A (en) 2014-02-27
BR112013019907A2 (en) 2016-10-11
AU2011358100B2 (en) 2016-07-07

Similar Documents

Publication Publication Date Title
AU2011358100B2 (en) Systems and methods for image-to-text and text-to-image association
US20210248631A1 (en) Conducting digital surveys that collect and convert biometric data into survey respondent characteristics
US7684651B2 (en) Image-based face search
US9558397B2 (en) Method and apparatus for automated analysis and identification of a person in image and video content
JP5383705B2 (en) Determining social relationships from personal photo collections
US8897485B2 (en) Determining an interest level for an image
US8873851B2 (en) System for presenting high-interest-level images
US9014510B2 (en) Method for presenting high-interest-level images
US9014509B2 (en) Modifying digital images to increase interest level
US20140002644A1 (en) System for modifying images to increase interestingness
CN103793697A (en) Identity labeling method of face images and face identity recognition method of face images
US11461630B1 (en) Machine learning systems and methods for extracting user body shape from behavioral data
CN108764007A (en) Based on OCR with text analysis technique to the measurement method of attention
Truong et al. Lifelogging retrieval based on semantic concepts fusion
CA3050456C (en) Facial modelling and matching systems and methods
JP2020067720A (en) Personal attribute estimation system, and information processing apparatus and information processing method using the same
KR20170141910A (en) System and method for making advertisement as automatically
KR102228873B1 (en) Construction system of criminal suspect knowledge network using public security information and Method thereof
KR20170141915A (en) System and method for making advertisement image
Mocanu et al. Indoor Localisation through Probabilistic Ontologies
Xu et al. A health management tool based smart phone
KR20150072212A (en) Figure identification apparatus using Social network service and method therefor
KR20240020787A (en) System and method for servicing preference-based user customized content

Legal Events

Date Code Title Description
FGA Letters patent sealed or granted (standard patent)
MK14 Patent ceased section 143(a) (annual fees not paid) or expired