Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
  • G06F16/5854—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using shape and object relationship
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/22—Indexing; Data structures therefor; Storage structures
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
  • G06F16/5838—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/95—Retrieval from the web
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
  • G06V40/172—Classification, e.g. identification

Definitions

• the present invention relates generally to image-to-text and text-to- image association.
• the present invention seeks to provide improved systems and methodologies for image-to-text and text-to-image association.
• a computerized system for classifying facial images of persons including a computerized facial image attribute-wise evaluator, assigning values representing a facial image to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives and a computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes.
• the computerized facial attribute-wise evaluator includes a database including a multiplicity of stored values corresponding to a plurality of facial images, each of the facial images having at least some of the plurality of discrete facial attributes, at least some of the discrete facial attributes having the values, represented by adjectives, associated therewith.
• the system also includes facial attribute statistic reporting functionality providing statistical information derived from the multiplicity of stored values.
• the computerized facial attribute-wise evaluator includes a database including a multiplicity of stored facial images, and a multiplicity of stored values, each of the stored facial images having at least some of the plurality of discrete facial attributes, at least some of the discrete facial attributes having the values, represented by adjectives, associated therewith, and an adjective-based comparator, comparing a facial image with the multiplicity of stored facial images by comparing the plurality of discrete facial attributes of the facial image, attribute- and adjective- wise with the multiplicity of stored facial images.
• the adjective-based comparator queries the database in an adjective-wise manner.
• the system also includes a computerized identifier operative in response to an output from the computerized classifier for identifying at least one stored facial image corresponding to the output.
• the computerized identifier is operative for generating a ranked list of stored facial images corresponding to said output.
• the system also includes a social network interface for making available information from a social network to the computerized facial image attribute-wise evaluator.
• the system also includes face model generation functionality operative to generate a face model corresponding to the facial image.
• the computerized identifier employs the face model.
• a computerized method for classifying facial images of persons including assigning values representing a facial image to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives, and classifying the facial image in accordance with the plural ones of the discrete facial attributes.
• each of the facial images has at least some of the plurality of discrete facial attributes and at least some of the discrete facial attributes have the values, represented by adjectives, associated therewith.
• the method also includes providing statistical information derived from the multiplicity of stored values.
• each of the stored facial images has at least some of the plurality of discrete facial attributes, and at least some of the discrete facial attributes have the values, represented by adjectives, associated therewith, and the method preferably also includes comparing a facial image with a multiplicity of stored facial images by comparing the plurality of discrete facial attributes of the facial image, attribute- and adjective- wise with the multiplicity of stored facial images.
• the comparing queries a database in an adjective- wise manner.
• the method also includes identifying at least one stored facial image corresponding to an output of the classifying.
• the identifying is operative for generating a ranked list of stored facial images corresponding to the output.
• the method also includes making available information from a social network to the computerized facial image attribute-wise evaluator.
• the method also includes face model generation operative to generate a face model corresponding to the facial image.
• the identifying employs the face model.
• a system for registration of persons in a place including a facial image/person identification acquisition subsystem acquiring at least one facial image and at least one item of personal identification of a person, and a computerized subsystem receiving the at least one facial image and the at least one item of personal identification of the person, the computerized subsystem including face model generation functionality operative to generate a face model corresponding to the at least one facial image and image-to-attributes mapping functionality operative to assign values represented by adjectives to a plurality of facial attributes of the facial image, and a database which stores information and the values of facial attributes for a plurality of the persons.
• the system also includes attributes-to-image mapping functionality operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image and thereby to identify a particular individual utilizing the face model.
• the computerized subsystem also includes a value combiner is operative to combine the face model and the collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a particular individual.
• the system also includes a subsequent facial image acquisition subsystem acquiring at least one facial image and supplying it to the computerized subsystem, and the computerized subsystem is preferably operative to create a face model corresponding to the subsequent facial image, assign values represented by adjectives to a plurality of facial attributes of the subsequent facial image, and identify a corresponding stored facial image and thereby the subsequent facial image as a particular individual, at least one item of personal identification relating to whom is stored in the database.
• a subsequent facial image acquisition subsystem acquiring at least one facial image and supplying it to the computerized subsystem
• the computerized subsystem is preferably operative to create a face model corresponding to the subsequent facial image, assign values represented by adjectives to a plurality of facial attributes of the subsequent facial image, and identify a corresponding stored facial image and thereby the subsequent facial image as a particular individual, at least one item of personal identification relating to whom is stored in the database.
• the value combiner is employed to combine the face model and the collection of values corresponding to the subsequent facial image and thereby to identify the particular individual.
• the at least one item of personal identification of the person is obtained from pre-registration data.
• the system also includes a social network interface for making available information from a social network to the computerized subsystem.
• the facial image/person identification acquisition subsystem is operative for acquiring at least one facial image and at least one item of personal identification of a person other than a person interacting with the subsystem.
• the facial image/person identification acquisition subsystem is operative for acquiring at least one facial image of an otherwise unidentified person other than a person interacting with the subsystem.
• the system is embodied in a computerized facial image attribute-wise evaluator, assigning values representing a facial image to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives and a computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes.
• a system for recognizing repeated presence of persons in a place including a facial image/person identification acquisition subsystem acquiring at least one facial image of a person, and a computerized subsystem receiving the at least one facial image, the computerized subsystem including face model generation functionality operative to generate a face model corresponding to the at least one facial image, and image-to-attributes mapping functionality operative to assign values represented by adjectives to a plurality of facial attributes of the facial image, and a database which stores information and the values of facial attributes for a plurality of the persons.
• the computerized subsystem also includes attributes-to-image mapping functionality operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image associated with a particular individual, utilizing the face model.
• the computerized subsystem also includes a value combiner is operative to combine the face model and the collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values.
• the system also includes a subsequent facial image acquisition subsystem acquiring at least one facial image and supplying it to the computerized subsystem, and the computerized subsystem is preferably operative to create a face model corresponding to the subsequent facial image, assign values represented by adjectives to a plurality of facial attributes of the subsequent facial image, and identify a corresponding stored facial image and thereby the subsequent facial image as being that of a particular individual, for recognizing repeated presence of that particular person.
• a subsequent facial image acquisition subsystem acquiring at least one facial image and supplying it to the computerized subsystem
• the computerized subsystem is preferably operative to create a face model corresponding to the subsequent facial image, assign values represented by adjectives to a plurality of facial attributes of the subsequent facial image, and identify a corresponding stored facial image and thereby the subsequent facial image as being that of a particular individual, for recognizing repeated presence of that particular person.
• the value combiner is employed to combine the face model and the collection of values corresponding to the subsequent facial image thereby to recognize repeated presence of a person.
• the system also includes a repeat presence statistics generator employing the face models and the collections of values for generate attribute-wise statistics regarding persons repeatedly present at a place.
• the system also includes a social network interface for making available information from a social network to the computerized subsystem.
• the facial image/person identification acquisition subsystem is operative for acquiring at least one facial image and at least one item of personal identification of a person other than a person interacting with the subsystem. Additionally or alternatively, the facial image/person identification acquisition subsystem is operative for acquiring at least one facial image of an otherwise unidentified person other than a person interacting with the subsystem.
• the system is embodied in a computerized facial image attribute-wise evaluator, assigning values representing a facial image to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives, and a computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes.
• a method for generating a computerized facial image attribute-wise evaluator capable of assigning values, each represented by an adjective, to plural ones of discrete facial attributes of a facial image
• the method including gathering a multiplicity of facial images, each having at least one facial image attribute, characterized by an adjective, associated therewith, and generating a function operative to receive a facial image to be evaluated and to utilize results of the gathering for assigning values to plural ones of discrete facial attributes of the facial image to be evaluated, the values being represented by adjectives.
• the gathering includes collecting a multiplicity of facial images, each having at least one facial image attribute, characterized by an adjective, associated therewith from publicly available sources, and employing crowdsourcing to enhance correspondence between adjectives and facial attributes appearing in the multiplicity of facial images.
• the crowdsourcing includes employing multiple persons who view ones of the multiplicity of facial images and the adjectives and indicate their views as to the degree of correspondence between the adjectives and the facial attributes in the ones of the multiplicity of images.
• the values are numerical values.
• a system for recognizing user reaction to at least one stimulus including a computerized facial image attribute-wise evaluator, assigning values representing a facial image obtained at a time corresponding to user reaction to a stimulus to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives, and a computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes.
• the system also includes a computerized attribute comparator comparing the plural ones of the discrete facial attributes prior to and following application of the at least one stimulus.
• the method also includes comparing the plural ones of the discrete facial attributes prior to and following application of the at least one stimulus.
• a computerized system for classifying persons including a relationship coefficient generator which generates relationship coefficients representing the probability of a person to be in a particular context at a particular time, and a computerized classifier which classifies the person in accordance with the plural ones of the relationship coefficients.
• the context is one of a geographic location and an event.
• the relationship coefficients include a value and a decay function.
• the decay function is a linear function.
• the decay function is an exponential function.
• the system also includes a computerized classifier which classifies facial images in accordance with plural ones of discrete facial attributes.
• FIGs. 1A, IB and 1C are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with a preferred embodiment of the present invention
• FIGs. 3A and 3B are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with yet another preferred embodiment of the present invention
• FIGs. 5A and 5B are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with yet another preferred embodiment of the present invention
• Fig. 7 is a simplified illustration of an image/text/image database generation methodology useful in building a database employed in the systems of Figs. 1A - 6;
• Fig. 8 is a simplified flow chart illustrating a training process for associating adjectives with images
• Fig. 9 is a simplified flow chart illustrating the process of training a visual classifier
• Fig. 10 is a simplified flow chart illustrating a process for retrieving adjectives associated with an image
• Fig. 11 is a simplified flow chart illustrating a process for retrieving images associated with one or more adjectives.
• Fig. 12 is a simplified flow chart illustrating a process for retrieving facial images similar to a first image.
• Figs. 1A, IB and 1C are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with a preferred embodiment of the present invention.
• the system of Figs. 1A - 1C preferably includes a computerized facial image attribute- wise evaluator, assigning values representing a facial image to plural ones of discrete facial attributes of the facial image, the values being represented by adjectives, and a computerized classifier which classifies the facial image in accordance with the plural ones of the discrete facial attributes.
• the registration stand preferably includes a computer 102 connected to a store computer network, and a digital camera 104 connected to computer 102.
• the valued customer registration process includes entering personal identification details of the customer, such as his full name, and capturing a facial image 108 of the customer by digital camera 104.
• personal identification details of the customer may be retrieved, for example, from a pre-existing personal social network account of the customer.
• the customer may register as a valued location over the internet from a remote location.
• Computerized person identification system 110 which preferably includes face model generation functionality 112, image-to-attributes mapping functionality 1 14, attributes-to-image mapping functionality 116 and a value combiner 1 17.
• Computerized person identification system 1 10 also preferably includes a valued customer database 1 18 which stores registration details and values of facial attributes of all registered customers. It is appreciated that database 1 18 may be any suitable computerized information store.
• Face model generation functionality 112 is operative to generate a face model 120 which corresponds to facial image 108. It is appreciated that face model generation functionality 1 12 may employ any suitable method of face model generation known in the art. As seen in Fig. 1A, face model 120 generated by face model generation functionality 1 12 and corresponding to facial image 108 is stored in database 118 as one of the attributes of Mr. Jones.
• image-to-attributes mapping functionality 1 14 is operative to assign values represented by adjectives 122 to a plurality of facial attributes of facial image 108.
• the adjectives 122 representing the facial attributes may include, for example, adjectives describing hair color, nose shape, skin color, face shape, type and presence of or absence of facial hair.
• adjectives generated by attributes mapping functionality 1 14 which correspond to facial image 108 are stored in database 118 as values of attributes of Mr. Jones.
• attributes-to-image mapping functionality 1 16 is operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image, and thereby to identify a particular individual.
• value combiner 117 preferably is operative to combine a face model and a collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a particular individual.
• a customer enters the AAA Department Store and a digital camera 150, mounted at the entrance to the store, captures a facial image 152 of the customer.
• Facial image 152 is transmitted to computerized person identification system 110 where a face model 160 corresponding to facial image 152 is preferably generated by face model generation functionality 112.
• values 162 represented by adjectives are preferably assigned to a plurality of facial attributes of facial image 152 by image-to-attributes mapping functionality 1 14.
• face model 160 and adjectives 162 are preferably combined by value combiner 1 17 into a combined collection of values, which is compared to the collections of values stored in database 1 18, and are found to match the face model and adjectives assigned to Mr. Jones, thereby identifying the person portrayed in facial image 152 captured by camera 150 as being Mr. Jones. It is appreciated that the collection of values combined by value combiner 117 and which are compared to the collections of values stored in database 1 18 may be any subset of face model 160 and adjectives 162.
• Fig. 1C it is shown that for example, upon identifying the customer who has entered the store as Mr. Jones, who is a registered valued customer, the manager is notified by system 110 that a valued customer has entered the store, and the manager therefore approaches Mr. Jones to offer him a new product at a discount.
• FIGs. 2A and 2B are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with another preferred embodiment of the present invention.
• a customer of the AAA Department Store enters the store and a digital camera 200 mounted at the entrance to the store captures a facial image 202 of the customer.
• Facial image 202 is transmitted to a computerized person identification system 210 which preferably, includes face model generation functionality 212, image-to-attributes mapping functionality 214, attributes- to-image mapping functionality 216 and a value combiner 217.
• Computerized person identification system 210 also preferably includes a customer database 218, which preferably stores values of facial attributes of all customers who have ever entered the store, and a visit counter 219 which preferably tracks the number of accumulated visits that each particular customer has made to the store. It is appreciated that database 218 may be any suitable computerized information store.
• Face model generation functionality 212 is operative to generate a face model 220, which corresponds to facial image 202. It is appreciated that face model generation functionality 212 may employ any suitable method of face model generation known in the art. As seen in Fig. 2A, face model 220 generated by face model generation functionality 212 and corresponding to facial image 202 is stored in database 218 as one of the attributes of the customer of facial image 202.
• image-to-attributes mapping functionality 214 is operative to assign values represented by adjectives 222 to a plurality of facial attributes of facial image 202.
• the adjectives 222 representing the facial attributes may include, for example, adjectives describing age group, gender, ethnicity, face shape, mood and general appearance.
• attributes-to-image mapping functionality 216 is operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image, and thereby to identify a particular individual. It is appreciated that the collection of values may also include non-physical characteristics of the customer's appearance such as clothing type and color which may be used to identify an individual within a short period of time in a case where current values of facial attributes are not available.
• value combiner 217 preferably is operative to combine a face model and a collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a particular individual.
• face model 220 and adjectives 222 are preferably combined by value combiner 217 into a combined collection of values, which is compared to the collections of values stored in database 218, and are found to match the face model and adjectives corresponding to a returning customer. Therefore, the visit counter 219 of the customer is incremented. It is appreciated that the collection of values combined by value combiner 217 and which are compared to the collections of values stored in database 218 may be any subset of face model 220 and adjectives 222.
• the combined collection of values generated by value combiner 217 is not found to match any of the collections of values stored in database 218, the combined collection of values generated by value combiner 217 and facial image 202 are preferably stored in database 218 as representing a new customer, and the counter 219 of the new customer is initialized to 1.
• Fig. 2B it is shown that at closing time, such as at 5:00 PM on January 1, the manager of the store preferably receives a first report 230 from system 210 which includes a segmentation of customers who have entered the store over the course of the January 1.
• the segmentation may be according to any of the adjectives stored in database 218, such as gender, age group, ethnicity and mood.
• Report 230 also preferably includes information regarding the number of previous visits that were made to the store by the customers of January 1.
• the manager of the store may also receive a second report 234 from system 210 which includes a segmentation of returning customers who have entered the store over the course of the January 1.
• the segmentation may be according to any of the adjectives stored in database 218, such as gender, age group, ethnicity and mood. It is appreciated that reports 230 and 234 may be useful, for example, for planning targeted marketing campaigns, or for evaluating the success of previously executed marketing campaigns.
• FIGs. 3A and 3B are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with yet another preferred embodiment of the present invention.
• a customer of the AAA Department Store enters the store and browses merchandise in the store's toy department.
• a digital camera 250 mounted in the toy department captures a facial image 252 of the customer.
• additional digital cameras are preferably mounted throughout the various departments of the store.
• Facial image 252 is transmitted to a computerized person identification system 260 which includes face model generation functionality 262, image-to-attributes mapping functionality 264, attributes-to-image mapping functionality 266 and a value combiner 267.
• Computerized person identification system 260 also preferably includes a customer database 268, which preferably stores values of facial attributes of all customers who have entered the store during the day, and information indicating which of the store's departments each customer visited. It is appreciated that database 268 may be any suitable computerized information store.
• Face model generation functionality 262 is operative to generate a face model 270, which corresponds to facial image 252. It is appreciated that face model generation functionality 262 may employ any suitable method of face model generation known in the art. As seen in Fig. 3 A, face model 270 generated by face model generation functionality 262 and corresponding to facial image 252 is stored in database 268 as one of the attributes of the customer of facial image 252. In accordance with a preferred embodiment of the present invention, image-to-attributes mapping functionality 264 is operative to assign values represented by adjectives 272 to a plurality of facial attributes of facial image 252. The adjectives 272 representing the facial attributes may include, for example, adjectives describing age group, gender, ethnicity, face shape, mood and general appearance. As seen in Fig. 3 A, adjectives generated by attributes mapping functionality 264 which correspond to facial image 252 are stored in database .268 as values of attributes of the customer of facial image 252.
• attributes-to-image mapping functionality 266 is operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image, and thereby to identify a particular individual. It is appreciated that the collection of values may also include non-physical characteristics of the customer's appearance such as clothing type and color which may be used to identify an individual within a short period of time in a case where current values of facial attributes are not available.
• value combiner 267 preferably is operative to combine a face model and a collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a particular individual.
• system 260 records the department which the customer has visited in database 268 as being the toys department.
• Fig. 3B it is shown that at closing time, such as at 5:00 PM on January 1 , the manager of the store preferably receives a report 280 from system 260 which includes a segmentation of customers who have entered the store's toy department over the course of the January 1.
• the segmentation may be according to any of the adjectives stored in database 268, such as gender, age group, ethnicity and mood. It is appreciated that report 280 may be useful, for example, for planning targeted marketing campaigns, or for evaluating the success of previously executed marketing campaigns.
• Figs. 4A, 4B and 4C are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with yet another preferred embodiment of the present invention.
• a potential attendee registers to attend the florists' annual conference, preferably via a computer 300.
• the potential attendee is preferably prompted to enter personal identification details, such as his full name, and to upload at least one facial image 302 of himself.
• the potential attendee may choose to import personal identification details and one or more facial images, for example, from a pre-existing personal social network account.
• the personal identification details and facial image 302 are transmitted to a computerized conference registration system 310 which preferably includes face model generation functionality 312, image-to-attributes mapping functionality 314, attributes-to-image mapping functionality 316 and a value combiner 317.
• Computerized conference registration system 310 also preferably includes a database 318 which stores registration details and values of facial attributes of all registered attendees. It is appreciated that database 318 may be any suitable computerized information store.
• Face model generation functionality 312 is operative to generate a face model 320, which corresponds to facial image 302. It is appreciated that face model generation functionality 312 may employ any suitable method of face model generation known in the art. As seen in Fig. 4A, face model 320 generated by face model generation functionality 312 and corresponding to facial image 302 is stored in database 318 as one of the attributes of potential attendee Mr. Jones.
• image-to-attributes mapping functionality 314 is operative to assign values represented by adjectives 322 to a plurality of facial attributes of facial image 308.
• the adjectives representing the facial attributes may include, for example, adjectives describing hair color, nose shape, skin color, face shape, type and presence of or absence of facial hair.
• adjectives generated by attributes mapping functionality 314, which correspond to facial image 302 are stored in database 318 as values of attributes of potential attendee Mr. Jones.
• attributes-to-image mapping functionality 316 is operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image, and thereby to identify a particular individual.
• value combiner 317 preferably is operative to combine a face model and a collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a particular individual.
• Registration booth 330 includes a digital camera 332 which captures a facial image 334 of the attendee.
• Facial image 334 is transmitted computerized conference registration system 310 where a face model 340 corresponding to facial image 334 is preferably generated by face model generation functionality 312.
• values 342, represented by adjectives are preferably assigned to a plurality of facial attributes of facial image 334 by image-to-attributes mapping functionality 314.
• face model 340 and values 342 are preferably combined by value combiner 317 into a combined collection of values, which is compared to the collections of values stored in database 318, and are found to match the face model and values assigned to Mr. Jones, thereby identifying the person portrayed in facial image 334 captured by camera 332 as being Mr. Jones.
• the collection of values combined by value combiner 317 and which are compared to the collections of values stored in database 318 may be any subset of face model 340 and adjectives 342.
• Fig. 4C it is shown that while attending the conference, attendees who wish to be introduced to other attendees, allow other attendees to capture a facial image 350 of them, using, for example, a digital camera embedded in a mobile communicator device 352.
• Mobile communicator devices 352 of conference attendees are granted access to computerized conference registration system 310 via a computer network.
• the computer network may be, for example, a local computer network or the internet.
• an attendee may access computerized conference registration system 310 to register new, currently unregistered attendees to the conference, by capturing a facial image of the new attendee and transmitting the facial image, preferably together with associated personal identification information, to registration system 310.
• mobile communicator device 352 Upon capturing image 350 of a conference attendee, mobile communicator device 352 transmits image 350 over the computer network to computerized conference registration system 310 where a face model 360 corresponding to facial image 350 is preferably generated by face model generation functionality 312. Additionally, values 362 represented by adjectives are preferably assigned to a plurality of facial attributes of facial image 350 by image-to-attributes mapping functionality 314.
• face model 360 and values 362 are combined by value combiner 317 into a combined collection of values, which is compared to the collections of values stored in database 318, and are found to match the face model and values assigned to Mr. Jones, thereby identifying the person portrayed in facial image 350 captured by mobile communicator device 352 as being Mr. Jones.
• the collection of values combined by value combiner 317 and which are compared to the collections of values stored in database 318 may be any subset of face model 360 and adjectives 362. Notification of the identification of the attendee portrayed in image 350 as Mr. Jones is transmitted by computerized conference registration system 310 back to mobile communicator device 352, which notification enables the operator of mobile communicator device 352 to know that he is approaching Mr. Jones.
• Figs. 5A and 5B are simplified illustrations of an identification system employing image-to-text and text-to-image association in accordance with yet another preferred embodiment of the present invention.
• a relationship coefficient which measures the relationship between a person and a context is employed.
• the context may be, for example, a geographic location or an event, and the relationship coefficient comprises a value and a predefined decay function.
• a single person may have a relationship coefficient with multiple contexts simultaneously.
• the relationship coefficient can be used, for example, to predict the probability of a person being at a given location at a particular time.
• the decay function may be any mathematical function.
• the decay function for a geographical location may be a linear function, representing the tendency of a person to gradually and linearly distance himself from the location over time.
• the decay function for a one-time event may be, for example, an exponential decay function.
• the current value of the generated relationship coefficient between the person and the context is set to be high.
• the value of the relationship coefficient is increased, potentially in an exponential manner.
• contexts may be hierarchical.
• a geographic location may be within a larger geographical area such as a city or a country. Therefore, a person who has a relationship coefficient with a particular geographic location will also have a lower relationship coefficient with all other geographical locations hierarchical thereto, which decreases as a function of the distance between the particular geographic location and the related hierarchical geographic locations.
• relationship coefficient of different people may be at least partially interdependent. For example, a first person who has been sighted together with a second person at multiple locations at multiple times will be assigned a relatively high relationship coefficient to a new location where the second person has been sighted.
• a friend of the diner captures a facial image 400 of the diner using a digital camera which is part of a handheld mobile device 402 and registers the sighting of the diner by transmitting facial image 400 together with an associated time and location over the internet to a computerized person identification system 410.
• the location may be provided, for example, by a GPS module provided with device 402. Alternatively, the location may be retrieved, for example, from a social network. Using the associated time and location, a relationship coefficient which relates the diner to the location is generated as described hereinabove.
• Computerized person identification system 410 includes face model generation functionality 412, image-to-attributes mapping functionality 414, attributes- to-image mapping functionality 416 and a value combiner 417.
• Computerized person identification system 410 also preferably includes a sightings database 418 which preferably stores values of facial attributes of all persons who have been sighted and registered, together with an associated time and location. It is appreciated that database 418 may be any suitable computerized information store.
• Face model generation functionality 412 is operative to generate a face model 420, which corresponds to facial image 400. It is appreciated that face model generation functionality 422 may employ any suitable method of face model generation known in the art. As seen in Fig. 5A, face model 420 generated by face model generation functionality 412 and corresponding to facial image 400 is stored in database 418 as one of the attributes of the individual of facial image 400.
• image-to-attributes mapping functionality 414 is operative to assign values represented by adjectives 422 to a plurality of facial attributes of facial image 400.
• the adjectives 422 representing the facial attributes may include, for example, adjectives describing age group, gender, ethnicity, face shape, mood and general appearance.
• adjectives generated by attributes mapping functionality 414 which correspond to facial image 400 are stored in database 418 as values of attributes of the individual of facial image 400. Additionally, the time and location associated with facial image 400 are also stored in database 418.
• attributes-to-image mapping functionality 416 is operative to utilize a collection of values of facial attributes to identify a corresponding stored facial image, and thereby to identify a particular individual. It is appreciated that the collection of values may also include non-physical characteristics of the customer's appearance such as clothing type and color which may be used to identify an individual within a short period of time in a case where current values of facial attributes are not available.
• value combiner 417 preferably is operative to combine a face model and a collection of values of facial attributes into a combined collection of values which can be matched to a corresponding stored collection of values, and thereby to identify a particular individual.
• Fig. 5B it is shown that on a later date, such as on February 1, 2011, a diner dines at Cafe Jaques which is in close proximity to the Eiffel Tower in Paris, France.
• a bystander captures a facial image 450 of the diner using a digital camera which is part of a handheld mobile device 452 and registers the sighting of the diner by transmitting facial image 450 together with an associated time and location over the internet to a computerized person identification system 410 where a face model 460, corresponding to facial image 450, is preferably generated by face model generation functionality 412.
• values 462 represented by adjectives are preferably assigned to a plurality of facial attributes of facial image 450 by image- to-attributes mapping functionality 414.
• face model 460, values 462 and the time and location associated with facial image 450 are preferably combined by value combiner 417 into a combined collection of values, which is compared to the collections of values stored in database 418, and are found to match the combined values assigned to the diner who was last seen at the Eiffel Tower on January 1, 2011. It is appreciated that the collection of values combined by value combiner 417 and which are compared to the collections of values stored in database 418 may be any subset of face model 460 and adjectives 462. Notification of the identification of the diner portrayed in image 450 is transmitted over the internet by computerized person identification system 410 back to mobile communicator device 452.
• the relationship coefficient which relates the diner to the location may also be used as an attribute value which increases the reliability of the identification of the diner.
• the combination of the values of the facial attributes associated with a facial image together with additional information such as a particular location frequented by an individual is operative to more effectively identify individuals at the particular location or at related locations, such as at other locations which are in close proximity to the particular location.
• identification of individuals according to the present embodiment of the current invention is not limited to precise identification of particular individuals based on personal identification information such as first and last name, but rather also includes identification of individuals according by facial attributes and aggregating behavioral information pertaining to the individuals.
• FIG. 6 is a simplified illustration of a user satisfaction monitoring system employing image-to-text association in accordance with yet another preferred embodiment of the present invention.
• a viewer uses a multimedia viewing device 480 to view computerized content 482.
• device 480 may be, for example, a television device or a computer.
• Content 482 may be, for example, a video clip, a movie or an advertisement.
• a digital camera 484 connected to multimedia viewing device 480 preferably captures a facial image 486 of the viewer at predefined intervals such as, for example, every few seconds, and preferably transmits images 486 over the internet to an online computerized content satisfaction monitoring system 490.
• images 486 may be monitored, stored and analyzed by suitable functionality embedded in device 480.
• system 490 includes image-to-attributes mapping functionality 492 and a viewer expressions database 494.
• database 494 may be any suitable computerized information store.
• image-to-attributes mapping functionality 492 is operative to assign a value represented by an adjective 496 to the expression of the viewer as captured in facial images 486, and to store adjectives 496 in database 494.
• Adjectives 496 may include, for example, "happy”, “sad”, “angry”, “content” and “indifferent”. It is appreciated that adjectives 496 stored in database 494 may be useful, for example, for evaluating the effectiveness of content 482.
• Fig. 7 is a simplified illustration of an image/text/image database generation methodology useful in building a database employed in the systems of Figs. 1A - 6.
• Image repository 502 may be, for example, a publicly available social network or textual search engine which associates text with images appearing on the same page as the images or on one or more nearby pages.
• image repository 502 may be, for example, a publicly available social network or textual search engine which associates text with images appearing on the same page as the images or on one or more nearby pages.
• one or more associated characteristics are provided by the image repository with each of images 500.
• the characteristics may include, for example, a name, age or age group, gender, general appearance and mood, and are generally subjective and are associated with the images by the individuals who have publicized the images or by individuals who have tagged the publicized images with comments which may include such characteristics.
• Computerized person identification training system 510 first analyzes each of the characteristics associated with each of images 500 and translates each such suitable characteristic to an attribute value. For each such value, system 510 then sends each of images 500 and its associated attribute value to a crowdsourcing provider, such as Amazon Mechanical Turk, where a plurality of individuals voice their opinion as to the level of correspondence of each image with its associated attribute value. Upon receiving the crowdsourcing results for each image-attribute value pair, system 510 stores those attribute values which received a generally high correspondence level with their associated image in a database 520.
• a crowdsourcing provider such as Amazon Mechanical Turk
• Fig. 8 is a simplified flow chart illustrating a training process for associating adjectives with images.
• an adjective defining a facial attribute is chosen by the system from a list of adjectives to be trained, and one or more publicly available textual search engines are preferably employed to retrieve images which are associated with the adjective. Additionally, one or more publicly available textual search engines are preferably employed to retrieve images which are associated with one or more translations of the adjective in various languages.
• the list of adjectives may be compiled, for example, by collecting adjectives from a dictionary.
• a visual face detector is employed to identify those retrieved images which include a facial image. Crowdsourcing is then preferably employed to ascertain, based on a majority vote, which of the facial images correspond to the adjective. The adjective and corresponding facial images are then used to train a visual classifier, as described hereinbelow with regard to Fig. 9. The visual classifier is then employed to associate the adjective with an additional set of facial images, and crowdsourcing is further employed to ascertain the level of correspondence of each of the additional set of facial images with the adjective, the results of which are used to further train the visual classifier. It is appreciated that additional cycles of crowdsourcing and training of the visual classifier may be employed to further refine the accuracy of the visual classifier, until a desired level of accuracy is reached. After the training of the visual classifier, the classifier is added to a bank of attribute functions which can later be used by the system to classify facial images by adjectives defining facial attributes.
• Fig. 9 is a simplified flow chart illustrating the process of training a visual classifier.
• the results of the crowdsourcing process described hereinabove with regard to Fig. 8 are employed to generate two collections of images.
• a first, "positive” collection includes images which have been ascertained to correspond to the adjective, and a second, "negative" collection includes images which have not been ascertained to correspond to the adjective.
• the images of both the positive and the negative collection are then normalized to compensate for varying 2- dimensional and 3 -dimensional alignment and differing illumination, thereby transforming each of the images into a canonical image.
• the canonical images are then converted into canonical numerical vectors, and a classifier is learned from a training set comprising of pairs of positive and negative numerical vectors using a supervised-classifier, such as a Support Vector Machine (SVM).
• SVM Support Vector Machine
• Fig. 10 is a simplified flow chart illustrating a process for retrieving adjectives associated with an image.
• the image is first analyzed to detect and crop a facial image which is a part of the image.
• the facial image is then converted to a canonical numerical vector by normalizing the image to compensate for varying 2-dimensional and 3 -dimensional pose-alignment and differing illumination.
• the bank of attribute functions described hereinabove with regard to Fig. 8 is then applied to the numerical vector, and the value returned from each attribute function is recorded in a numerical vector which represents the adjectives associated with the facial image.
• Fig. 10 is a simplified flow chart illustrating a process for retrieving adjectives associated with an image.
• FIG. 1 is a simplified flow chart illustrating a process for retrieving images from a pre-indexed database of images, which are associated with one or more adjectives.
• a textual query for images having adjectives associated therewith is first composed.
• NLP Natural Language Processing
• adjectives are extracted from the textual query.
• the system retrieves images from a previously processed database of facial images which are best-matched to the adjectives extracted from the query, preferably by using Latent Dirichlet Allocation (LDA).
• LDA Latent Dirichlet Allocation
• the retrieved facial images are ordered by the level of correlation of their associated numerical vectors to the adjectives extracted from the query, and the resulting ordered facial images are provided as output of the system.
• Fig. 12 is a simplified flow chart illustrating a process for retrieving facial images which are similar to a first image.
• the first image is first analyzed to detect and crop a facial image which is a part of the image.
• the facial image is then converted to a canonical numerical vector by normalizing the image to compensate for varying 2-dimensional and 3- dimensional pose-alignment and differing illumination.
• the bank of attribute functions described hereinabove with regard to Fig. 8 is then applied to the numerical vector, and the value returned from each attribute function is recorded in a numerical vector which represents the adjectives associated with the facial image.
• a previously indexed database comprising numerical vectors of images, such as a KD tree, is searched using a similarity-function, such as Euclidian distance, to find a collection of numerical vectors which represent images which closely match the numerical vector of the first image.
• a similarity-function such as Euclidian distance

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Library & Information Science (AREA)
• Databases & Information Systems (AREA)
• Data Mining & Analysis (AREA)
• General Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• General Health & Medical Sciences (AREA)
• Oral & Maxillofacial Surgery (AREA)
• Human Computer Interaction (AREA)
• Multimedia (AREA)
• Software Systems (AREA)
• Processing Or Creating Images (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Image Analysis (AREA)
• Image Processing (AREA)

Priority Applications (7)

Applications Claiming Priority (2)

Publications (1)

Family

ID=46602131

Family Applications (1)

Country Status (8)

Cited By (5)

Families Citing this family (8)

Family Cites Families (10)

• 2011
  • 2011-03-31 WO PCT/IL2011/000287 patent/WO2012104830A1/en active Application Filing
  • 2011-03-31 BR BR112013019907A patent/BR112013019907A2/pt active Search and Examination
  • 2011-03-31 AU AU2011358100A patent/AU2011358100B2/en not_active Ceased
  • 2011-03-31 KR KR1020137023294A patent/KR101649322B1/ko active IP Right Grant
  • 2011-03-31 CA CA2826177A patent/CA2826177C/en not_active Expired - Fee Related
  • 2011-03-31 CN CN201180069586.1A patent/CN103620590B/zh active Active
  • 2011-03-31 MX MX2013008985A patent/MX345437B/es active IP Right Grant
  • 2011-03-31 JP JP2013552321A patent/JP5857073B2/ja active Active

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events