Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04H—BROADCAST COMMUNICATION
  • H04H60/00—Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
  • H04H60/61—Arrangements for services using the result of monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
  • H04H60/65—Arrangements for services using the result of monitoring, identification or recognition covered by groups H04H60/29-H04H60/54 for using the result on users' side
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04H—BROADCAST COMMUNICATION
  • H04H60/00—Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
  • H04H60/35—Arrangements for identifying or recognising characteristics with a direct linkage to broadcast information or to broadcast space-time, e.g. for identifying broadcast stations or for identifying users
  • H04H60/45—Arrangements for identifying or recognising characteristics with a direct linkage to broadcast information or to broadcast space-time, e.g. for identifying broadcast stations or for identifying users for identifying users
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/41—Structure of client; Structure of client peripherals
  • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
  • H04N21/4223—Cameras
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/441—Acquiring end-user identification, e.g. using personal code sent by the remote control or by inserting a card
  • H04N21/4415—Acquiring end-user identification, e.g. using personal code sent by the remote control or by inserting a card using biometric characteristics of the user, e.g. by voice recognition or fingerprint scanning
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
  • H04N21/4508—Management of client data or end-user data
  • H04N21/4532—Management of client data or end-user data involving end-user characteristics, e.g. viewer profile, preferences
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
  • H04N21/454—Content or additional data filtering, e.g. blocking advertisements
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
  • H04N21/466—Learning process for intelligent management, e.g. learning user preferences for recommending movies
  • H04N21/4661—Deriving a combined profile for a plurality of end-users of the same client, e.g. for family members within a home
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/16—Analogue secrecy systems; Analogue subscription systems
  • H04N7/162—Authorising the user terminal, e.g. by paying; Registering the use of a subscription channel, e.g. billing
  • H04N7/163—Authorising the user terminal, e.g. by paying; Registering the use of a subscription channel, e.g. billing by receiver means only

Definitions

• This invention relates to the field of consumer electronics, and in particular to a system and method for automated control of devices based on images from a video capture device in the vicinity of the controlled device.
• Face recognition is commonly used for security purposes.
• security badges containing facial photographs are used to control access to secured areas or secured material.
• face recognition software is used to similarly match a current image of a person, from, for example, a video camera, with a stored image.
• the user identifies himself or herself, and the face recognition software compares the video image with one or more stored images of the identified person.
• identification badges In an office environment, for example, employees can be expected to wear identification badges, or to purposefully engage biometric scanners, such as a facial recognition device, a retinal scanner, a fingerprint scanner, and so on.
• biometric scanners such as a facial recognition device, a retinal scanner, a fingerprint scanner, and so on.
• having to type in an identification sequence, or to carry an identification token, or to purposefully engage a recognition device, in order to activate or otherwise control an entertainment device would likely be viewed as an annoyance, or at least an inconvenience.
• a controller accesses a database containing a user profile corresponding to the recognized user.
• This user profile contains the user's preferences for presentation material, such as preferences for particular types of entertainment at particular times of the day.
• the controller activates one or more devices, such as a television set or stereo set, and selects or recommends material for presentation to the user that is likely to coincide with the user's desires at that time.
• FIG. 1 illustrates an example television controller with face recognition capabilities in accordance with this invention.
• FIG. 2 illustrates an example flow diagram of an appliance controller with face recognition in accordance with this invention.
• This facilitation of control includes, for example, automatically activating a device, providing recommendations for selecting from among a variety of options, limiting access to select material or controls, and so on.
• the television or stereo When a user initially enters a room, the television or stereo may be activated, depending upon the user profile, and depending upon the time of day and other factors. The particular television channel or music selection is also selected based on the user profile, time of day, and so on. As the user traverses from one room to another, a television in the room that is entered is turned on and tuned to the station that the user was viewing in the prior room, and the television in the prior room is turned off, unless another locator device is detected in the prior room.
• Copending U.S. patent application "REMOTE CONTROL PROGRAM SELECTION BY GENRE", U.S.
• the system provides control, or recommendations, based on an identification of the current user or users of the device being controlled.
• an 'active' method of user identification such as the entry of a user name or code, or carrying an identification token, may not be consistent with a consumer's view of "convenience".
• this user identification is provided in a 'passive' manner, based on a recognition of the presence of the user in the vicinity of the device being controlled.
• An image capture device such as a video camera, captures an image of a scene in the vicinity of the device being controlled, and a user- recognition application, such as a face-recognition application, identifies the user from the image.
• FIG. 1 illustrates an example device control system 100 with user-recognition capabilities in accordance with this invention.
• a television system 130 is presented as the device being controlled, although any other user-controllable device, such as a stereo system, an Internet access device, and so on may be the device being controlled.
• An imaging device 110 such as a video camera, captures an image of objects within its field of view 115, and provides this image to a user-recognizer 120.
• the user-recognizer 120 is a multi-stage face recognition device that searches the image for an area of the image having the general characteristics of a face, based on the location of flesh tones, the location of non-flesh tones corresponding to eye brows, demarcation lines corresponding to chins, nose, and so on (see the referenced US patent 5,835,616). If a face is detected, it is characterized for comparison with reference faces that are stored in a database associated with the face recognizer 120.
• This characterization of the face in the video image is preferably the same characterization process that is used to characterize the reference faces, and facilitates a comparison of faces based on characteristics, rather than an 'optical' match, thereby obviating the need to have two identical images (current face and reference face) in order to locate a match.
• the number of reference faces is relatively small, typically limited to the number of people in a household, or other small sized environment, thereby allowing the face recognition process to be effected quickly.
• a television is illustrated as the paradigm device 130 being controlled.
• the controller 140 includes a 'profile' corresponding to each recognizable user that preferably includes the user's preferences for the operation of the device 130, including preferences that are time dependent. These preferences may be explicitly input by each user, or may be based on prior experiences. For example, if a particular user always turns on the television at 6 p.m., and views a news program on channel 8, the profile will contain this 'preference'. Note that the term 'user preference' is used herein in the general sense.
• a child's profile may be defined, for example, based on the child's parents' preferences. These preferences may include a "preference" for only educational shows during particular hours, no access during other hours, no access to particular classes of material, a limit to the number of hours that a television can be viewed per day, and so on.
• Any of a variety of machine learning techniques may also be used to develop correlations between each user and typical control activities, and thereafter imitate the execution of the control activities based on these correlations.
• the controller 140 executes control options corresponding to the assumed or explicit preference of the user.
• the controller 140 turns on the television 130, and tunes it to channel 8. If the user generally turns on the television whenever the user enters the room, and generally leaves the television tuned to whatever channel is currently turned on until 6 p.m., the controller 140 will imitate this behavior by turning the television on when the user is detected, and tuning to channel 8 only at 6 p.m.
• a gait recognition device can be used to identify a person based on the motion of a person in the image.
• a less complex recognition device may be based on the user's size.
• the user profiles may be segregated into 'adult' and 'child' profiles, wherein some or all controls are based on whether the user is an adult or child based on the user's size.
• the recognition may merely be the presence of a person in a particular location, such as a user's favorite chair, using for example, detection via an infrared image collector.
• FIG. 2 illustrates an example flow diagram of a device control system with user recognition in accordance with this invention. Face recognition is used in FIG. 2 as the paradigm of image-based user recognition, for ease of understanding.
• the video image (from the imaging device 110 of FIG. 1) is scanned to determine the presence of a face in the image. This scanning may be continuous, periodic, event-driven, or otherwise initiated. For example, the scanning may be continuous until a recognized face is detected, then periodically scanned, for example, at intervals corresponding to the traditional 'program change' periods (in the U.S., programs generally change on half-hour or hour boundaries). Alternatively, the scanning may be coupled to a television, and activated at the beginning of each commercial break.
• the characteristics of the detected face are determined, at 220. If a user has been identified, via a prior image scan, the characteristics of the detected face are compared to the characteristics of the identified user, at 225. If the detected face does not correspond to the identified user, or if there is no currently identified user, a database of user faces 201 is searched, at 230, for a user face having the same characteristics of the detected face. If, at 235, a user is identified that corresponds to the detected face, the identification of the "current" user is set, at 240, and the device is controlled corresponding to this current user identification, at 260.
• the new user's face characteristics can be added to the database of user faces 201, and a new user profile is created.
• the new user profile is based on a default profile that can be created by a 'system administrator' of the device control system.
• this lack of user- identification at 235 can also be used to create the database of user faces 201, or to provide alternative images of existing identified users in the database 201.
• the controller may default to a defined "guest" user profile.
• the identification of a "current" user is cleared, at 250, and the device is controlled corresponding to the absence of a current user identification, at 260.
• the identification of the "current" user comprises a set of identifications corresponding to each current user, and the process 210-260 is repeated for each face that is detected, or not detected, in each image. That is, for example, the decision block 215 results in a "yes” determination for each face in the image, then results in a "no” determination when there are no other faces in the image.
• Block 250 receives this "no” determination, and removes any identifications of users who have not been identified in the current image from the set of current users.
• the control of the device is dependent upon a database of user profiles 202. If a single user is identified, the control of the device corresponds directly to the determined user's profile. In a simple embodiment of this invention, if multiple users are identified, only unanimous preferences are effected. That is, for example, if each user generally turns on the television, or generally leaves the television in whatever state (on/off) it was in, when the user enters the room, the control block 260 is configured to turn on the television whenever the first user who generally turns on the television enters the room. If, however, another user, who normally turns off the television, subsequently enters the room, the control block 260 is configured to take no action until the other users are no longer in the room.
• the control block 260 takes no action until a coincidence of preferences occurs.
• the database 202 includes multi-user profiles, such as the channels that are normally viewed when users A and B, but not C, are present, and so on.
• control block 260 and the user profile 202 may also operate using state-dependent rules. For example, if a particular user generally leaves and enters a room repeatedly, the rules may be structured to invoke a 'waiting period' before clearing the user's identification from the current set, at 250.
• control system 100 may be configured to operate with other control systems, or within other control system architectures, as well.
• the devices 110-140 may be interconnected via a home automation network and control protocol.
• Home automation standards include IEEE 1394, X-10, HAVi, HomeAPI, Jini, and the like.
• IEEE 1394 and X-10 are communication protocols; HAVi is a software architecture using IEEE 1394; Home API is an open industry specification that defines a standard set of software services and application programming interfaces which enable software applications to monitor and control home device; and Jini is a distributed software architecture (network) wherein clients see devices and services as objects.
• the television controller 140 may include, for example, a personal computer with an IEEE 1394 interface to a HAVi-compliant satellite television receiver, and the face recognizer 120 is a software application that is also located within the personal computer.
• control of the device may be via the control an ancillary device, such as controlling the channel being viewed on a television by controlling a cable or satellite receiver system, a VCR tuner, and the like.

Landscapes

• Engineering & Computer Science (AREA)
• Signal Processing (AREA)
• Multimedia (AREA)
• Databases & Information Systems (AREA)
• Theoretical Computer Science (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• General Health & Medical Sciences (AREA)
• Computer Security & Cryptography (AREA)
• Biomedical Technology (AREA)
• Life Sciences & Earth Sciences (AREA)
• Artificial Intelligence (AREA)
• Health & Medical Sciences (AREA)
• Bioinformatics & Computational Biology (AREA)
• Human Computer Interaction (AREA)
• Data Mining & Analysis (AREA)
• Evolutionary Biology (AREA)
• Evolutionary Computation (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
• Image Analysis (AREA)
• Image Processing (AREA)
• Collating Specific Patterns (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=24753255

Family Applications (1)

Country Status (5)

Cited By (2)

Families Citing this family (43)

Family Cites Families (3)

• 2001
  • 2001-09-26 EP EP01974299A patent/EP1329105A2/en not_active Withdrawn
  • 2001-09-26 KR KR1020027007393A patent/KR20020081220A/ko not_active Application Discontinuation
  • 2001-09-26 WO PCT/EP2001/011141 patent/WO2002032136A2/en not_active Application Discontinuation
  • 2001-09-26 CN CN01803050A patent/CN1395797A/zh active Pending
  • 2001-09-26 JP JP2002535392A patent/JP2004511870A/ja not_active Withdrawn

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events