JP2003516049A - Personality-based intelligent camera system - Google Patents

Abstract

SUMMARY A camera system includes a camera embedded or otherwise associated with an anthropomorphic object, such as a creature having controllable features. The features are controlled such that the particular configuration indicates the operational status of the camera system. The camera may be, for example, a zoom camera embedded in the eye or other feature of the object. In the illustrated embodiment, the object has a head and a body, and the controllable feature is a facial feature, such as a pair of controllable eyebrows, a pair of controllable eyelids, and a controllable mouth. is there. The controllable features may be performed using, for example, mechanical motor controls or screen-based animation controls. Pan and tilt controls are provided for the head of the anthropomorphic object so that the embedded camera can pan, tilt, and zoom. The system may include additional cameras, for example, a video presence camera, to provide a variety of different camera angles. The controllable features of the anthropomorphic object are adjusted according to an interactive protocol that specifies a particular form of the feature for a corresponding operating state of the camera system, thereby providing a natural and easy to understand user interface. provide.

Description

Detailed Description of the Invention

FIELD OF THE INVENTION The present invention relates to the field of video signal processing, and more particularly to camera systems used in video conferencing and other applications.

BACKGROUND OF THE INVENTION Camera systems that include tracking capabilities are well known. For example, pan-tilt-zoom (PTZ) cameras that track humans or other objects of interest are an important aspect of many video-camera based systems, such as video conferencing and video surveillance systems. For example, while it is often desirable in video conferencing systems to frame the heads and shoulders of certain conferees in the resulting output video signal, video surveillance systems, for example, fall within the limits monitored by the system. Or, it is desirable to attach a frame to the whole body of the human being.
Intelligent camera systems that can be controlled by the content of the resulting video or audio signal are also well known.

A significant problem with conventional intelligent camera systems is that such systems generally do not provide a sufficient user interface. For example, conventional systems do not recommend user interaction, mitigate the stress-induced effects of the "eyes" of the moving camera looking at the user, and also provide a natural intuitive interface and indication of various operating states of the system. It also does not provide appropriate user input. This problem is especially seen in home-based applications such as home-based video conferencing, video surveillance, inputs and outputs controlling appliances or other devices. Many users who come into contact with the camera system in such an environment are not sufficiently familiar with its operation to interact with it in a meaningful way. Therefore, conventional systems have been developed to facilitate human interaction with the system in applications such as home-based video conferencing, video surveillance, device control, or any other application in which an intelligent camera system may be implemented. There is a need for an improved camera system that incorporates a user interface that is more natural and easily understood than that provided by.

SUMMARY OF THE INVENTION The present invention provides a personality-based camera system that includes an anthropomorphic object, for example, an object such as a creature, which anthropomorphic object has an associated controllable face. Have the features of. According to the present invention, the creation of an anthropomorphic object is controlled to provide a user interface with which a camera is embedded or otherwise associated with the object. The feature is controlled such that its particular form indicates the operational state of the camera system. The camera may be, for example, a zoom camera embedded in the eyes or other features of the object.

In an exemplary embodiment of the invention, the object has a head and a body and the controllable feature is a facial feature, such as a controllable pair of eyebrows, a controllable pair of eyelids, and It has a controllable mouth. The controllable feature may be performed using, for example, mechanical motor control or screen-based animation control. Embedded camera pans,
Pan and tilt controls are also provided for the head of the anthropomorphic object so that it can tilt and zoom. The controllable features of the anthropomorphic object are adjusted according to a dialogue protocol that specifies the particular form of the feature for the corresponding operating conditions of the camera system.

According to the invention, the camera associated with the anthropomorphic object controls and negotiates the interaction with the user. For video presence applications, for example, an additional camera covering activity in the corresponding room may be configured to operate independently of the anthropomorphic camera, thereby providing separate functionality for user interface and content coverage. To be able to do.

Advantageously, the present invention controls the creation of anthropomorphic objects in an intelligent manner to facilitate human interaction with personality-based camera systems. Anthropomorphic objects provide a natural and easy-to-understand user interface. The above and other features of the present invention will become more apparent from the accompanying drawings and the following detailed description.

DETAILED DESCRIPTION OF THE INVENTION The present invention provides an improved intelligent camera system in which a camera is embedded or otherwise associated with an anthropomorphic object such as a living being. The system is configured to provide control of features of a creature-like device to facilitate user interaction with the device. The camera system may be capable of panning, tilting, and zooming, and may be used in applications such as home-based video conferencing, video surveillance, input and output to control appliances or other devices. System software executed by a computer or other processor-based device associated with the camera system is used to determine what operation the camera system will be suitable for a given application. The camera system uses multi-modal input to try to keep the activity of most interest in the frame at all times,
For example, conventional tracking capabilities may be included to keep the speaker or other object of interest within the video frame.

By implementing the camera system with a living-like personality, the present invention provides a significantly improved user interface that can guide the user in a natural and intuitive way to the optimal way to interact with the system. To do. As a result, the camera system is less of a threat to the user, and such a user is therefore likely to talk or otherwise interact with the camera system. For example, the camera system may be provided with a dog-like personality in accordance with the present invention, and the user is likely to talk and interact with the camera system as if it were a dog. More specifically, users call gestures by name before they expect to do something, use widely and easily recognizable gestures when interacting, and use normal words to allow errors and to accept system failures. It is likely that you will generally use explicit commands that are different from and that you can enjoy and understand the behavior while having fun. Using simple words or other remarks, such as "hump", barking sounds, etc., the system may indicate an acknowledgment of commands and instructions. this is,
It tends to further reduce the stress of the user's interaction with the camera system.

Moreover, by using common creature-like facial features to indicate the internal state of the camera system, the system can intuitively represent complex user interface problems. For example, the on / off state is by closing your eyes, the zoom of the camera is by narrowing or opening your eyes, the search is by moving your eyes back and forth quickly, the confusion is by turning your eyes, and the comprehension is nodding. This will be shown. Many other states of facial expression that map well from known emotions to the internal states of the system may also be used. The system may be configured to provide consistency in that once a particular form of interaction has been learned by a user, it should behave in essentially the same way each time. Appropriate feedback may be provided to ensure that the user understands that the system is in fact consistently behaving.

FIG. 1 is a diagram illustrating a personality-based intelligent camera system 100 according to an exemplary embodiment of the present invention. The system 100 includes a zoom camera 102 controlled by a camera controller 104. As will be described in more detail in conjunction with FIGS. 2 and 4, the camera 102 can pan and tilt certain anthropomorphic objects, such as facial features such as eyebrows, mouth, and eyelids. Associated with a creature-like object with some common features, such as having a head. The creature may be human, animal, alien, insect, etc. More specifically, the camera 102 may be embedded, attached, or otherwise attached to an anthropomorphic object, eg, forming at least a portion of an eye or other element of the object. .

Advantageously, the present invention enables such features to be controlled in an intelligent manner to facilitate human interaction with the personality-based camera system 100. As mentioned above, anthropomorphized objects provide a natural and easy-to-understand user interface.

The camera 102 produces an analog video signal which is input to and supplied to an analog-to-digital (A / D) converter 106. The A / D converter 106 is connected to the camera 102.
From the analog video signal to a digital video signal suitable for being processed by a set of system software 108. The system software 108 produces camera control commands that are provided to the camera controller 104 and used to produce the appropriate camera adjustment signals that are provided to the camera 102. The system software 108 also generates motor control commands that are supplied to the motor controller 110. The motor controller 110 includes a head control motor 112 and a face making motor 114.
To generate a motor operation signal. The motors 112 and 114 control the physical movements of the head and face features of the anthropomorphic object coupled to the camera 102, respectively.

FIG. 2 is a diagram showing a personality-based intelligent camera system 200 displaying a more specific implementation of the system of FIG. The system 200 includes a zoom camera 202, a camera controller 204, an A / D converter 206, and system software 20.
8 and a motor controller 210. As in the system of FIG. 1, camera 202 produces an analog video signal that is provided to A / D converter 206 for conversion into a digital video signal. The digital video signal is provided to system software 208, which produces the appropriate camera control commands to provide to camera controller 204. The system software 208 is
It also generates motor control commands that are supplied to the motor controller 210.

In the system 200, the zoom camera 202 is embedded within the head 215 of a particular anthropomorphic object, such as a human-like, animal-like, or other creature-like object. For example, the camera may be part of the anthropomorphic object's eye or other element. As shown in FIG. 2, a pair of eyebrows 221 and 22 are associated with the head 215.
There are several facial features including two, a pair of eyelids 223 and 224, and a mouth 225. The head 215 pans and tilts in the manner shown. Eyebrows 221 and 222
It is tilted and moved up and down in the direction of the arrow. The eyelids 223 and 224 are opened and closed, the mouth has one degree of freedom, and the center of the lips is raised when the face is uncomfortable and lowered when smiling.

Advantageously, the particular state of a given set of facial features is indicative of the particular state of the camera system. For example, a closed eyelid may indicate that the system is in an off state, and a raised eyebrow may indicate input required by the user. It should be emphasized that this particular arrangement of features is exemplary only, and numerous other arrangements are of course used.

Motor controller 210 controls six motors 230, 232, 234, 236, 238, and 240 to control the movement of head 215 and associated facial features.
Supply a motor operation signal to the set of. More specifically, the motor operation signals supplied by the motor controller 210 to the head pan motor 230 and the head tilt motor 232 control the pan and tilt of the head 125, respectively. The motor operation signals supplied to the eyebrow tilting motor 234 and the eyebrow height motor 236 by the motor controller 210 control the tilting and up / down operations of the pair of eyebrows 221 and 222, respectively. Similarly, the motor operation signal supplied to the upper / lower eyelid motor 238 by the motor controller 210 controls opening / closing of the eyelids 223 and 224, and the motor operation signal supplied to the mouth motor 240 by the motor controller 210 is Controls the movement of mouth 225.

FIG. 3 is a diagram showing a personality-based intelligent camera system 300 displaying another possible implementation of the system of FIG. The system 300 includes the zoom camera 3
02, camera controller 304, A / D converter 306, system software 308
, And a motor controller 310. As in the system of FIGS. 1 and 2, the camera 302 includes an A / D converter 306 for converting to a digital video signal.
To generate an analog video signal. The digital video signal is provided to system software 308, which produces the appropriate camera control commands to provide to camera controller 304. The system software 308 also generates motor control commands that are provided to the motor controller 310. The motor controller 310 in this example is designed for a specific anthropomorphic object head 316.
The motor operation signal is supplied to the head control motor 312 that controls the pan and tilt, for example. The camera 302 may be embedded or otherwise coupled to the head or other suitable portion of this anthropomorphic object.

In system 300, the anthropomorphic object has a liquid crystal display (LCD) screen 318 that provides facial feature animation. For example, screen 3
18 may be attached to or otherwise coupled to head 316 and is configured to provide a feature animation similar to that provided by the mechanical motor-based control described with system 200 of FIG. May be. The system software 308 supplies the screen graphics commands to the LCD screen driver 314, which generates corresponding LCD pixel control signals to provide the desired animation on the screen 318.

FIG. 4 is a diagram illustrating an example appearance of a personality-based intelligent camera system 400 according to an exemplary embodiment of the present invention. The appearance of system 400 is an anthropomorphic object of a human, animal, or other creature having head 415 and body 417. In relation to the head 415, eyebrows 421 and 422, eyelids 423 and 424, and mouth 4
There are several facial features, including 25. Head 415 and associated facial features may be controlled using, for example, the motor-based control technique described in conjunction with FIG. 2, the screen-based control technique described in conjunction with FIG. 3, or a combination of these and other control techniques.

A zoom camera, such as the camera 102, 202, or 302 in the embodiments described above, may be used in all desired locations of the anthropomorphic object, for example, in the eyes of the head 415 or as part of other features. It may be embedded, attached, or otherwise installed in the body 417 or in all other places. For this example, the zoom camera is head 4
Suppose it is embedded in one of fifteen eyes. System 400 includes a head-mounted indicator 430 and side-mounted cameras 432 and 434, which are side-mounted cameras with corresponding indicators 436.
And 438. The laterally mounted cameras 432 and 434 are also referred to herein as "video present" cameras.

As used herein, the term “video presence” extends the concept of video conferencing. Conventional video conferencing generally advocates communication-based interaction in which one set of attendees attempts to talk to another set of attendees on an audio / video link. Video presence extends the concept to include all activities that use audio / video links to create the feeling of "being" at remote locations. This is 1. Half of the attendees are at one end of the link and the other half are at the other end,
2. 2. Nights when attendees can see and hear each other, but are not actively speaking. 3. The situation where communication is less concentrated in the conversation, eg showing new babies, new furniture, etc. to the family. For example, a common experience in which one attendee “takes” another attendee on a trip. It may also include activities such as shared television viewing, where each attendee can see and listen to other attendees' reactions to a common broadcast event.

These are merely examples, and the concept of “video present” as used in this application refers to audio / audio
It may also include all other types of activities that are normally considered outside the definition of a video conference, where video links may be used.

Each indicator 430, 436, and 438 may be coupled with, for example, a suitable light source configured to provide indicator information to a user. For example, the indicator light may be turned on or off based on the particular state of the system 400, for example, indicators 430, 436, and 438 may be activated by corresponding head base and side mounted cameras, respectively. It may be turned on when the cameras are turned off and turned off when the cameras are turned off.

FIG. 4 thus includes a number of cameras, and anthropomorphic objects such as living things
"Hold" by a camera embedded in its head and a living-like object
Formed with two attached cameras that appear to be. In this arrangement, the embedded camera acts as a personality-based camera and the other camera acts as a traditional tracking camera that follows the speaker or other objects of interest. For example, by using two completely independent cameras,
The scene can be taken in a pseudo-professional manner, for example, one camera zooming for a framed shot of the speaker while another camera captures a wide established shot of the room.

When a wide shot corresponds to a live feed, the close-up camera can be moved, allowing for sophisticated directional techniques that mimic a live standard television director model.

System 400, as well as other systems described herein, provide audio, voice, or other processing and generation suitable for providing audio, voice, or other audibly perceptible information to a user. It may include a circuit. For example, a conventional acoustic module may be incorporated into a camera system to enable the generation of twitter, squeaks, and other noise.

The following is an exemplary interaction protocol that may be implemented in the above camera system 400 of FIG. 4 or other personality-based intelligent camera system according to the present invention. Note that this protocol is designed for use in applications such as video presence or video conferencing. The protocol assumes that a single camera is embedded in the eye of a living object and two video present cameras are mounted on each side, as shown in FIG. It is assumed that the video present camera includes a shutter that can be opened and closed on the lens of the camera. Further, for the purposes of this example, assume that the camera system interacts with a receiver and LCD display, as well as a set top box that includes the ability to establish a network connection, such as a telephone or Internet connection, for sending and receiving communications with remote devices. . These assumptions are for illustration purposes only and should not be construed as limiting the scope of the invention in any way. 1. Off position. The system indicates that it has been completely powered down and no video is being transmitted or processed. The eyelids close on the eye camera and the shutter closes on the two video present cameras. All indicator lights are off. 2. On / wake up. The eyelids open over the camera of the eye and the head is turned so that it faces the user. The shutters on the two video present cameras remain closed, indicating that no video is being transmitted. 3. Send video to receiver. When live video is sent to the receiver in the set top box, the shutter on the video present camera opens and the indicator light illuminates. The receiver and LCD display of the set top box may also be used to show textual information indicating whether the video is recorded on a VCR or other storage device or transmitted to a remote location, for example. Good. 4. Acknowledgment. If the command is heard and understood, the system nods quickly in the degree of tilt of the head. A confirmation chirp is generated. 5. Misunderstanding. If the command is misunderstood, the system tries to show the user how it can be better understood. The following are some examples of possible misunderstandings.

[0029]   (A) Not accepting commands-The system displays a confused look.

(B) A command is accepted, but it cannot be executed now, for example, in the case of "pointing to the right" when completely facing to the right-shaking the head "no" in the degree of freedom of pan, and Generates a negative beep.

(C) Audio too low or no gesture-the system shows a confused look and narrows the eyes narrow. 6. Dialing. When the system is dialing, the system displays the waiting behavior, looking up to the left, looking down to the center, looking up to the right, and looking down to the center to raise the eyebrows. Blink when you are there. 7. Connection. The system displays a smile and emits a successful twitter. The LCD on the set top box indicates the transmission status. 8. Connection bell / retry / waiting for retry / busy / unanswered call bell. The system displays an unpleasant and confused look. The unpleasant look is so slight that it does not look very angry. The LCD of the set top box provides textual information about the cause of the problem. 9. Receive calls from outside. Generates a noteworthy sound. The system raises the eyebrows and widens the eyelids while attracting the user's attention. 10. On and connected. There are several states when the device is on and connected and sending to the set top box. These indicate transitions between automatic and manual modes, focus of attention, etc. These are described in detail below. 11. auto mode. In this state, the system operates under automatic camera control in response to direct commands from the user. In this state, the system is expressionless, for example, the eyebrows are not raised or lowered, the eyelids are opened 7/8,
The mouth shows a smile and an unpleasant face. It may generate a low, almost inaudible and ambiguous automatic mode audio signal.

When in automatic mode, the embedded camera is generally pointed in the same direction as the currently present video present camera. However, when the automatic tracking system detects a pattern in the conversation, it allows the system to predict where the embedded camera will turn next, with the embedded camera pointing in that direction in front of the video presence camera. To be This allows a person outside the camera to respond to the person on the camera. As soon as the first response occurs, the embedded camera is turned to show the acknowledgment of the response. If the response continues and the second speaker becomes the primary speaker, the video presence camera will soon follow. If the response is only a confirmatory utterance, the embedded camera is aimed at the main speaker. 12. Focus of attention. The video present camera may be pointed to one side of the room,
The device may receive commands from users on the other side of the room. in this case,
The embedded camera is aimed at the user controlling the system and this direction indicates who controls the system.

13. Overload. If for some reason the camera system is unable to understand activity in the room, for example due to excessive movement, excessive dispersed voice, etc.,
The system may indicate that the user should control the camera. In this case, the head “spins” (coordinates pan and tilt) and the eyebrows move rapidly up and down. 14. Manual mode. In manual mode, the system indicates that it is paying close attention to the user's commands. The focus of attention of the embedded camera is directed towards the controlling user. The eyebrows are slightly raised, showing interest and attention, producing a slight smile. Generates special concentrated noise. 15. pointing. One of the capabilities of the manual mode is to allow the user to point at something in the room and follow the pointing at the camera. In this case, the device turns on the bright, focused red light of the head and points it towards the pointing location to guess the head, to the user controlling the system where the device thinks where the camera should be. Show. If this is incorrect, the user may adjust the points using feedback to adjust the location. System is short time,
For example, if at the same location for 3-5 seconds, the red light is turned off and the device nods slightly to indicate keeping the camera at this location.

The interaction protocol described above is exemplary only and is directed to the particular implementation of the camera system. Similar protocols suitable for use with other embodiments of the invention can be formed in a straightforward manner.

Although particularly suitable for use in home-based video conferencing, the exemplary systems 100, 200, 300, and 400 described above may also be used in a wide variety of other applications. For example, these systems may be used for video presence and video surveillance applications. Further, it should be apparent that the present invention may be used with image capture devices other than zoom cameras, including, for example, pan-tilt-zoom (PTZ) cameras. The term "camera" as used herein is therefore intended to include all types of image capture devices that may be used with personality based systems.

The elements or groups of elements of the exemplary camera system described above represent corresponding elements, which are otherwise computers, set-top boxes, etc., as well as portions or combinations of these and other processing devices. Note that it is okay. Moreover, the functionality of some or all of the individual components of systems 100, 200, 300, and 400 may be combined into a single device. For example, one or more of these system elements may be implemented as an application specific integrated circuit (ASIC) or circuit card to be incorporated into a computer, television, set top box, or other processing device.

The system software may be configured to be executed by a microprocessor, central processing unit, microcontroller, or any other data processing element that may be implemented in the system. Furthermore, the systems 100, 200, 3
00 and 400 are electronic memories, optical or magnetic disk-based memories, tape-based memories, and these and other types of storage devices for use in storing and executing corresponding system software. May be configured to include a combination or a part of. Further, the computer or other processing device used to execute the system software may be co-located with the anthropomorphic object and associated camera, or the signals provided may be camera, motor control. It may also be remotely located by sending it through a conventional remote control arrangement between the device etc. and the processing device.

It is again emphasized that the above embodiments of the present invention are exemplary only. For example,
The present invention can be used with a wide variety of different creature-like objects in many different combinations of controlled features. Further, while illustrated using a system with a single zoom camera, the present invention includes systems with multiple cameras, systems with PTZ cameras, and other types and arrangements of image capture devices. It is also applicable to a system that does. Further, the present invention may utilize many different types of techniques to detect and track objects of interest and provide feedback to the user.
These and numerous other embodiments within the scope of the appended claims will be apparent to those skilled in the art.

[Brief description of drawings]

1 is a block diagram illustrating a general implementation of a personality-based intelligent camera system according to an exemplary embodiment of the present invention.

2 is a block diagram illustrating a more specific implementation of the personality-based intelligent camera system of FIG.

3 is a block diagram illustrating another exemplary implementation of the personality-based camera system of FIG.

FIG. 4 is a diagram showing the appearance of an example of a personality-based camera system according to the present invention.

─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Lions, Damien M             Netherlands, 5656 Earth Ardine             Fen, Plov Holstran 6 F-term (reference) 5C022 AA12 AB66 AC21 AC27 AC77                 5C064 AA02 AC04 AC20 [Continued summary] Provides the interface.

Claims (16)

[Claims] 1. A camera associated with an anthropomorphic object having a controllable feature, and controlling the feature of the anthropomorphic object such that a particular form of the feature is indicative of the operating state of a camera system. A camera system coupled to the camera to generate a control signal. 2. The camera system of claim 1, wherein the camera comprises a zoom camera with adjustable zoom settings. 3. The camera system of claim 1, wherein the camera is embedded during a particular feature of the anthropomorphic object. 4. The camera system of claim 1, wherein the anthropomorphic object comprises a creature-like object. 5. The camera of claim 1, wherein the anthropomorphic object has a head and a body and the controllable feature of the anthropomorphic object has a facial feature associated with the head. system. 6. The camera system of claim 5, wherein the controllable facial feature comprises at least one of a controllable pair of eyebrows, a controllable pair of eyelids, and a controllable mouth. 7. The camera is embedded within the head of the anthropomorphized object, and the controller is further effective for generating control signals for controlling pan and tilt of the head. Item 5. The camera system according to Item 5. 8. The method of claim 1, further comprising a plurality of cameras, at least one of the cameras being associated with the controllable feature of the anthropomorphized object and at least one of the additional cameras being a video presence camera. The described camera system. 9. The camera system of claim 1, wherein the controller comprises system software and a motor controller, the system software generating control commands supplied to the motor controller. 10. The camera system of claim 9, wherein the motor controller drives one or more head control motors that control pan and tilt of the anthropomorphized object. 11. The controllable feature of the anthropomorphized object has at least one of a pair of controllable eyebrows, a pair of controllable eyelids, and a controllable mouth, the motor control The device is an eyebrow tilting motor that controls the tilt of the eyebrow, an eyebrow height motor that controls the height of the eyebrow, an eyelid motor that controls the up and down movements of the eyelid, and a mouth motor that controls the mouth. The camera system according to claim 9, which drives at least one of the above. 12. The controllable feature of the anthropomorphic object is performed at least partially on a display screen of the camera system, and the system software feeds a display screen driver to control the feature. 10. The camera system according to claim 9, wherein the camera system generates a controlled signal. 13. The camera system of claim 9, wherein the system software produces a control signal for the camera and processes a video signal produced by the camera. 14. The system software controls signals for the camera and the controllable feature of the anthropomorphic object according to a dialogue protocol that identifies a particular configuration for an operating state corresponding to the camera system. The camera system according to claim 9, wherein 15. A method of implementing a camera system, associating a camera with an anthropomorphic object having a controllable feature, wherein a particular form of the feature is indicative of an operating condition of the camera system. Generating a control signal controlling the feature of the anthropomorphic object. 16. A storage medium for storing one or more programs of a set of system software for controlling a camera system, wherein the one or more programs are executed by a processor of the camera system. A storage medium having a step of generating a control signal controlling a feature of an anthropomorphic object associated with a camera of the camera system such that a particular configuration of the camera system indicates an operating state of the camera system.