TWI545947B - Display device with image capture and analysis module - Google Patents

Description

Display device with image capture and analysis module

The present invention relates to display devices, and in particular to displays having image capture and analysis components.

This application claims the benefit of priority to the following U.S. Patent Applications: (1) Application filed on August 29, 2011, Serial No. 13/220, 612; (2) Application filed on November 11, 2011, Serial No. 13/ 294,977; (3) Application on September 2, 2011, serial number 61/530,872; (4) Application on November 11, 2011, serial number 13/294,964; (5) Application on September 2, 2011, Serial No. 61/530,867; and (6) filed on April 8, 2011, Serial No. 13/082,568.

People and businesses have always wanted to get more from their handheld mobile smart phone devices. Only audio conference calls have become available and have been widely used on mobile phones for many years. The videoconferencing on the mobile device is still in its infancy. It is desirable to have a mobile device to assist a user in understanding, interacting, planning or reviewing the details of an activity and other social and professional requirements while on the road. It is also desirable to have a mobile device that provides an interactive video conferencing experience that is superior to the currently available mobile video conferencing experience. Figure 1 illustrates an example of an active video conferencing environment. The mobile smart phone display of Figure 1 contains facial images of two participants in a video conference. A significant portion of the display contains a background image of the location of each participant that is not required for the video conference as desired. Specifically, participants may not even want to transmit background information to other parties in the call. people. This problem is advantageously solved by the U.S. Patent Application Serial Nos. 12/883, 183, 12/883, 191, and 12/ 883, 192 assigned to the same assignee.

With the right tools, advantageously, premium video conferencing will become available to anyone on the move who carries a smart phone device. The inventors of the present application have recognized that there is a particular need for a situation in which no light, low light, and/or uneven illumination conditions, and for persons and/or vehicles in which the mobile device is held, are in motion relative to the background. Premium mobile video conferencing solutions.

Mobile video conferencing participants are most likely to be in an environment with low or uneven lighting conditions, because if the participant has an opportunity to utilize the default video conferencing environment for the call, they may not choose to use it. A smart phone is used for the call. The display illustrated at Figure 2 contains a portion that is substantially neither uniformly nor sufficiently illuminated at the lower left corner. Uneven and low light conditions can cause undesired effects of the user's facial image being displayed in a video conference, as small details such as a person's smile or communicative facial expression often make video conferencing extremely desirable. However, such small details are often difficult to resolve in low light or uneven light conditions. Accordingly, embodiments for providing improved face of a participant displayed in an active video conference under such low or uneven illumination conditions are set forth below.

An active video conferencing participant may also be walking, driving, or otherwise moving during a call, also because if the participant is in a static environment, such as a conference room, office or computer desk, or even has a laptop One of the computer's sitting positions, with special pre-set lighting, a comfortable seat, and a webcam fixed to ground or on a desk, may not use a smart phone for the call. While the participant attempts to hold the phone stationary relative to its face, the background will typically be fast moving. Therefore, in this paper, by focusing on the reference Embodiments that reduce or eliminate the processing and/or transmission of background images to effectively use the limited computing resources of a mobile smart phone environment.

Electronic display devices are commonly used as televisions or with computers to display two-dimensional images to a user. In the case of computing, the electronic display device provides visual interaction with one of the computer's operating systems.

In most cases, a user provides input to a computer by using an external input device, more commonly by a keyboard and a mouse or trackball combination. Recently, however, touch screen devices (e.g., capacitive or resistive touch screens) built into electronic displays have become popular as an alternative to providing input to a computing device or television display.

Electronic displays have evolved from large and heavy cathode ray tube monitors (CRTs) to lighter and thinner liquid crystal displays (LCDs) and organic light emitting diode (OLED) displays. Many displays have now incorporated additional features such as cameras and Universal Serial Bus (USB) to improve computing or TV experience.

A computer user typically spends most of his time interacting with a computer. For example, an office worker can spend hours in front of a display driven by a desktop or other computer. If the user uses the computer in an ergonomically unsuitable manner, such as viewing the display from a non-optimal location and/or other adverse conditions that are calibrated by user behavior, it can adversely affect its health. Various techniques have been proposed for ensuring ergonomically suitable computers, but there is still room for improvement.

The present invention provides a handheld camera-capable video conferencing device that includes a housing configured to be held in a user's hand. A processor and memory system are contained within the housing. The memory embeds a code for programming the processor, including a video conferencing component, A face detection component, a face recognition component, and an associated image processing component. The memory further contains facial information associated with one or more specific user identities. The device also includes a display built into the housing and configured to be viewable by a user during a video conference. A camera is also built into the housing and configured to capture images of the user while viewing the display. The camera includes an infrared (IR) light source and an IR-sensitive image sensor for capturing images of the user in low or uneven light conditions (or both) to enable the face detection component to detect Measure the face of the user. The facial recognition component is configured to associate a particular identity of a user to a detected face. The image processing component is configured to replace facial data of the detected face with facial data stored in the memory according to the particular identity of the user to enhance low light or uneven light conditions (or both) One of the detected faces is captured and transmitted to a far-end video conferencing participant.

The facial data may include chromaticity data or illuminance data or both. The face detection component or face recognition component or both may include a classifier trained to detect a face or recognize a face under low light or uneven light conditions (or both). The IR light source can include one or more IR LEDs coupled to the housing and disposed to illuminate a user's face during a video conference. The memory can include a face tracking component to track the detected face to enable the device to transmit an approximately continuous video image of the user's face during the video conference.

The memory can include a component to estimate a distance to the user's face and control one of the IR light source output powers based on the estimated distance. The estimate of the distance can be determined using autofocus data and/or can be based on the size detected by one of the user's faces. The memory can include a component to determine the position of a user's face relative to one of the devices and to control the direction of one of the IR light sources that illuminate the face of the user.

The present invention provides another handheld camera-capable video conferencing device that includes a housing configured to be held in one or both of a user's hands, and a processor and memory contained within the housing . The memory embeds a code for programming the processor, including a video conferencing component and a foreground/background segmentation component or a combination thereof. A display is built into the housing and is configured to be viewable by a user during a video conference. A camera is built into the housing and configured to capture images of the user while viewing the display. A communication interface transmits audio/visual data to a remote video conferencing participant. The foreground/background segmentation component is configured to extract user identity data without background material by identifying different motion vectors of the foreground versus background material.

User identity data can include facial information. The foreground/background segmentation component can be calibrated to match specific user identity data as foreground material.

The present invention provides another handheld camera-capable video conferencing device that includes a processor configured to be held in one or both of a user's hands and a processor and memory contained within the housing. The memory embeds a code for programming the processor, including a video conferencing component and a foreground/background segmentation component or a combination thereof. A display is built into the housing and is configured to be viewable by a user during a video conference. A camera is built into the housing and configured to capture images of the user while viewing the display. A communication interface transmits audio/visual data to a remote video conferencing participant. The foreground/background segmentation component is configured to extract user identity data without background material by matching the detected facial data as foreground data.

The camera can include an infrared (IR) light source and an IR sensitive image sensor for capturing images of the user in low or uneven light conditions (or both) to enable a face detection component to Detecting a user's face. The image processing component can replace the facial data of the detected face with the facial data stored in the memory according to the specific identity of the user to enhance the low or uneven light. One of the detected faces of the situation (or both) is captured and transmitted to a far-end video conferencing participant.

The memory can include a face tracking component to track the detected face to enable the device to transmit an approximately continuous video image of the user's face during the video conference.

The specific user identity data can include an image of a detected face. The information may include a neck, a partial torso or a shirt or one or two arms, or portions thereof or a combination thereof.

The memory can contain facial information associated with one or more particular user identities such that the particular user identity data is extracted based on the facial data in the matching memory.

The present invention provides a method for dynamically changing a display parameter, comprising: detecting a user parameter of a user positioned in front of an electronic display, and automatically adjusting one of the displays based on the detected user parameter The user prefers or displays an indicator.

In some embodiments, the user parameter is one of the ages of the user.

In another embodiment, an amount of displayable content is added when the user is old. In another embodiment, the amount of displayable content is reduced when the user is a child or a minor.

In some embodiments, the privacy setting is added when the user is a child or a minor. In other embodiments, privacy settings are reduced when the user is an adult or an elderly person.

In some embodiments, the user parameter is a distance from the user to the electronic display. In one embodiment, a distance indicator is displayed when the distance is less than an optimal distance.

In one embodiment, the user parameter is a time when the user is positioned in front of the display. In some embodiments, a time indicator is displayed when the time is greater than a predetermined time limit.

In one embodiment, the user parameter is a head angle. In some embodiments, an ergonomic indicator is displayed when the head angle is not suitable.

In some embodiments, the user parameter is a ambient light level. In other embodiments, the user parameter is a ambient light level and a pupil closure percentage.

In one embodiment, the display is automatically brightened if the ambient light level is low and the pupil closure percentage is high. In another embodiment, the display is automatically dimmed if the ambient light level is low and the pupil closure percentage is low. In an alternate embodiment, the display is automatically illuminated if the ambient light level is high and the pupil closure percentage is high. In yet another embodiment, the display is automatically dimmed if the ambient light level is high and the pupil closure percentage is low.

The method of claim 13 wherein the display is automatically dimmed or brightened based on the detected ambient light level.

In some embodiments, the user parameter is an unknown user.

In one embodiment, the display is dimmed or turned off when an unknown user is detected. In another embodiment, the display is locked and an security indicator is displayed on the display when an unknown user is detected. In some embodiments, the security indicator informs the unknown user that access to the display is denied.

In some embodiments, the detecting step includes detecting user parameters by means of a sensor disposed on or near the electronic display. In one embodiment, the sensor includes a camera.

In some embodiments, the electronic display includes a computer monitor. In other embodiments, the electronic display includes a cellular telephone.

In one embodiment, the automatic adjustment step includes processing the user parameters with a controller and automatically adjusting user preferences or display indicators on the display based on the detected user parameters.

There is also provided an electronic display comprising: a sensor configured to detect a user parameter of a user positioned in front of the display; a screen configured to place text Or the image is displayed to the user; and a processor configured to adjust a user preference or display an indicator based on the detected user parameter.

In one embodiment, the user parameter is age.

In another embodiment, the user parameter is a distance from the user to the electronic display.

In some embodiments, the user parameter is one of the user's head angles.

In one embodiment, the user parameter is an unknown user.

In some embodiments, the user parameter is a ambient light level.

In some embodiments, the sensor includes a camera.

In one embodiment, the electronic display includes a computer monitor.

In another embodiment, the electronic display includes a cellular telephone.

In some embodiments, the electronic display includes a tablet.

The present invention provides a method for dynamically adjusting a display parameter, comprising: determining whether a user's face is positioned in front of an electronic display by means of a sensor, and if the user's face is not positioned in front of the electronic display, The sensor monitors the face of the user for a predetermined period of time, and if the face of the user is not positioned in front of the electronic display during the predetermined time period, a power saving routine on the electronic display is initiated.

In some embodiments, the power saving routine includes dimming the display.

In other embodiments, the power saving routine includes powering down the display.

In some embodiments, after powering down the display, the method includes energizing the sensor of the electronic display to monitor the body of the user.

Some embodiments of the method further include: determining, by means of the sensor, whether the user's eyes are staring toward the electronic display, if the user's eyes are not staring toward the electronic display, Using the sensor to monitor the eyes of the user staring towards the display during a predetermined period of time, and to initiate power savings on the electronic display if the user's eyes are not staring toward the electronic display during the predetermined time period Regular style.

In some embodiments, the predetermined period of time is user adjustable.

In other embodiments, a darkening percentage is user adjustable.

The present invention provides an electronic display comprising: a sensor configured to detect a face of a user positioned in front of the display; and a processor configured to be in a predetermined period of time A power saving routine is implemented during which the user's face is not positioned in front of the display.

In one embodiment, the sensor includes a camera.

In other embodiments, the electronic display includes a computer monitor.

In some embodiments, the electronic display includes a cellular telephone.

The present invention provides a method for dynamically changing a display parameter, comprising: detecting a user parameter of a user positioned in front of an electronic display, and automatically adjusting a text on the display based on the detected user parameter. One size.

In some embodiments, the user parameter is one of the ages of the user.

In one embodiment, the font size is increased as the user is old. In another embodiment, the font size is reduced when the user is a child or a minor.

In some embodiments, the user parameter is a distance from the user to the electronic display.

In one embodiment, the font size is increased when the distance is greater than an optimal distance. In another embodiment, the font size is reduced when the distance is less than an optimal distance.

In some embodiments, the font size changes dynamically on the fly as the distance from the user to the electronic display changes.

In other embodiments, the font size decreases instantaneously as the distance from the user to the electronic display becomes smaller, wherein the font size increases instantaneously as the distance from the user to the electronic display becomes greater.

In some embodiments, the detecting step includes detecting user parameters by means of a sensor disposed on or near the electronic display. In one embodiment, the sensor includes a camera.

In some embodiments, the electronic display includes a computer monitor. In other embodiments, the display includes a cellular telephone.

In one embodiment, the automatic adjustment step includes processing the user parameters by means of a controller and automatically adjusting the font size of the text on the display by means of the controller based on the detected user parameters.

The present invention also provides a method for dynamically changing a display parameter, comprising: detecting a user parameter of a user positioned in front of an electronic display, and automatically adjusting one of the displays based on the detected user parameter The size of the illustration.

In some embodiments, the user parameter is one of the ages of the user.

In one embodiment, the graphical size is increased as the user is old. In another embodiment, the illustrated size is reduced when the user is a child or a minor.

In some embodiments, the user parameter is a distance from the user to the electronic display.

In one embodiment, the graphical size is increased when the distance is greater than an optimal distance. In other embodiments, the illustrated size is reduced when the distance is less than an optimal distance.

In some embodiments, the detecting step includes detecting the user parameter by means of a sensor disposed on or adjacent to the electronic display. In one embodiment, the sensor includes a camera.

In some embodiments, the electronic display includes a computer monitor. In other embodiments, the display includes a cellular telephone.

In some embodiments, the automatic adjustment step includes processing the user parameters with a controller and automatically adjusting the font size of the text on the display by the controller based on the detected user parameters.

In other embodiments, the font size changes dynamically from the user to the electronic display and changes dynamically on the fly.

In one embodiment, the font size decreases instantaneously as the distance from the user to the electronic display becomes smaller, wherein the graphical size increases instantaneously as the distance from the user to the electronic display becomes greater.

The present invention provides an electronic display comprising: a sensor configured to determine a user parameter of a user positioned in front of the display; a screen configured to display text or images to a user; and a processor configured to adjust a size of the text or image based on the determined user parameter.

In some embodiments, the user parameter is age.

In another embodiment, the user parameter is one distance from the user to the electronic display.

In some embodiments, the sensor includes a camera.

In another embodiment, the electronic display includes a computer monitor. In an additional embodiment, the electronic display includes a cellular telephone. In some embodiments, the electronic display includes a tablet.

A display device having an ergonomic sensor can be used, the ergonomic sensor including an imaging device that interfaces to the processing hardware to obtain and analyze image data of a user of the display device. The ergonomic sensor can be pre-configured with information indicative of the ergonomic use of the display device such that the user's image can be analyzed with minimal or no user calibration or settings. Alternatively, the ergonomic sensor can provide imaging data for analysis to provide immediate feedback, such as warnings or suggestions when the user's behavior exceeds one of the ergonomic uses of the display device. In some embodiments, the ergonomic sensor is integrated with the display device, but in other embodiments, a separate component or pre-stored imaging device can be used.

This example is not intended to limit the subject matter, but rather to provide a brief overview. Additional examples are set forth in the embodiments below. The objectives and advantages of the subject matter can be determined based on one or more teachings herein based on the description of the embodiments and/or the practice.

900‧‧‧ display

902‧‧‧ screen

904‧‧‧ sensor

906‧‧‧Users

1000‧‧‧ display

1008‧‧‧ Tips or symbols

1108‧‧‧ Tips

1210‧‧‧User timer indicator or symbol

1306‧‧‧Users

1312‧‧‧ Ergonomic indicators or symbols

1400‧‧‧ display

1402‧‧‧ screen

1406‧‧‧Users

1414‧‧‧ second user

1700‧‧‧ display

1706‧‧‧Users

1800‧‧‧ display

1802‧‧‧ screen

1804‧‧‧ sensor

1806‧‧‧Users

1906‧‧‧Users

1908‧‧‧ font

2006‧‧‧Users

2010‧‧‧System icon

2100‧‧‧ display

2106‧‧‧Users

2108‧‧‧ font

2110‧‧‧ icon

2112‧‧‧Distance

2202‧‧‧ Ergonomic sensor module

2204‧‧‧ display

2204A‧‧‧Structural components

2205‧‧‧Computer System/Computer

2206‧‧‧Users

2208‧‧‧Sensor/Image Sensing Device

2210‧‧‧Processing components

2212‧‧‧Input/Output Interface

2214‧‧‧ memory

2216‧‧‧ Ergonomic analysis routine

2217‧‧‧ Ergonomic analysis routine

2218‧‧‧Feedback/Warning Message

2220‧‧‧ memory

2302‧‧‧ Ergonomic sensor module

2304‧‧‧Display

2402‧‧‧ Ergonomic sensor module

2404‧‧‧Display

2405‧‧‧ computer

Figure 1 illustrates an example of an active video conferencing environment.

Figure 2 illustrates an example of one of the images of one of the faces taken under low light conditions (e.g., 30 lux).

3 illustrates an example of a handheld device having one or more infrared light emitters (eg, a circle of infrared light emitters).

Figure 4 illustrates a single infrared light emission at 40 cm without any external visible light source (i.e., the facial system is completely dark at 0 lux without an infrared light emitter) The device detects one of the faces.

Figure 5 illustrates an example of one of the images taken at a low light condition (e.g., 30 lux) in which the skin tone is applied as a piece of skin tone from one of the calibration images taken at a higher light level.

Figure 6 illustrates an example of one of the calibration images taken at a higher light level than the image of Figure 5, and ideally at an optimal light level.

7A-7B illustrate a sequence of two images taken using a handheld camera having different backgrounds but having similar facial positions indicated by similar contour shapes in both.

Figures 8A-8B illustrate a sequence of two images of Figures 7A-7B, but this time with motion vector arrows indicating the direction and magnitude of motion of the background to the foreground facial objects in an exemplary mobile video conferencing environment.

Figure 9 is an illustration of one of the users in the field of view of a display.

Figure 10 is an illustration of one of the children's users in the field of view of a display.

11A-11B are illustrations of different users in the field of view of a display.

Figure 12 is an illustration of one of the users in the field of view of a display having a user timer.

Figure 13 illustrates an ergonomic indicator of one of the users in the field of view of a display.

Figure 14 is an illustration of one of the privacy settings when two users are detected in the field of view of a display.

Figure 15 illustrates one of the indicators when a user is not recognized by a display.

Figure 16 illustrates a display that illuminates only one of the sections of the display corresponding to a user's gaze.

Figure 17 illustrates one of the distance indicators for a user in the field of view of the display.

Figure 18 is an illustration of one of the users in the field of view of a display.

19A-19B illustrate adjusting the size of text on a display based on the age of the user.

20A-20B illustrate the size of the graphic on a display based on the age of the user.

21A-21B illustrate the adjustment of the size of text and/or icons on a display based on the distance of the user from the display.

Figure 22 is a block diagram showing an exemplary ergonomic sensor module.

Figure 23 is a diagram showing one example of an ergonomic sensor module integrated into a display.

Figure 24 is a diagram showing one example of an ergonomic sensor module used external to a display.

Figure 25 is a flow chart showing one of the steps in an exemplary processing method performed when an ergonomic sensor module is used.

Figure 26 is a diagram showing one example of a yaw angle with respect to a user of a display.

Figure 27 is a diagram showing one example of a pitch angle relative to a user of a display.

The good or natural illumination conditions used to capture digital images make an object appear to be uniformly illuminated from all directions without excessive or too little light. Poor lighting conditions can include low light, uneven light, and no light conditions. The uneven light includes light that is oriented such that one of the items on one side is slightly brighter on one side than the other side (eg, left-right, up-down, along a diagonal line, etc.), or only One or more shades of light somewhere on the object. Figure 2 illustrates the inclusion in a low light In the case (for example, 30 lux), one of the images of one of the faces is taken. The facial system in the image illustrated in Figure 2 is both dimly lit and also unevenly illuminated, i.e., one side of the face appears much darker than the other side. The area containing the forehead, neck, one ear, nose and one cheek is somewhat discernible, such as one of the eyes, mouth and chin, torso or shirt, hair and two ears. dark.

In general, a low light condition allows an item such as a face to be or may not be detected, and object/face tracking may be difficult to lock (if present), and in any event the image material contains less information than desired. For example, in low light conditions, such as in Figure 2, only certain areas of an object may be discernible while other areas are indistinguishable. In another example, one or more parameters may be insufficiently determinable, such as, for example, illumination, color, focus, hue reproduction, or white balance information, or such as a participant being smiling or blinking or being partially Information that is occluded or otherwise shaded. Some descriptions of poor illumination conditions and certain solutions for disposing of them are found in US20080219517 and US Pat. No. 2,110,102, 553, the disclosure of each of which is incorporated herein by reference.

In the absence of light, objects (eg, a face) are even indistinguishable or undetectable. In the absence of any visible light, a person is visually indistinguishable from any area or parameter of the object. The present invention provides an applicant's advantageous device comprising an infrared source and sensor as set forth below in accordance with certain embodiments and with reference to Figures 3 through 4 to enhance an action video conference when the illumination condition is less than optimal.

Images captured under low or uneven illumination will typically have shaded/dark areas mixed with brighter areas and are generally less pleasing to the eye than those captured under normal or optimal light conditions. In fact, unlike the images captured in a professional studio, most of the pictures taken by people with smart phones and handheld consumer digital cameras are less than the most Shoot in various locations with good lighting conditions. In some embodiments, it may be advantageous to store a calibration image previously captured by the user (eg, under a preferred light condition, such as at normal or even optimal light level) and then used to enhance Reconstructing, or even replacing, certain image regions, features, parameters or properties that are not sufficiently discernible or desirable within the captured image of one of the reversed illuminations, such as skin tones, occlusions or Shadow features, color balance, white balance, exposure, etc.

Reproducibly copying certain information from a current original image (eg, face size, eye and lip movement, focus, hue, color, orientation, or relative exposure or overall exposure) to use skin tones (see, for example, US Pat. No. 7,844,076, 7, 460, 695 and 7, 315, 631, which are incorporated by reference, and/or from one or more other properties of the calibrated image to provide an appearance as close as possible to a natural face. US 20080219581, US Pat. No. 20,110,102, 638, and US Patent No. 20090303343 are hereby incorporated by reference herein inco Work together and enhance further solutions for these low-light images. In some embodiments, the background may also be replaced with an artificial background, or with one of the images captured from one of the images with better illumination, or with a blurred or arbitrary background.

The various embodiments described herein relate to the use of information and/or information from previously stored images to improve viewing by any participant in a conference participant (regardless of where they are located) at either end of a video conference Skin tone and/or other properties of items such as facial images. Effective resource usage of the handheld device is also illustrated, for example, by transmitting only audio and foreground facial data with or without surrounding material, and in particular, according to embodiments described herein, does not include background material that is different from foreground data.

In some embodiments, mobile video conferencing has been advantageously enhanced in particular with respect to images captured in low light conditions and non-uniform light conditions. In some embodiments, an array of IR-emitting diodes (LEDs) that are well-positioned by emitting infrared (IR) light (eg, one turn) can be utilized to improve IR in the low light/none Detection of a user's face under light conditions. 3 illustrates an example of one hand-held device that includes one or more infrared light emitters (eg, a circle of infrared light emitters). In other embodiments, only a single IR emitter is used, or two IR emitters are placed left-right or up-down on one side of the device, or four IR emitters are provided, included in the device One on each side. Various configurations are possible, including placing opposing IR emitters on either of the six sides of the device, and that can be fixed or movable relative to the device.

These IR LEDs can have a controlled current and thus have an output power that is controlled in dependence on parameters such as the distance of the face from the handheld device. This feature can also advantageously reduce the power usage of existing flash-based cameras. In a further embodiment, the IR emitter is initially focused in a seek mode while maintaining constant focus on the face once the face tracker module of the device is tracked to the face.

Illuminating the face with one or more infrared light LEDs provides improved detection of a face that is a short distance from a device equipped with an IR sensor that is reflected from the face or other target object IR light. Figure 4 illustrates the use of a single infrared light emission at 40 cm without an external source of visible light (i.e., the face is completely dark at 0 lux without an infrared light emitter) The device detected one of the faces.

Skin tones can be reconstructed using a calibration image previously captured by the user (following specific instructions) at the optimal light level. Information from the current original image (such as face size and eye and lip movement) can be closely copied to use skin color from one or more of, for example, calibrated images Tune to provide as close as possible to a natural facial appearance. Figure 5 illustrates an example of one of the images taken at a low light condition (e.g., 30 lux) in which the skin tone is applied as a piece of skin tone from one of the calibration images taken at a higher light level. Similarly, the background can be replaced with an artificial background and/or one of the images taken with good light conditions. Figure 6 illustrates an example of one of the calibration images taken at a higher light level than the image of Figure 5 and ideally taken at an optimal or normal light level.

When a mobile video conferencing participant uses a handheld device, the motion of the background is typically greater or faster than the motion of the foreground relative to the camera lens/sensor. The foreground may include a face of a participant with or without any peripheral area such as hair, neck, torso, shirt, arm, hat, scarf or other peripheral items or areas, see US 20110081052 and US 20070269108, which are incorporated by reference. The manner is incorporated herein. Participants usually try to keep their faces still relative to the camera lens/sensor. If the participant is successful in this effort, the participant and the camera lens/sensor may move statically or substantially together or on average, while the background may be static, or alternatively relative to the camera lens/sensory The detector moves relatively quickly.

By distinguishing objects moving rapidly relative to the camera from objects that move significantly slower, in accordance with some embodiments the apparatus is capable of splitting the foreground objects and regions and background objects and regions in one of the images captured by the device. According to some embodiments, by transmitting only the foreground to one or more other video conferencing participants, the device is more resource efficient and does not need to otherwise process the image of the blurred moving background but simply discards it. can. Another option is to transmit a blurred background image without further processing, as it may be desirable to transmit only a blurred background to, for example, maintain privacy (see U.S. Patent Serial No. 12/883,192 to the same assignee). , which is incorporated herein by reference) and/or avoids the expense of processing resources in terms of background information.

Figures 7a-7b illustrate a sequence of two images taken using a handheld camera having different backgrounds but having similar facial positions indicated by similar contour shapes in both. Since it may not be desirable to transmit background information or to view background information by one of the participants, some participants, or even all of the participants, the device may be advantageous in other image processing that may otherwise be expected to provide viewable images. Abandonment would otherwise involve continuously providing resource-intensive calculations that alter the background image (including deblurring, color and white balance enhancement, and focus enhancement). Alternatively, their image enhancements are better spent on the actual desired image material, for example, the conference participant's face.

Figures 8a-8b illustrate the sequence of the two images of Figures 7a through 7b, but this time with motion vector arrows indicating the direction and magnitude of motion of the background to the foreground facial object in an exemplary mobile video conferencing environment. The camera can use motion vector information to identify the foreground and background.

In an mobile video conferencing environment, the user typically keeps the mobile device stable and oriented at its face as much as possible. However, as both the device and the user can move in this environment, the background will typically change rapidly from frame to frame. Thus, the foreground (eg, face) will generally be relatively stable compared to the background, except where the person is stationary, in which case both the background and the foreground will be approximately equally stable. Segmentation of variable background data from foreground data greatly enhances the efficiency of the device in an active video conferencing environment.

This understanding of the difference between the foreground and the background is used to distinguish the background from the stability of the foreground and also to use specific information about the user (such as facial recognition, sputum correction, skin tone, eye color, facial beautification or user). The particular other user selects image processing or automatic image processing because the mobile device is primarily a single user device), providing a background-to-foreground discriminant algorithm based on an embodiment that is effective and suitable for the needs of the mobile video conferencing. The following items are incorporated herein by reference, to describe various examples of some of these techniques: by US US 20100026831, US 20090080796, US 20110064329, US 20110013043, US 20090179998, US 20100066822, US 20100321537, US 20110002506, US 20090185753, US 20100141786, US 20080219517, US, filed on December 2, 2010, by the same assignee. 20070201726, US 20110134287, US 20100053368, US 20100054592 and US 20080317378 and US Serial No. 12/959,151. Advantageously, motion vectors of different objects/pixels in the image are used to aid in decision making, for example, whether an object belongs to the background or foreground, and thus resources can be efficiently allocated.

In some embodiments, one of the foreground regions can be used to speed up facial recognition, face tracking, or face detection by focusing only on the candidate region. In addition, since a typical mobile device is primarily used by a single user, previously calibrated images can be used to speed up facial recognition, face detection, and image enhancement and enhance background separation, such as separation of foreground and non-face data.

In some embodiments, once the foreground (eg, face range) is detected, the background of a user can be used instead of the background. This provides an effective implementation of one of the contexts in the case of an active video conferencing application using a valid foreground/background separation method.

Since the relatively unimportant background information will generally change more quickly than the face of a known user, the background information will additionally involve more bandwidth usage for transmission to the other end of the mobile video conference room. Advantageously, efficient use of the frequency bandwidth is provided herein for mobile video conferencing by detecting and transmitting only compressed faces and/or other foreground information.

Once the foreground is detected and transmitted, the background can be replaced with the preferences of one of the users at the receiving end or other automatically selected material. One of the background substitutions provided by this advantageous separation method is effectively implemented. In addition, improved compression performance is provided by maintaining a constant facial skin tone even when the lighting conditions change. This improves the frequency bandwidth efficiency of mobile video conferencing. In some embodiments, the background-to-face or other foreground distinction is based on analyzing the difference between the motion vectors of the objects in the images captured in the mobile video conferencing environment. Other foreground/background segmentation techniques may be used in place of or in conjunction with this technique, as described in several of the patent applications incorporated herein by reference.

U.S. Patent Nos. 7,953,287 and 7,469,071 are incorporated herein by reference in their entirety in their entirety in the extent of the disclosure of the disclosure of the present disclosure. U.S. Patent Nos. 7,868,922, 7, 912, 285, 7, 957, 597, 7, 796, 822, 7, 796, 816, 7, 680, 342, 7, 606, 417, and 7, 692, 696, and U.S. Patents 7,317,815 and 7,564,994 relate to facial tools and facial imaging workflows and are also incorporated by reference. US Patent Nos. 7,697,778 and 7,773,118 and US 20090167893, US 20090080796, US 20110050919, US 20070296833, US 20080309769, and US 20090179999, and USSN 12/941,983 are incorporated herein by reference, and incorporate A description of the embodiment of the uniform light compensation. US 20080205712 and US 20090003661 are incorporated by reference, which are incorporated herein by reference in its entirety in its entirety in the the the the the the the the Description of embodiments related to illumination detection of a classifier chain.

[Fig. 9 to Fig. 17]

The present invention provides techniques and methods for adjusting user preference settings based on parameters or conditions detected by an electronic display system or monitor. In some embodiments, the display system can detect and/or determine an age of a user. In another embodiment, the display system can detect and/or determine a distance between the user and the display. In still another embodiment, the display system can detect and/or determine ambient light or a combination of the age or distance conditions detected separately or in combination with the above detected age or distance conditions. The amount of light on the user's face. In some embodiments, the display system can identify a user's face and can additionally identify a user's gaze or determine the user's pupil diameter.

Any number of user preferences or display settings can be dynamically adjusted based on parameters or conditions detected or determined by the display. For example, in one embodiment, the displayable content or user privacy settings can be adjusted based on the age detected by the user. In another embodiment, the type of content or file type that can be displayed can be defined based on the age detected by the user. In some embodiments, a particular user is individually identified and the displayable content and/or privacy settings can be individually adapted to the particular individual identified by the display. In some embodiments, a user timer can determine when a predetermined time limit is exceeded and indicate to the user to stop use of the display. Additionally, the display can be indicated to the user when the user is sitting or tilting in one of the ways that can cause injury, pain, or discomfort. In some embodiments, the brightness of the screen may be based on the age detected by the user, the diameter of the pupil of the user, the ambient light around the user or on the face of the user, the distance between the user and the display, or all Automatic adjustment of any logical combination of the foregoing conditions.

9 illustrates a display 900, such as a computer monitor, a television display, a cellular telephone display, a flat panel display, or a laptop display having a screen 902 and a plurality of sensors 904. The sensors can include, for example, an imaging sensor (such as a camera including a CCD or CMOS sensor), a flash or other form of illumination, and/or configured to detect or enable Any other sensor that the object is imaged (such as ultrasound, infrared (IR) light, thermal sensors, or ambient light sensors). The sensor can be placed on or incorporated into the display, or another selection, the sensor can be integrated into the display. Any number of sensors can be included in the display. In some embodiments, a combination of sensors can be used. For example, in one embodiment, a camera, a flash, and an infrared sensor can all be included in a display. It should be appreciated that any sensor combination or any number of sensors can be included on or near the display. Such as As shown in FIG. 9, user 906 is shown positioned in front of display 900 within the detection range or field of view of sensor 904.

Various embodiments relate to a camera mounted on or near a display that is programmed to detect, track, and/or identify a facial or partial facial or a facial region (such as one or two eyes, or a mouth) A region or a processor coupled with a facial expression or expression (such as a smile or blink). In some embodiments, the processor is integrated into the display or disposed on the display. In other embodiments, the processor is separate from the display. The processor can include memory and software configured to receive signals from the sensors and process the signals. Some embodiments include sensing the characteristics of a user or a user with the sensors and determining parameters related to the face based on, for example, orientation, pose, tilt, hue, color balance, white balance, relative exposure, or overall Exposure, face size, or size of the face area (including the size of the eye or the area of the eye such as the pupil, iris, sclera, or eyelid), a focus condition, and/or a distance between the camera or display and the face. In this regard, the following items are hereby incorporated by reference, which are incorporated herein by reference in its entirety in its entirety in the the the the the the the the Application No. 13/035,907, No. 12/883,183, filed on September 16, 2010, and No. 12/944,701, filed on November 11, 2010, each of which is assigned to the same assignee And U.S. Patents 7,853,043, 7,844,135, 7,715,597, 7,620,218, 7,587,068, 7,565,030, 7,564,994, 7,558,408, 7,555,148, 7,551,755, 7,460,695, 7,460,694, 7,403,643, 7,317,815, 7,315,631 and 7,269,292.

A number of techniques can be used to determine the age of a user sitting in front of a display or monitor. In one embodiment, the age of the user may be determined based on the size of the user's eyes, the size of the user's iris, and/or the pupil size of the user.

Depending on the sensor included in the display, one of the images or other data of the user, such as one of the users, may be obtained by the display including the sensors. Information about the acquired data (including distance to the user or object, aperture, CCD or CMOS size, focal length and depth of field of the lens) can be recorded on the image or recorded along with the image. Based on this information, the display can determine the potential size range of one of the eye, iris, pupil or red-eye area (if a flash is used).

In this case, variability is not only for different individuals, but also based on age variability. Fortunately, in the case of the eye, the size of the eye is relatively constant as the person grows from a baby to an adult. This is the reason for the impact of the "big eyes" seen in babies and young children. The average eyeball measurement of the infant was approximately 19.5 mm from front to back and, as described above, grew to an average of 24 mm during the life of the person. Based on this data, in the case of eye detection, one of the objects in the field of view (which can be a pupil, the portion of the pupil of the pupil) is limited in size, allowing for a variability of 9 mm ≦ iris size ≦ 13 mm.

Thus, by detecting or determining the size of a user's eyes relative to other facial features by means of the sensor 904, the age of the user can be calculated. Further details regarding methods and processes for determining the age of a user based on eye, iris or pupil size can be found in U.S. Patent No. 7,630,006 to De.

In another embodiment, the person's face can be detected and classified according to the age of the subject (see, for example, U.S. Patent No. 5,781,650 to Lobo et al.). A number of image processing techniques can be combined with anthropometric data on facial features to determine an estimate of one of the age categories of a particular facial image. In a preferred embodiment, anthropometric data within a digital image is used to verify facial features and/or eye regions. The opposite approach can also be employed and it can involve a probabilistic reasoning, also known as Bayesian Statistics.

In addition to determining the age of the user, the display can also determine or detect the distance from the user to the display, gaze (or more specifically, the position and direction of the user's gaze), the posture of the user or the amount of head tilt, and Contains the ambient light and the amount of light on the user's face. Details on how to determine the distance of the user from the display, the user's gaze, head tilt or direction, and illumination level are also found in U.S. Patent No. 7,630,006 to DeLuca et al., and U.S. Patent Application Serial No. 13/035,907.

The distance can be easily determined by using an IR sensor or an ultrasonic sensor. In other embodiments, one of the images of the user can be captured by a camera and can be determined by comparing the relative size of the detected face with the detected features of the face, such as the eyes, nose, lips, and the like. The distance of the user. In another embodiment, the relative spacing of features on the face can be compared to the detected size of the face to determine the distance of the user from the sensor. In still another embodiment, the focal length of the camera can be used to determine the distance of the user from the display, or another selection system that can be related to the detected features of the user (such as the size of the face or the relative size of the facial features). ) Combine to determine the distance of the user from the display.

In some embodiments, determining the user's gaze can include acquiring and detecting a digital image comprising at least a portion of one face (including one or two eyes). At least one of the eyes can be analyzed and a degree of coverage of one of the eyeballs can be determined. Based on the degree of coverage of the eyeball against the eyeball, one of the vertical gaze of the eye can be determined to be approximately the direction. Analysis of at least one of the eyes can further comprise determining one of the horizontal gaze directions. In some embodiments, the technique includes initiating a further action or initiating a different action (or both) based at least in part on the determined direction of the horizontal gaze. The analysis of the one or two eyes can include spectrally reflecting one of the light from the one or both eyes. This can include analysis of one of the visible on at least one side of the iris Scleral volume. In other embodiments, this may include calculating a ratio of the amount of sclera visible on the opposite side of the iris.

In some embodiments, the digital image can be analyzed to determine an angular offset of the face from the normal state, and based on the angular offset and based in part on the degree of coverage of the eyeball by the eyelid to determine the approximate direction of the eye's vertical gaze. .

Some embodiments include extracting one or more related features of the face, which are typically highly detectable. Such items may include the eyes and lips or nose, eyebrows, eyelids, eye features such as the pupil, iris and/or sclera, hair, forehead, chin, ears, and the like. For example, the combination of the two eyes and the center of the lips can form a triangle that can be detected to determine not only the orientation of the face (eg, head tilt) but also the rotation of the face relative to a facial shot. The orientation of the detectable feature can be used to determine an angular offset of the face from the normal state. Other highly detectable portions of the markable image, such as the nostrils, eyebrows, hairline, bridge of the nose, and neck, extend as a physical entity to the face.

The ambient light can be determined by means of a surrounding light sensor or a camera. In other embodiments, ambient light may be determined based on the relative size of a user's pupil to its eye size or other facial features.

Any number of user preferences can be dynamically adjusted or changed by means of such settings or parameters detected by the display (including age, eye, pupil and iris size, distance from the display, gaze, head tilt and/or ambient illumination) Set to suit specific users and settings.

In one embodiment, the displayable content and privacy settings can be automatically changed based on the age detected by one of the users. Referring to Figure 10, upon detecting that a child or minor is in front of the display 1000, a prompt or symbol 1008 can be displayed to indicate that a child or minor has been detected and that appropriate displayable content has been enabled for the display and Privacy settings. In one embodiment, if a child or minor is detected in front of the display, the default privacy and filtering options can be enabled (ie, It is programmed or selected by an adult or administrator to control the type of content displayed on display 1000. For example, web browser filtering can be enhanced to prevent younger users from encountering materials or content that are considered unsuitable for age by a long or administrator (eg, pornography, swearing language, violence, etc.).

The determination of which age group is equivalent to a "child", a "minor" and an "adult" or an "elderly" person may be pre-programmed or selected by an administrator. However, in some embodiments, a child may be a person under 15 years of age, a minor may be from one of 15 to 17 years of age, and an adult may be from one of 18 to 65 years old, and one Older people can be one of those older than 65 years of age.

Additionally, content currently contained on a computer attached to one of the displays may be considered to be non-displayable depending on the age or classification of the detected user. For example, if the user in front of the display is determined to be too young, the privacy financial file, photo, video or other sensitive file or material may automatically become inaccessible or undisplayable. As described above, an administrator can pre-program or select an age limit cutoff point for determining whether the material is unreachable or undisplayable.

In addition to changing the displayable content and/or privacy settings based on the age detected by the user, in some embodiments, the display can detect or identify a particular individual user and adjust based on the detected individual user. Display content, privacy settings, and/or personal settings. Referring to Figure 11A, a first user (e.g., user 1) is identified by the display, and as indicated by prompt 1108, individual user preferences, displayable content, and privacy settings for the user are automatically loaded on the display. Similarly, in FIG. 11B, a second user (eg, user 2) is identified by the display, and as indicated by prompt 1108, individual user preferences, displayable content, and privacy settings of the user are automatically loaded on On the display. Since these settings can be customized by the user or by someone else (eg, a long or administrator), it should be understood that the setting of User 1 can be different from the setting of User 2. For example, an administrator can change the user's content and privacy settings for all potential users of the system, and can enter a photo of each user or other identifiable features of each potential user. When the user is positioned in front of the display, the display can capture an image of the user and compare it to known users of the system, and automatically adjust the displayable content and privacy settings based on the detected user.

Figure 12 is an illustration of one of the users in the field of view of a display having a user timer. In this embodiment, the display can detect the presence of a user positioned in front of the display for a predetermined time limit and indicate to the user by a prompt or symbol 1208 that the user has exceeded a predetermined time limit in front of the display. . This can be used, for example, to limit the amount of time a user spends in front of the display, or to advise on frequent rest (eg, exercise, reduce eye strain, etc.). In some embodiments, the predetermined time limit may vary depending on the age of the user detected by the display. For example, a family may want to limit the amount of time a child spends in front of the display. In this example, if the display detects that the user is a child, an indicator or symbol may be displayed to advise the user to stop using the display after a predetermined time limit. In another embodiment, a user timer indicator or symbol 1210 may advise the user to take a short break, such as complying with local, state, or federal regulations requiring an employee to rest after a certain amount of time. In some embodiments, the display can be automatically turned off after a predetermined time limit is reached. In addition, the display can remain off for a stylized duration to prevent further use (eg, until the next day, or until a predetermined period of time has elapsed before the display can be used again).

In addition to determining whether a user has exceeded a predetermined time limit in front of the display, the display may also determine if the user is sitting unsuitably or tilting his or her head in a manner that may result in injury or discomfort. For example, referring to Figure 13, the display can detect or determine that the user 1306 is staring at the display in a bad posture or with a tilted head, which can potentially cause pain, cramps, or other discomfort. An unsuitable head tilt can be determined to be tilted at an angle from one of the heads of a normal or vertical head position shift. In this example, the display can display an ergonomic indicator or symbol 1312 to notify the user or to indicate to the user that the posture or head tilt that is unsuitable is corrected. This feature may be able to correct an unintended user's unsuitable posture or head tilt to prevent future pain, discomfort or injury.

The face detection, eye detection, distance and age determination described above can be further used in conjunction with light detection (eg, ambient light detection or illumination level of the user's face) to change or adjust additional user preferences. . In one embodiment, the display brightness can be varied based on the amount of ambient light detected by the display. In addition, it can be determined that an older user requires a brighter screen than a younger user, so that the brightness of the screen can be automatically adjusted depending on the age detected by the user. In another embodiment, ambient light is detected based on the brightness detected by one of the faces. In still another embodiment, the display can detect a pupil closure percentage and combine it with one of the illumination levels on the face and/or the ambient light level to determine the brightness level of the screen.

For example, if a light is shining on the face of a user, the user's pupil will be relatively closed and it will require a brighter screen. In this example, the screen can be automatically illuminated based on the illumination level of the user's face and/or the user's pupil size. On the other hand, if there is high ambient light in the user's background rather than the user's face, the user's pupil will be more open, but the screen can be sufficiently illuminated by the backlight and no adjustments can be made. In still another scenario, both the user's face and the user's background are dark or have low ambient light, and a bright screen is also required, and the brightness of the display can be automatically increased or adjusted to compensate.

In still another embodiment, user or screen privacy may be adjusted as an additional person enters the field of view of the sensor or when an unidentified user enters the field of view. In the first embodiment, as shown in FIG. 14, the screen 1402 of the display 1400 can be turned off when a second user 1414 enters the field of view with the user 1406. Similarly, referring to Figure 15, if a user 1506 is not recognized by the display, an indicator or symbol 1516 can be displayed on the display to indicate to the user Not recognized. In this embodiment, the display can be programmed to automatically turn off after displaying the indicator 1516, or another selection system can display a locked screen until the recognized user enters the field of view, or until the unknown user is assigned 1506. Access to the system.

In still another additional embodiment, the display can gaze with the user and only illuminate the screen segment 1518 corresponding to the user's gaze, as shown in FIG. The display can also illuminate the appropriate portion of the screen based on the user's reading speed based on a user moving across the screen's eyes as they read the multi-column text.

In still another embodiment, as shown in FIG. 17, the display 1700 can be positioned closer to the display when the user is sitting too close to the display (ie, when the display detects that the user is positioned closer to the display than an optimal viewing distance) When indicated by a pointer or icon 1722 to a user 1706.

In some embodiments, the display can automatically adjust the brightness of the display or fully turn the display on/off based on the detected user settings to conserve power.

The system may also include features for power saving based on features detected by the user as described above. The power saving process can involve multiple steps. For example, if the display does not recognize a face and/or two eyes in front of the display for a predetermined period of time, the display may initiate a power saving agreement. In one embodiment, a first level of power savings can be initiated. For example, a first level power saving can be such that when a user is not detected in front of the display for a predetermined period of time, the display can be dimmed by a set percentage. If the display continues to detect the user's face and/or eyes for an additional period of time, the display can be completely powered off. This process can have multiple intermediate power level steps that can be configured by one of the system administrators based on individual power saving goals.

In another embodiment, the entire sensor system can be turned off with the display processor and display when the display enters a power save mode. Sensor and processor can be configured Scans a face and/or eyes with occasional turn-on (eg, briefly after a predetermined period of time has elapsed) and returns to the off state without detecting a user and/or eyes . This can be extremely advantageous in a software-only implementation in which the soft system operates on a low power processor.

[Fig. 18 to Fig. 21]

The present invention provides techniques and methods for adjusting user preference settings based on parameters or conditions detected by a display system or monitoring device. In some embodiments, the display system can detect and/or determine the age of a user. In another embodiment, the display system can detect and/or determine a distance between the user and the display. In still another embodiment, the display system can detect and/or determine the amount of ambient light or light on a user's face, either alone or in combination with the detected age or distance conditions. In some embodiments, the display system can identify a user's face and can additionally identify a user's gaze or determine the user's pupil diameter.

Any number of user preferences or display settings can be dynamically adjusted based on parameters or conditions detected or determined by the display. For example, in one embodiment, the font size or graphic size may be adjusted based on the age detected by the user. In another embodiment, the font size or graphic size may be adjusted based on the detected distance of the user from the display. In some embodiments, a particular user is individually identified and the font or graphic size can be individually adapted to the particular individual identified by the display.

18 illustrates a display 1800, such as a computer monitor, a television display, a cellular telephone display, a flat panel display, or a laptop display having a screen 1802 and a plurality of sensors 1804. The sensors can include, for example, an imaging sensor (such as a camera including a CCD or CMOS sensor), a flash or other form of illumination, and/or configured to detect an object or Any other sensor that images the object, such as ultrasound, infrared (IR) Light or thermal sensor. The sensors can be placed on the display or integrated into the display, or another selection system that can be separated from the display. Any number of sensors can be included in the display. In some embodiments, a combination of sensors can be used. For example, in one embodiment, a camera, a flash, and an infrared sensor can all be included in a display. It should be appreciated that any sensor combination or any number of sensors can be included on or near the display. As shown in FIG. 18, user 1806 is shown positioned in front of display 1800 within the detection range or field of view of sensor 1804.

Various embodiments relate to a camera mounted on or near a display that is programmed to detect, track, and/or identify a facial or partial facial or a facial region (such as one or two eyes, or a mouth) A region or a processor coupled with a facial expression or expression (such as a smile or blink). In some embodiments, the processor is integrated into the display or disposed on the display. In other embodiments, the processor is separate from the display. The processor can include memory and software configured to receive signals from the sensors and process the signals. Some embodiments include sensing the characteristics of a user or a user with the sensors and determining parameters related to the face, such as orientation, pose, tilt, hue, color balance, white balance, relative exposure, or overall Exposure, face size, or size of the face area (including the size of the eye or the area of the eye such as the pupil, iris, sclera, or eyelid), a focus condition, and/or a distance between the camera or display and the face. In this regard, the following items are hereby incorporated by reference in their entirety to the extent of the disclosure of the disclosure of the disclosure of the disclosure of the disclosure of Case No. 13/035,907, No. 12/883,183, filed on September 16, 2010, and No. 12/944,701, filed on November 11, 2010, each of which is assigned to the same assignee, And U.S. Patents 7,853,043, 7,844,135, 7,715,597, 7,620,218, 7,587,068, 7,565,030, 7,564,994, 7,558,408, 7,555,148, 7,551,755, 7,460,695, 7,460,694, 7,403,643, 7,317,815, 7,315,631 and 7,269,292.

A number of techniques can be used to determine the age of a user sitting in front of a display or monitor. In one embodiment, the age of the user may be determined based on the size of the user's eyes, the size of the user's iris, and/or the pupil size of the user.

Depending on the sensor included in the display, one of the images or other data of the user, such as an image of the user, can be obtained by the display having the sensors. Information about the acquired data (including distance to the user or object, aperture, CCD or CMOS size, focal length and depth of field of the lens) can be recorded on the image or recorded along with the image. Based on this information, the display can determine the potential size range of one of the eye, iris, pupil or red-eye area (if a flash is used).

In this case, variability is not only for different individuals, but also based on age variability. Fortunately, in the case of the eye, the size of the eye is relatively constant from the time a baby grows to an adult, which is the cause of the "big eye" impact effect seen in babies and young children. The average eyeball measurement of the infant was approximately 19.5 mm from front to back and, as described above, grew to an average of 24 mm during the life of the person. Based on this data, in the case of eye detection, when a certain variability is allowed, the size of the object (which is the pupil, the part of the pupil of the pupil) is limited: 9 mm ≦ iris size ≦ 13 mm.

Thus, by detecting or determining the size of a user's eyes by means of the sensor 1804, the age of the user can be calculated. Further details regarding methods and processes for determining the age of a user based on eye, iris or pupil size can be found in U.S. Patent No. 7,630,006 to De.

In another embodiment, the person's face can be detected and classified according to the age of the subject (see, for example, U.S. Patent No. 5,781,650 to Lobo et al.). A number of image processing techniques can be combined with anthropometric data on facial features to determine an estimate of one of the age categories of a particular facial image. In a preferred embodiment, anthropometric data within a digital image is used to verify facial features and/or eye regions. The opposite approach can also be employed and it can involve a probabilistic reasoning, also known as Bayesian Statistics.

In addition to determining the age of the user, the display can also determine or detect the distance from the user to the display, gaze (or more specifically, the position and direction of the user's gaze), the posture of the user or the amount of head tilt, and Contains the ambient light and the amount of light on the user's face. Details on how to determine the distance of the user from the display, the user's gaze, head tilt or direction, and illumination level are also found in U.S. Patent No. 7,630,006 to DeLuca et al., and U.S. Patent Application Serial No. 13/035,907.

The distance can be easily determined by using an IR sensor or an ultrasonic sensor. In other embodiments, one of the images of the user can be captured by a camera and can be determined by comparing the relative size of the detected face with the detected features of the face, such as the eyes, nose, lips, and the like. The distance of the user. In another embodiment, the relative spacing of features on the face can be compared to the detected size of the face to determine the distance of the user from the sensor. In still another embodiment, the focal length of the camera can be used to determine the distance of the user from the display, or another selection system that can be related to the detected features of the user (such as the size of the face or the relative size of the facial features). ) Combine to determine the distance of the user from the display.

In some embodiments, determining the user's gaze can include acquiring and detecting a digital image comprising at least a portion of one face (including one or two eyes). At least one of the eyes can be analyzed and a degree of coverage of one of the eyeballs can be determined. Based on the eyelids' judgment on the eyeball By determining the degree of coverage, one of the vertical gaze of the eye can be determined to be approximately the direction. Analysis of at least one of the eyes can further comprise determining one of the horizontal gaze directions. In some embodiments, the technique includes initiating a further action or initiating a different action (or both) based at least in part on the determined direction of the horizontal gaze. The analysis of the one or two eyes can include spectrally reflecting one of the light from the one or both eyes. This can include analyzing the amount of one sclera visible on at least one side of the iris. In other embodiments, this may include calculating a ratio of the amount of sclera visible on the opposite side of the iris.

In some embodiments, the digital image can be analyzed to determine an angular offset of the face from the normal state, and based on the angular offset and based in part on the degree of coverage of the eyeball by the eyelid to determine the approximate direction of the eye's vertical gaze. .

Some embodiments include extracting one or more related features of the face, which are typically highly detectable. Such items may include the eyes and lips or nose, eyebrows, eyelids, eye features such as the pupil, iris and/or sclera, hair, forehead, chin, ears, and the like. For example, the combination of the two eyes and the center of the lips can form a triangle that can be detected to determine not only the orientation of the face (eg, head tilt) but also the rotation of the face relative to a facial shot. The orientation of the detectable feature can be used to determine an angular offset of the face from the normal state. Other highly detectable portions of the markable image, such as the nostrils, eyebrows, hairline, bridge of the nose, and neck, extend as a physical entity to the face.

The ambient light can be determined by means of a surrounding light sensor or a camera. In other embodiments, ambient light may be determined based on the relative size of a user's pupil to its eye size or other facial features.

Any number of user preferences can be dynamically adjusted or changed by means of such settings or parameters detected by the display (including age, eye, pupil and iris size, distance from the display, gaze, head tilt and/or ambient illumination) Set to suit specific users and settings. For which age groups The determination of a group equivalent to a "child", a "minor" and an "adult" or an "elderly" person may be pre-programmed or selected by an administrator. However, in some embodiments, a child may be a person under 15 years of age, a minor may be from one of 15 to 17 years of age, and an adult may be from one of 18 to 65 years old, and one Older people can be one of those older than 65 years of age.

In one embodiment, the font size displayed on display 1800 can be dynamically changed based on the age detected by one of the users. Referring now to Figures 19A-19B, in Figure 19A, user 1906 is detected as an older user, and thus the size of font 1908 can be automatically increased based on the age determination of the user. Similarly, in Figure 19B, the user is detected as a younger user, and thus the size of the font 1908 can be automatically reduced based on the age determination of the user.

Similarly, in addition to dynamically changing the font size based on the age detected by the user, the display can also automatically change the size of the system icon based on the age determination of the user. Referring to FIGS. 20A-20B, in FIG. 20A, the user 2006 is detected as an older user, and thus, the size of the system icon 2010 can be automatically increased based on the age determination of the user. Similarly, in FIG. 20B, the user is detected as a younger user, and thus the size of the system icon 2010 can be automatically reduced based on the age determination of the user.

In addition to changing the font or size based on the age detected by one of the users, the display can also automatically change the font and/or size based on a detected distance between the user and the display. Referring now to Figures 21A-21B, in Figure 21A, a distance 2112 between the user 2106 and the display 2100 is increased, and the size of the font 2108 and/or the representation 2110 can be increased on the display to aid visualization. Similarly, in FIG. 21B, the distance 2112 between the user 2106 and the display 2100 is reduced, and the size of the font 2108 and/or the illustration 2110 can be reduced on the display. In one embodiment, a user can be pre-programmed at an optimal distance from one of the displays (eg, from a 24" screen > 80 cm) and the display can be configured to move away from or toward the display for the user. One cm or In English, the font size is automatically increased or decreased by a predetermined percentage. In some embodiments, the display can determine both the age of the user and the distance of the user from the display to determine the size of the font and/or icon. In some embodiments, the display can detect whether a person has difficulty viewing the display, such as due to the user's detected age, its proximity to the display, its distance from the display, and the detected skew. Wait. Once a viewing issue is detected, the system automatically zooms in on the font and/or size of the image as a response.

The amount of change in the size of the font and/or icon can be adjusted by the user or by an administrator. For example, when sitting at a distance, individual users may prefer fonts that are larger than normal, or prefer smaller fonts when sitting close. Graphical changes based on distance and/or age can be fully customized by the user or system administrator.

The embodiments disclosed herein are applicable to a television, desktop monitor, laptop monitor, tablet device, other mobile devices such as smart phones, and other electronic devices having displays.

Example embodiments will now be described more fully hereinafter with reference to the accompanying drawings in which: FIG. Rather, the embodiments are provided so that this disclosure will be thorough and complete, and the subject matter will be fully conveyed by those skilled in the art.

22 shows an exemplary ergonomic sensor module 2202 for use with a display 2204 that is used with a computer system 2205. The computer system can include a desktop, laptop, server, gaming system, mobile device, or other computing device to drive display 2204. Of course, display 2204 can include any suitable display type including, but not limited to, an LCD, plasma, CRT or other display or even a television. The sensor module 2202 is positioned such that it can produce image data for a user 2206 of the display device 2204. This instance The sensor module 2202 is positioned on or in one of the structural elements 2204A (such as a display panel or housing) of the display 2204. The sensor module 2202 can be positioned at any suitable location relative to the display 2204 and can even be positioned away from the display. User 2206 can sit, stand, or otherwise approach display 2204.

As shown in the illustration, the sensor module 2202 includes one or more image sensing devices (sensors 2208), a processing component 2210, and an input/output interface 2212. For example, sensor 2208 can include a CMOS or other image sensing technology that can be used to provide still and/or video image data. Processing component 2210 can include a microprocessor, digital signal processor (DSP), special application integrated circuit (ASIC), or can be configured to sample data from sensor 2208 and provide output via I/O interface 2212. Other hardware logic.

The processing component 2210 is configured to obtain image data from the image sensing device, and in this example, analyze the image data to determine whether the image data is based on accessing predefined data defining a range of ergonomic use of the display device A display is indicated to be used by a user of the display within the scope of ergonomic use. In this example, processing component 2210 is further interfaced to memory 2214, which represents any suitable non-transitory computer readable medium, and includes configuration processing component 2210 to obtain and analyze data. The code of the formula 2216. For example, memory 2214 can include RAM, ROM, cache or other memory or a storage device (eg, a magnetic disk, optical disk, flash memory, etc.). However, as mentioned above, embodiments may use a hardware-based approach (eg, an ASIC, a programmable logic array, or other hardware logic that causes processing component 2210 to perform analysis and produce an output).

In some embodiments, the I/O interface 2212 is coupled to the display device 2204 and the processing component 2210 is further configured to respond to the determination that the image data indicates that one of the displays is not in use by the user within the scope of ergonomic use A feedback message 2218 is output using the display device. For example, ergonomic analysis routine 2216 can direct processing component 2210 to display warning message 2218 using the I/O interface without the intervention or processing of computer 2205.

The computer 2205 includes a processor 2218, a memory 2220, and other conventional computer components (eg, a bus, a network interface, a display interface, a storage medium, etc.). In some embodiments, in addition to the ergonomic analysis routine 2216 or the alternative ergonomic analysis routine 2216, the ergonomic analysis routine 2217 is also performed by the computer 2205. For example, an ergonomic sensor module including sensor 2208, processing element 2210, and I/O interface 2212 can provide image data only to ergonomic analysis routine 2217. In some embodiments, a webcam or other imaging device functions as an ergonomic sensor module 2202.

23 is a diagram showing one example of an ergonomic sensor module 2302 integrated into one of the structural elements of a display. In this example, the components of the sensor module 2302 are enclosed within a panel of the display 2304 with an aperture in front of the display to image a user. For example, the ergonomic sensor module 2302 can be configured as a built-in webcam that provides image data to ergonomics loaded on a computer 2305 (not shown) that is interfaced to the display 2304. Analyze the normal form 2317. However, the sensor module 2302 can include sufficient processing power to host the ergonomic analysis routine 2316 and provide output directly using the display 2304.

24 is a diagram showing one example of an ergonomic sensor module 2402 used externally to a display 2404 that is interfaced to a computer 2405. In this example, the ergonomic sensor module 2402 includes a webcam. For example, the webcam can provide image data via a USB or other interface for use by a processor of the computer 2405 to analyze and determine appropriate feedback. The webcam can be attached to or positioned on display device 2404, such as on top of the display device, on the front of one of the display devices, and the like. Of course, the webcam can be positioned on the side of the display device or can be located elsewhere. The computer 2405 can be provided with software or firmware (in a non-transitory computer readable medium) Implemented in the body to cause the computer 2405 to perform some or all of the ergonomic analysis based on image data from the webcam.

The size and location of the webcam and integration shown above are for illustrative purposes only. The imaging device can be positioned at any suitable point to provide an image of one of the users of display 2404. In some embodiments, the imaging device is positioned to capture light representing one of the images of the user as seen from display 2404 (eg, using one of the sensors or optics that taps toward the front of one of the displays) .

Figure 25 is a flow chart showing one of the steps of an exemplary method 2500 when an ergonomic sensor module is used. For example, based on the image data, the method 2500 can be performed by the ergonomic analysis routine 2516 implemented by the module 2502, and/or can be performed by the ergonomic analysis routine 2517 implemented by the computer 2505. The sensor module can perform method 2500 during the execution of the software or firmware. However, method 2500 can also be performed in a hardware-based implementation, such as by hardware logic, such as a special application integrated circuit (ASIC), programmable logic array (PLA), logic gate configuration, or Input values (eg, pixel values) are obtained and processed to determine an output (eg, whether the pixel values indicate an ergonomic use) other hardware. In practice, method 2500 or another image analysis method can be performed periodically or continuously to provide instant feedback to one or more users.

Block 2502 represents obtaining image data from an image sensing device (eg, one or more image sensors). For example, the block may include accessing image data from the image sensing device, and determining that the image data indicates a user of a display device. If the user does not exist, the rest of the routine need not be performed. The presence of the user can be determined by analyzing the field of view, such as using a motion detection algorithm, comparing a background image to image data, detecting a face, or in some other manner. In some embodiments, multiple users can be identified by, for example, using face detection.

In summary, blocks 2504 through 2508 represent analyzing image data to determine whether the image data indicates that the user of the display is using the display within an ergonomic range based on predefined data defining an ergonomic range of use of the display device. . If multiple users have been identified, the routine can determine if each user is making an ergonomic use of the display. However, in some embodiments, analyzing the image includes selecting one of the users (eg, the primary user) to determine if the user is performing ergonomic use. For example, a user can be selected by determining the maximum face size seen by the imaging system at a given time.

Block 2504 represents accessing data defining one or more ergonomic use ranges of the display device. The range of ergonomic use can be defined as the range of various parameters of the ergonomic metric used to characterize the user's posture and surrounding conditions. At block 2506, one or more image analysis algorithms are applied to the image data to determine parameter values for the corresponding ergonomic metrics, and at block 2508, the parameter values are compared to the ergonomic range of application. A determination is made as to whether the user is in one of the one or more ergonomic measurements of ergonomic use.

In certain embodiments, the data is analyzed to determine a parameter value of one or more of the following ergonomic metrics, wherein the parameter value is compared to the ergonomic range of use listed below. However, such metrics and ranges are provided for exemplary purposes only. Embodiments may use additional ergonomic measurements and/or ergonomic use ranges to suit particular needs.

The image data can be analyzed in any suitable manner to produce parameter values for ergonomic measurements. For example, in some embodiments, the analysis includes using a facial recognition algorithm to determine where the user's face is in the image. The use of a facial recognition algorithm may allow the sensor module to analyze the use of the display independently of the user's facial shape (eg, without considering the user's face being elliptical, square, or some other shape). The algorithm looks for skin tone and detection of facial features such as eye, lip/mouth position to determine the presence of a person and is therefore independent of the actual shape of the face itself. Based on the location of the user's face, the facial portion of the image can be subjected to additional analysis algorithms to determine parameter values for various ergonomic measurements.

In addition, by using image analysis, the problem of ergonomic use can be solved independently of the user's pre-fetching of the posture data or by requiring the user to match a predefined posture or position. Alternatively, the image data itself is used to determine whether the user characteristics detectable in the image and/or the surrounding conditions detectable in the image are consistent (or inconsistent) with the ergonomic use. The algorithm measures one of the distances between the pupils (the distance between the centers of the two eyes) to detect the distance of a face from the display and uses the same metric to determine if the face has a yaw/tilt/rotation angle.

For example, the distance from the monitor can be determined by identifying one of the features in the image (eg, the user's eyes). Based on the information indicating the location of the sensor module, the user can be used The distance or angle of the user is estimated by the parallel parallax or triangulation of the eye or even the entire face of the user.

In one embodiment, a feature recognition algorithm locates a user's eyes based on analyzing the image to identify shadows beneath the user's eyes. In particular, the pixel intensity value of the image can be evaluated to identify a darker region that can correspond to a shadow; if the darker region is similarly shaped and spaced apart by an acceptable distance, the feature recognition algorithm can infer the user's eye It is above the shadow.

Image analysis that identifies a user's eyes can be used to determine if the user has stared for too long without blinking. For example, a blink recognition algorithm can analyze a series of images to determine how long the user's eyes have remained open (ie, appearing in the series of images). If the user's eyes have not blinked after a threshold period has elapsed, a warning or other feedback may be provided.

In some embodiments, the user's eyes, face, and/or other distinctive features can be used to determine if the same user has remained close to the display (eg, in front of it) without a break. For example, a threshold period of time can be defined for the ergonomic use of the display. By analyzing the length of time that the user continues to appear, the sensor module can determine if the user has exceeded the limit and should take a break. The algorithm can also find a minimum rest duration to ensure that the user leaves the display for a minimum period of time.

In some embodiments, the image data is analyzed to determine information about the spatial position of the user's face relative to the display (eg, relative to one of the planes of the display). For example, one or more of a user's face rotation angle, yaw angle, or pitch angle with respect to the display is used to determine whether the user's face is in a ergonomic condition based on the determined one or more angles Learning to use. The rotational, pitch and yaw angles may be defined as the angle of rotation indicative of the plane of the user's face relative to the plane of the display.

Figure 26 illustrates the yaw angle of the user relative to the display. The yaw angle measures how far the user is offset from the point on the display to the right or left. As shown in Figure 26, the yaw angle is the angle between a line extending from the user to one of the points on the display and a vertical line extending from the point on the display when viewed from the top or bottom of the display. For example, the line between the user and the display can include a point on the user and a point in the display that is close to a point between the eyes of the user.

Figure 27 illustrates the pitch angle of a user relative to the display. The pitch angle measures how far the user deviates upward or downward from a point on the display. As shown in Figure 27, the pitch angle is the angle between a line extending from the user to a point on the display and a vertical line extending from the point on the display as viewed from the side of the display. For example, the line between the user and the display can include a point on the user that is close to a point near the user's eye and a point on the top of the display.

Glare and ambient light can be identified using a search image to find an algorithm that corresponds to an intensity pattern that is too bright or too dim glare and/or ambient light. For example, the average intensity of the image can be found and scaled to determine one of the ambient light conditions. The glare from the monitor can be identified by searching for an image area in which the intensity is rapidly rising - for example, the face area of the user such as the user's cheek/forehead can be analyzed to determine whether the user's face is reflecting a large amount of light . By continuously analyzing the intensity values across the entire image, one of the ergonomic analysis routines can determine ergonomic use independently of changes in ambient light conditions. The measured intensity across the image is limited to determine the low light condition. The algorithm selects an image area that is close to the user's face and above to remove the effect of the user's darkening (lowering the average intensity value).

The glare towards the monitor can be identified by analyzing the image for high back illumination - assuming the image sensor is facing the user, if the user is behind the illumination (ie, the face area has a comparison There may be glare towards the monitor with a low pixel intensity around the area of the user's face. This intensity difference can be used to determine a parameter value to be compared to the ergonomic range of glare.

As mentioned above, at block 2708, the ergonomic analysis routine determines whether the user is in one or more ergonomic use ranges, such as by comparing the parameter values calculated from the image with the defined range of use. Taking data, the ergonomic analysis routine can determine whether a user is within, outside or near one of the limits of the ergonomic use of the display.

This ergonomic analysis routine operates with a display having multiple orientations. For example, some displays allow a user to rotate a display by about 90° such that the display is wider than its height in one orientation, commonly referred to as a landscape orientation, and in a second orientation the display is The width is higher, commonly referred to as portrait orientation. This ergonomic analysis routine determines the orientation of the display and, if necessary, adjusts based on the orientation. In one embodiment, the ergonomic analysis routine monitors a control signal and determines the orientation of the display from the state or level of the control signal.

Block 2710 represents the output of a feedback message. The format, content, and triggering criteria of a feedback message can vary, and in some embodiments, the message is provided on-the-fly with image analysis. As an example, if the analysis shows that the user is outside an ergonomic range of use, a feedback message can be provided, wherein the message indicates which one or more metrics have been "violated" (eg, distance, angle, lack of blinks) , ambient light, etc.). This allows the user to perform corrective actions.

Feedback can also be provided for indication when a user is near an edge of the ergonomic range of use. For example, if the user is almost too close or too far away from the display (eg, 3 to 4 cm from the limit), a warning may be provided to allow for corrective action. Furthermore, feedback can also be provided when the user is inside an ergonomic range of use to, for example, enhance good use.

The format of the feedback message can be varied as mentioned above. In one embodiment, a visual message is provided by transmitting the data to display 2704. For example, a quick view or overlay can be generated from text or graphics. Other examples include sound or other feedback.

Depending on the particular implementation, the information of the feedback message may be provided by the sensor module 2702 itself or by the computer 2705. For example, in one embodiment, the module 2702 is integrated into the display and can provide the information directly to the display while partially or completely obscuring other information provided by the computer 2705 (eg, from the computer 2705) The displayed material (if present) is rendered on one of the overlays provided in the message). However, in some embodiments, the ergonomic analysis routine 2716 performed by the module 2702 provides information indicative of one of the output messages to be generated, and the computer 2705 utilizes a complementary ergonomic analysis of one of the masters of the computer 2705. Equation 2717 is to present a window or otherwise provide the message. Further, the module 2702 can provide only image data, wherein the image data is analyzed by an analysis routine 2717, which is controlled by the computer 2705, and the analysis routine also presents the window or otherwise provides the message.

A number of examples using one of the above ergonomic sensor modules 2702 utilize one sensor. It will be appreciated that multiple sensors can be used within one module 2702 and multiple modules 2702 can be used simultaneously for a single display or for multiple displays.

Any suitable non-transitory computer readable medium may be used to implement or practice the presently disclosed subject matter, including but not limited to magnetic disks, disk drives, magnetic storage media, optical storage media (eg, CD-ROM, DVD-ROM and its changes), flash, RAM, ROM, register memory, cache memory and other memory devices. For example, an implementation includes, but is not limited to, implementing instructions that cause a processor to perform the methods as recited herein and/or operations performed during operation including, but not limited to, embodiments of the examples discussed herein. Non-transitory computer readable media.

The subject matter can be implemented by any computing device that performs a series of operations based on a command. The one or more hardware circuits or components comprise a general purpose and special purpose processor that accesses instructions stored in a computer readable medium causing the processor to perform operations as discussed herein, and configured to perform as herein The hardware logic of the operations discussed (eg, Field Programmable Gate Array (FPGA), Programmable Logic Array (PLA), Special Application Integrated Circuit (ASIC)).

Although an exemplary embodiment of the invention has been illustrated and described, it is understood that the scope of the invention is not limited to the specific embodiments discussed. Therefore, the present embodiments are to be considered as illustrative and not restrictive, and it should be understood that those skilled in the art In the case of these embodiments, changes are made to them.

In addition, in the methods that can be performed in accordance with the preferred and alternative embodiments and the scope of the claims herein, the operations have been described in a selected printing sequence. However, the sequences are selected for the purpose of printing convenience and are therefore ordered, and are not intended to imply any particular order for performing such operations unless a particular ordering is explicitly indicated as being required or skilled in the art. It is considered necessary.

Claims (17)

A handheld camera-capable video conferencing device includes: a housing configured to be held in the hands of a user; a processor in the housing; a memory in the housing Embedded in the code for programming the processor, comprising a video conferencing component, a face detection component, a face recognition component, and an associated image processing component, and wherein the memory further includes being associated with one or more specific user identities Facial data; a display built into the housing and configured to be viewable by a user during a video conference; and a camera built into the housing and configured to capture the usage The image of the user when viewing the display, the camera includes an infrared (IR) light source and an IR sensitive image sensor for capturing images of the user in low or uneven light conditions or both. So that the face detecting component can detect the face of the user, and view the face as the foreground of the image of the user, and the rest is the background; and wherein the face recognition component is configured to be a user A specific identity is associated with a detected face that is considered to be foreground: and wherein the image processing component replaces the face data of the detected face with the face data stored in the memory according to the specific identity of the user To enhance one of the detected faces captured in low light or uneven light conditions or both, and transmit the foreground to a far-end video conferencing participant. The device of claim 1, wherein the facial data comprises chromaticity data. The device of claim 1, wherein the facial data comprises illuminance data. The device of claim 1, wherein the face detection component or the face recognition component or both comprise a classifier trained to detect a face or recognize a face or both in low light or uneven light conditions or both. A device as claimed in claim 1, wherein the IR light source comprises one or more IR LEDs coupled to the housing and arranged to illuminate a face of the user during a video conference. The device of claim 1, wherein the memory further comprises a face tracking component to track the detected face to enable the device to transmit an approximate continuous video image of the user's face during the video conference. The device of claim 1, wherein the memory further comprises a component to estimate a distance to the face of the user and to control an output power of the IR light source based on the estimated distance. The apparatus of claim 7, wherein the estimate of the distance is determined using an autofocus material. The device of claim 8, wherein the estimate of the distance is determined based on a size detected by one of the faces of the user. The device of claim 8, wherein the memory further comprises a component to determine a position of a user's face relative to the device and to control a direction of the IR light source that illuminates the face of the user. A handheld camera-capable video conferencing device comprising: a housing configured to be held in the hands of a user; a processor in the housing; a memory in the housing in which is embedded a code for programming the processor, including a video conferencing component and a foreground/background segmentation component or a combination thereof, a display built into the casing and configured Illustrated by a user during a video conference; a camera built into the housing and configured to capture images of the user while viewing the display; and a communication interface for audio Transmitting/visual data to a far-end video conferencing participant; and wherein the foreground/background segmentation component is configured to extract user identity data that does not include background material by identifying different motion vectors of the foreground versus background material, the communication The interface delivers visual material that does not contain background material to the far-end video conferencing participant; wherein the foreground/background segmentation component is calibrated to match the particular user identity data as foreground material. The device of claim 11, wherein the user identity data comprises facial information. A handheld camera-capable video conferencing device includes: a housing configured to be held in the hands of a user; a processor in the housing; a memory in the housing Embedded therein is a code for programming the processor, comprising a video conferencing component and a foreground/background segmentation component or a combination thereof; a display built into the casing and configured to be accessible by a user during a video conference Viewing; a camera built into the housing and configured to capture images of the user while viewing the display, the camera including an infrared (IR) source and an IR sensitive image sensor In low light or uneven The image of the user is captured in a uniform state or both to enable a face detection component to detect a user's face; and a communication interface for transmitting audio/visual data to a remote video conference Participant; and wherein the foreground/background segmentation component is configured to extract user identity data without background material by matching the detected facial data as foreground data, the communication interface transmitting visual material not including background material to The far-end video conferencing participant. The device of claim 13, wherein the memory further comprises a face tracking component to track the detected face to enable the device to transmit an approximate continuous video image of the user's face during the video conference. The device of claim 13, wherein the specific user identity data comprises an image of a detected face. The device of claim 15 wherein the particular user identity further comprises a neck, a portion of a torso or a shirt or one or both arms or a combination thereof. The device of claim 13, wherein the memory further comprises facial material associated with one or more particular user identities, and the device is configured to extract the particular use based on matching the facial metric in the memory Identity information.