US9270933B1

US9270933B1 - System and method for face-to-face video communication

Info

Publication number: US9270933B1
Application number: US14/494,558
Authority: US
Inventors: Yuan Qing Jiang
Original assignee: Individual
Current assignee: Individual
Priority date: 2013-11-23
Filing date: 2014-09-23
Publication date: 2016-02-23
Anticipated expiration: 2034-09-23

Abstract

The present invention relates to a system for video communication comprising a display having a reflective surface facing to a local user and a video camera mounted on a supporting structure and aimed toward the display; wherein said display displays dark screen images periodically; said camera captures of the local user's image reflected off from the surface during the dark screen image period. The captured reflective images of the local user are used to form a video stream for a remote user to view.

Description

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a non provisional patent application of U.S. provisional patent application Ser. No. 61/908,071, filed on Nov. 23, 2013, by Jiang, Yuan Qing, entitled “SYSTEM AND METHOD FOR FACE-TO-FACE VIDEO CONFERENCING”. The entire disclosure of each of which is herein incorporated by reference for all purposes.

FIELD OF INVENTION

The present invention relates generally to a system and method for face-to-face video communication, and more particularly to a system and method to provide eye contact between a local user and an image of a corresponding remote user participating the communication.

BACKGROUND OF INVENTION

Video connections through the internet and cellphone networks enable face-to-face and real time communication between people located at different parts of the world. As network communications have become faster and more reliable, both person to person and business multimedia communications are increasingly popular with integrated video cameras on smart phones, touch pads, laptop or desktop computers, and television sets. However, such kind of video communication is still no comparison to direct face-to-face meeting in terms of communication quality and effect. One of the problems is lacking of eye contacts during video conversation, because each of the communicators commonly looks at the video screen and not at the video camera, preventing the communicator from having direct eye-to-eye contact to the other remote viewers, as the video cameras are usually positioned away from the screen.

Eye contact is an important part of human interaction, for it conveys non verbal information in a subtle way. In a natural human interaction, we use our facial expression and eye contact to assist our expression. On the other hand, lacking of eye contact during a conversation makes virtual interaction unnatural. People usually dislike this behavior, since most people regard lacking eye contact during a conversation as lack of connection. It is unacceptable if a person looks away while talking, which will be perceived as unconfident, insincere and socially awkward, and his personality can be in jeopardy as the listeners may judge him as inattentive, untrustworthy, pretentious, even rude and snobby.

From speaker side of the view, if a listener is not making eye contact with the speaker while he or she is talking, the speaker may perceive that his or her topic is not interesting to the audience, or the speaker is not convincing enough to get the point crossed. Some speakers may cut short of the conversation for lacking of eye communication.

Another negative effect of having a video camera located on the margin of a display screen is that the image of people in the video is distorted and abnormal. The top portion of a human head looks larger and his or her chin looks smaller, when the camera is mounted on the upper part of the screen, like most laptop computer does currently. This is another problem causing unnatural video communication.

In order to solve the above problems and make the video communication looks more like a real face to face meeting, which means that the participants of the video communication can make eye contact to each other during the meeting, it is necessary to let the participants look at the web cam and the display simultaneously. A participant has to focus his or her eyes on the web cam in order to make eye contact with people who are viewing the image, meanwhile the participant has to look at the display so that he or she can see images of people participating the conference at a remote site. This technical difficulty of the device hampering video communication has been studied and a number of methods to mitigate the problem have been proposed.

US patent application publication 20130207952 disclosed a design that the web cam is mounted at the center of the display instead of on the top frame of the display as conventional design. By way of the above configuration, when a local user (the conference participant) looks at the center of display screen his or her eyes are lined up with the camera, and thereby the image of the local user shows eye contact.

The above disclosed design brought the web cam inline with the display. However, part of the display is occupied by the camera and its accessory parts.

U.S. Pat. No. 8,570,423 disclosed a video conference system with a projection display. Both projector and video camera are mounted behind a transparent projection screen so that the local user can view the display image and at the same time looks at the camera through the transparent screen. Therefore the camera can capture an eye contact image of the local user. However, it is difficult to use such system for mobile video conferencing.

U.S. Pat. No. 8,199,185 disclosed a video conferencing system with a semireflective transparent panel having an angle of about 30 to 70 degree with a display for reflecting an image of a local user. A video camera disposed on a side of the semireflective transparent panel opposite the image display.

U.S. Pat. No. 8,253,776 disclosed an image synthesis method to synthesis eye contact image from non-eye-contact images of a local user. However, it is difficult to synthesis a nature dynamic eye image in a real time with an ordinary image processor in a mobile device.

In view of the above problems, there is a need to develop an improved video communicating device with a convenient yet reliable design to enable eye contact between a local user and the corresponding image of a remote user participating the communication.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the invention and many of the attendant advantages thereof will be readily obtained and understood by referring to the following detailed description and the accompanying drawings in which like reference numerals denote like elements as between the various drawings. The drawings, briefly described below, are not to scale.

FIGS. 1A, 1B and 1C show schematic representations of a face-to-face video conference system in accordance with one or more embodiments of the present invention.

FIG. 2 shows an example of a face-to-face video conference system configuration with a desk top computer in accordance with one or more embodiments of the present invention.

FIGS. 3A, 3B, and 3C illustrate various types of configurations of reflective surfaces with display structure in accordance with one or more embodiments of the present invention.

FIG. 4A illustrates one mode of making eye contact during video communication.

FIG. 4B illustrates another mode of making eye contact during video communication.

FIG. 5A illustrates timing diagrams during a video communication in accordance to embodiments of the present invention.

FIG. 5B illustrates a normal display screen image including a sub-window of the remote user's image during a video communication in accordance to embodiments of the present invention.

FIG. 5C illustrates a local user's image reflected off from the screen during a dark screen image period in accordance to embodiments of the present invention.

FIG. 6 illustrates timing diagrams during a video communication in accordance to embodiments of the present invention.

FIG. 7 illustrates a method of synchronization in accordance with one or more embodiments of the present invention.

FIG. 8 illustrates face-to-face video conference in accordance with one or more embodiments of the present invention.

FIG. 9 illustrates timing diagrams during a video conference in accordance to one embodiment of the present invention.

FIG. 10 illustrates a method of synchronization in accordance with one or more embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Before proceeding with the detailed description, it is to be appreciated that the present teaching is by way of example, not by limitation. Thus, although the face-to-face video communication system and various implementations described herein are for the convenience of explanation shown and described with respect to exemplary embodiments of the system configuration and their implementations, it will be appreciated that the principles herein may be applied equally in other types of display-image capturing systems. Alternate embodiments may be devised without departing from the scope of the invention. Additionally, well-known elements of the invention will not be described in detail or will be omitted so as not to obscure the relevant details of the invention.

FIGS. 1A, 1B and 1C illustrate a configuration of a face-to-face video communication system in accordance with one or more embodiments of the present invention. The system 100 comprises a laptop computer (laptop PC) 110. The laptop PC has a flat panel display 115, such as liquid crystal display (LCD), light emitting diode display (LED), or electrophoretic display. The display 115 is positioned facing a local user 130 and the screen of the display is parallel to the face of the user 130 or may have an angle smaller than 45 degree respect to the parallel direction. The display 115 has a reflective surface 120 facing to a local user's face 130. The reflective surface having a reflectance larger than 3%, such as a laptop monitor with a glossy surface, is sufficient to reflect images of the local user. The system 100 also has a video camera 140 mounted on a supporting structure. As shown in FIGS. 1A, 1B and 1C, the camera is mounted on the outer margin of the key panel 145 and aimed toward the display 115.

The supporting structure of the camera 140 enables the camera aiming to the reflective surface 120 to capture images of the local user's face reflected off from the surface. The supporting structure of the camera can be adjusted both manually and automatically for the camera aiming to the display properly, so that the camera can receive the specular reflection of the image of the local user when the display turns into a dark background image or a dark screen. The supporting structure can also be designed to be detachably attached to any suitable structures such as on various portions of the laptop PC, or mounted on a desk, or on certain wearable devices on the body of the user such as on a head set gear or an eye glasses frame, and enabling the camera aiming toward the display 115. The supporting structure (175 in FIG. 1C) can also be placed outside of the keyboard structure. In addition, the position and orientation of the camera may be adjusted manually or automatically to allow the camera to aim to a desired location 170 on the reflective surface 120. The camera and the supporting structure can be hidden or retracted when the laptop is folded. In addition, an illumination element 180 such as an LED light may be mounted behind the camera to add light intensity on the local users face as shown in FIG. 1C. And auxiliary lights may be used to add illumination.

A similar system configuration of FIGS. 1A, 1B, and 1C can be applied to other displays of various systems, such as a desk-top computer, mobile devices (smart phone or touch pad), gaming machine, and TV monitors in according to one or more embodiments of the present invention. The camera can be detachably attached on a supporting structure similar to the one described for laptop PC, and is aimed to the display of the device to capture the reflective image of the local user during a period when the display shows a dark image or the screen is dark.

FIG. 2 shows an example of a face-to-face video conference system configuration 200 with a desk top computer (the computer is not shown). The system 200 has a keyboard 245 and a desktop PC monitor 215. The monitor can be a flat panel display or a projective display. If a projective display is used, element 215 represents a projection screen. The display 215 is positioned facing a local user and the screen of the display is parallel to the face of the user or may have an angle smaller than 45 degree from the parallel direction. The display 215 has a reflective surface 220 facing to the local user's face 230. The system 200 has a video camera 240 mounted on a supporting structure 241. The camera 240 is aiming toward the reflective surface 220 of the display 215 to capture the image of the local user face 230.

The

reflective surface

120 or 220 shown FIG. 1A, 1B, and FIG. 2 can be any kind of surface which provides specular reflection. For example, a surface with a reflectance larger than 1%. Yet, the reflected surface allows for visible light transmitted from the display source through the reflective surface for the local user to view the display image. One example of such reflective surface is a smooth glossy surface in front of a dark background which can be a very good reflector. The display surface of the current commercial products from many display manufacturers can be readily used to provide such reflective surfaces without further modifications. FIGS. 3A through 3C illustrate some examples of constructions of displays with the reflective surface. FIG. 3A shows a glass or a plastic plate has a smooth glossy surface (310) covering a display panel 315. For an ordinary LCD display, there is a protection glass panel (or referred as covering panel). If the surface of the covering panel is smooth and glossy and when the display lights are off or the display image is dark (dark period of the device), the display screen (covering panel) is reflective and can reflect the image of the user in front of it. Yet, the reflective surface allows more than 50% transmission rate of visible light from display panel 315 through the surface 310.

In according to one embodiment of the present invention, when a display protection panel 320 is not reflective, a reflective coating 325 can be applied to the surface of the protection panel as shown in FIG. 3B. Alternatively, a reflective sheet can be laminated on the display surface.

If a projection screen 335 instead of a flat panel display is used for video conferencing, a reflective panel 340 such as a glass panel or a transparent panel coated with reflective materials can be mounted in front of and parallel to the projection screen as shown in FIG. 3C. This configuration may also be applied to a flat panel display, where a separate reflector is mounted in front of and parallel to the flat panel display. This reflector should also be transparent for the local user to view the display images during the display period.

One of the important elements in the present invention to enable face-to-face video communication with eye contact to a remote user is to capture the local user's image with the camera aligned with the local user's eye through the reflection. This task is fulfilled by enabling the local user looks at the screen of the display and sees the remote user's image, and the local user's image is captured by the camera via the reflection during the dark screen period of the display. Referring back to FIG. 1B, if the local user is staring at point 150, the camera is aligned with the eyes via reflection path 160. The spot 150 is defined as “reflection point” where the reflection path 160 intersects with the reflective surface 120. Therefore the local user looks into the image of the eyes of the remote user on the display. In this way, the local user looks at both camera (through reflection) and the image of the remote user simultaneously, when the remote user's image is displayed on the spot of 150 during the normal display period. In real time and through internet or other network connection, the remote user of the video conference experiencing the same effect at his or her site if the same system is used.

There can be various modes of camera tracking for eye contact. In one embodiment, the camera position and orientation are automatically adjusted to track eye gazing spot on the screen. For example, as illustrated in FIG. 4A, if the eye gazing spot is at location 420 of a display 410, the position and orientation of the camera 430 are adjusted to be inline with the reflection path 435 at spot 420 of the eye image. One application of this mode is for one-to-many or many-to-many communication, such as remote lecturing, broadcasting, or group conferencing. In this mode, the image of the local user appears as looking at remote audients directly all the time regardless where the speaker is actually looking at. In another embodiment, the camera position and orientation are automatically adjusted to be inline with the reflection path 445 of the eye image as if the local user is looking at spot 450 where the image of a remote user is shown on the screen, as illustrated in FIG. 4B. In this mode, the remote viewer can sense where the local user is staring at. For example, when the local user is looking at display content shown at spot 440 instead of looking at the image of the remote user shown at the spot 450 on the screen, the remote user will sense that the local user is not making eye contact with the remote user. The image of the local user appears to look at display content shown on the left side of the image of the remote user on the screen. This mode can be particularly useful if both local and remote users are having synchronized display contents. Both parties can track where the other person is looking at and therefore they can follow each other on the content displayed on the screen so that the video communication appeals to be more natural for one to one communication.

Two distinctive time periods of the display are depicted in FIG. 5A. One is normal display period 505, during which the display device shows intended display contents. The local user sees normally intended display screen image 510 which may include a sub-window 520 of the remote user's image, as shown in FIG. 5B. The other distinctive time period is dark screen image period 506, during which the display screen shows a dark image, and the local user's image is reflected off from the screen and captured by the camera. Herein, the term “dark” is referring to any dark colors which can make a reflective image visible on the display screen. For example, when display screen shows black, deep blue, deep brown, deep green, or deep purple, the reflective image 530 of the local user will show up as illustrated by FIG. 5C. This image can be captured by a video camera and sent to the remote user.

FIG. 6 illustrates timing diagrams during a video communication in accordance to one embodiment of the present invention. The system is designed to have alternating period between the normally intended display period 505 and the dark image period 506. The frequency of the timing diagram 605 is larger than 1 Hz and preferably larger than 30 Hz for the local user to see continuous display image. The duration 507 of dark period 506 is less than 1 second, preferably less than 16 millisecond in order to avoid image flickering on the local display. The local user's images are captured by the camera during time period 612 which is within the dark image period 506. Herein the period 612 is defined as local user image capture period or image capture period. The timing of screen dark period 506 is synchronized with the image capture period 612. The duration 507 of screen dark period 506 is equal to or slightly longer than the duration 632 of image capture period 612 to ensure the video camera only captures the local user's image without the normally intended display images. The captured images are processed to form video stream and transmitted to a remote site for a remote user to view on his or her screen.

There are various ways to synchronize the timing of the dark period with the image capture period. For example, as shown in FIG. 7, a synchronization unit 710 can be used to trigger the video camera control unit 720 and the display backlight control unit 730 so that the video camera is triggered to capture images only during display dark image period and therefore the camera captures only the images of the reflection which is transmitted through the video output unit 740.

Alternatively, a light sensing device can be coupled to the camera trigger control. When the light sensor senses dark screen, it triggers the video camera to capture the reflected image from the reflective surface over the dark background.

FIG. 8 illustrates using the face-to-face video communication system. Each user, the local user 830 and remote user 860, has a

display

810 and 850 respectively. The captured video images 870 of the local user 830 are processed and transmitted to the remote site for the remote user 860 participating the video communication to view on a display screen 850, and vice versa, the remote user image 820 is displayed on the local display screen 810. In order to ensure direct eye contact between the local user and the image of the remote user, the location of the

image

820 and 870 are positioned at the reflection points on the

display

810 and 850 respectively. The reflection point is defined as

point

150 or 450 in FIG. 1B and FIG. 4B respectively.

FIG. 9 shows an alternative embodiment of the present invention. The display screen is periodically switched to a dark image either by turning off the display back light or by showing a dark image on the screen via software or hardware graphic control of the computer associated with the display. The duration of dark screen is labeled as 915 on the timing diagram 910. For example, time duration 915 can be 200 millisecond (ms) or less. Timing diagram 920 shows a time sequence of the video camera taking images from the reflective surface of the display screen. The time interval 925 between each consecutive image capture (camera frame period) is smaller than the duration of dark screen 915, and therefore the camera will capture images reflected off from the dark screen even if the

timing

910 and 920 are not synchronized well. For example, the camera frame rate can be larger than 5 Hz and the time interval 925 is less than 200 ms. Since a human eye will not be sensitive enough to sense image or light intensity changes in less than 16 ms, it is preferably to have a dark image duration 915 less than 16 ms and camera frame rate faster than 60 Hz.

When the dark image timing diagram 910 and camera capturing time diagram 920 are not synchronized, the camera will capture both reflected local user image and display screen images as illustrated by FIGS. 5A and 5B respectively. These images are processed by an image processor locally. The frames of the reflected images 935 are selected and transmitted to the remote site for the remote user to view as shown in timing diagram 930.

In according to one embodiment of the present invention, the synchronization between the dark image period and camera capture frame can be assisted by making a synchronization mark pattern on the display, as shown in FIG. 10. The synch pattern 1020 comprises one or more distinctive marks 1030. These marks are distributed as a predetermined pattern. For example, they can be evenly distributed on one line as 1020, or as a two dimensional array.

These marks can be overlaid with either the display image 1000 as shown in FIG. 10 or shown as bright images on the dark image. An image processor will look for the synch pattern to determine if an image frame is display image or a reflected image. With the synch pattern, it will be more efficient to filter out the display images and transmit the local user images to the remote site.

In one embodiment of the present invention, a sensing device is coupled to the camera. The sensing device is designed to recognize a given synch mark pattern. Once it senses the synch mark pattern, a trigger pulse will be produced to trigger the camera to capture the reflected local user image.

The foregoing examples illustrate certain exemplary embodiments from which other embodiments, alternatives, variations, and modifications will be apparent to those skilled in the art. Accordingly, the invention is intended to embrace all other such alternatives, modifications, and variations that fall within the spirit and scope of the appended claims.

Claims

What is claimed is:

1. A system for video communication comprising a display having a reflective surface facing to a local user and a video camera mounted on a supporting structure and aimed toward the display; wherein said display displays dark screen images periodically; said camera captures the local user's image reflected off from the surface during the dark screen image period.

2. The system of claim 1, wherein the display is a laptop PC monitor, desktop computer monitor, a display on a mobile device, touch pad computer, and smart phone, a television display, and a video game display.

3. The system of claim 2, wherein the camera is mounted on the outer margin of the key panel of the laptop computer.

4. The system of claim 1, wherein the display is a flat panel display with a reflective surface.

5. The system of claim 1, wherein the display is a projective display with a reflective surface covering the projection image screen.

6. The system of claim 1, wherein the reflective surface is a smooth glossy surface.

7. The system of claim 1, wherein the reflective surface is formed from a reflective coating or a reflective sheet laminated on the display surface.

8. The system of claim 1, wherein the display having a reflective surface comprises a transparent-reflector mounted parallel to and in front of the display.

9. The system of claim 1, wherein the dark screen images are achieved by periodically turning off the backlight of a liquid crystal display.

10. The system of claim 1, wherein the dark screen images are achieved by software or hardware graphic control of the computer associated with the display.

11. The system of claim 1, further comprises a synchronization unit to control the timing of the video camera trigger and exposure time and the dark screen images on the display so that the video camera captures local user's image only during the dark screen image period.

12. The system of claim 1, wherein the camera position and orientation is correlated with the reflection path of the image of the local user via the eye gazing spot on the display screen.

13. The system of claim 1, wherein the camera position and orientation is correlated with the reflection path of the image of the local user as if the local user's eye gazing spot is fixed on the image of a remote user shown on the display screen.

14. The system of claim 1, wherein the video images from the video camera include normally intended displayed images and local user images; the normally intended displayed images are filtered out and replaced by local user images with real time video image processing.

15. The system of claim 14, wherein the normally intended displayed images are filtered out according to a video tracing mark on the display.

16. A method of constructing a system for video communication to track eye contact comprising:

providing a display having a reflective surface facing to a local user;

placing a video camera on a supporting structure and aiming the camera toward the display; Constructing said display to display dark screen images periodically with each period less than 1 second;

arranging the camera to capture local user's image reflected off from the display surface during the dark screen image period.