JP2010213155A - Video conferencing system and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To create such a face-to-face environment that a remote user and a local user work side by side with each other and parallax is small with respect to the real face-to-face environment even when the position of the head of the local user moves. <P>SOLUTION: Remote images 10V and local images 20V are three-dimensionally processed on the basis of a remote viewpoint position 10P and a local viewpoint position 20P. In this manner, adjusted remote images 30V which are images of a remote user 10A viewed from the local viewpoint position 20P and in which the edge of a table on the remote side on the side of the remote user 10A and the edge of a table on the local side on the side of the local user 20A are aligned, are reconstructed by an image reconstruction part 30 and are displayed on the screen by an image display device 23 of the local user 20A. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a video conference technology, and more particularly to a video conference technology that can provide a sense of unity between a remote user and a local user.

  With the development of data communication technology represented by the Internet, a large-capacity video conference such as a video conference is being realized easily. In recent years, video conferencing has been used to generate an image that can provide a sense of unity between users at different locations and display them on the screen to simulate an environment where both users are sitting side by side. Research is progressing to build a system and facilitate joint work.

This kind of environment where you sit side by side in the same direction and collaborate is a kind of face-to-face environment, and when one user moves, the appearance of the other user next to you or the appearance of the desk in front of it It is natural to change according to the movement of the user.
On the other hand, in a general video conference system, a camera for photographing a remote user is fixed. For this reason, the image of the remote user displayed on the display on the local user side does not change even when the local user moves, and there is a large parallax compared to the actual facing environment.

  As a technology to resolve the parallax between the appearance in the actual face-to-face environment and the video displayed on the video conferencing system, the image of the remote user changes with the movement of rotating the head of the local user. A technique for displaying the image on the screen has been proposed (see, for example, Non-Patent Documents 1 and 2).

D. Nguyen, and J. Canny: MultiView: Improving Trust in Group Video Conferencing through Spatial Faithfulness.Proceedings of the 2007 Conference on Human Factors in Computing Systems (CHI'07), pp.1465-1474, 2007 K. Okada et al .: Multiparty Videoconferencing at Virtual Social Distance: MAJIC Design Proceedings of CSCW'94, pp.385-393, 1994 Akabane Shinpei, Omori Kenji: 3D position estimation method using angle of view, IEICE Technical Report. IE, Image Engineering, Vol.105, No.353, pp.1-6, 2005 http://graphics.stanford.edu/~naemura/publication/1996/RayBased-TR.pdf#search='Ray Space Method ', Naemura et al., Ray Space for Efficient Description of 3D Real Space by Ray Information Projection method, University of Tokyo

However, in such a conventional technique, the position where the remote user is displayed on the screen is limited to the position facing the local user, so a facing environment is constructed in which the remote user and the local user work side by side. It is not possible. In addition, although the problem of parallax related to the movement of rotating the head of the local user at the same position can be solved, the problem of parallax related to the movement of moving the position of the head cannot be solved.
The present invention is for solving such problems, and even if the position of the head of the local user moves, there is little parallax with the actual facing environment, and the remote user and the local user work side by side. An object of the present invention is to provide a video conferencing system and method capable of building such a face-to-face environment.

  In order to achieve such an object, a video conference system according to the present invention includes a first imaging device that captures a remote video including a remote user sitting at a first desk, and the remote user's remote video. A first viewpoint position detection unit that detects a remote viewpoint position indicating a three-dimensional viewpoint position; a second imaging device that captures a local image including a local user sitting at a second desk; and A second viewpoint position detection unit for detecting a local viewpoint position indicating a three-dimensional viewpoint position of the local user, and processing the remote video and the local video three-dimensionally based on the remote viewpoint position and the local viewpoint position , When the remote user is viewed from the local viewpoint position, and the remote user side edge of the first desk and the local user of the second desk An image reconstruction unit that reconstructs images aligned with the same edge as the adjustment remote image, and a screen display device that is arranged on the side of the second desk and displays the adjustment remote image on the screen. Yes.

  In the video conference method according to the present invention, the first photographing step includes a first photographing step for photographing a remote image including a remote user sitting at the first desk, and a first viewpoint position detecting unit. A first viewpoint position detecting step for detecting a remote viewpoint position indicating a three-dimensional viewpoint position of the remote user from the remote video, and a second imaging apparatus including a local user including a local user sitting at the second desk A second photographing step for photographing a video, a second viewpoint position detecting step in which a second viewpoint position detecting unit detects a local viewpoint position indicating a three-dimensional viewpoint position of the local user from the local video, and a video Based on the remote viewpoint position and the local viewpoint position, the reconstruction unit processes the remote video and the local video three-dimensionally, so that -The image when the user is viewed and the remote user side edge of the first desk and the local user side edge of the second desk are aligned on the same straight line is reconstructed as an adjusted remote video And a screen display device arranged on the side of the second desk to display the adjusted remote video on the screen.

  ADVANTAGE OF THE INVENTION According to this invention, even if the position of a local user's head moves, there is little parallax with an actual facing environment, and the facing environment where a remote user and a local user work side by side can be constructed | assembled.

It is a block diagram which shows the structure of the video conference system concerning this Embodiment. It is explanatory drawing which shows operation | movement of the video conference system concerning this Embodiment. It is a flowchart which shows operation | movement of the video conference system concerning this Embodiment.

Next, an embodiment of the present invention will be described with reference to the drawings.
First, the video conference system according to the present embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a block diagram showing the configuration of the video conference system according to the present embodiment. FIG. 2 is an explanatory diagram showing the operation of the video conference system according to the present embodiment.

This video conference system 1 has a function of constructing a face-to-face environment in which a local user can get a sense of unity with a remote user by displaying a video of a remote user existing at a different point on the screen on the local user side. .
The video conference system 1 includes, as main functional units, an imaging device (first imaging device) 11, a viewpoint position detection unit (first viewpoint position detection unit) 12, an imaging device (second imaging device) 21, A viewpoint position detection unit (second viewpoint position detection unit) 22, a screen display device 23, and a video reconstruction unit 30 are provided.

The imaging device 11 is composed of one or more general video cameras installed around the remote user 10A, and has a function of capturing a remote video 10V including the remote user 10A sitting on a desk (first desk) 10T. have.
The viewpoint position detection unit 12 has a function of detecting a remote viewpoint position 10P indicating a three-dimensional viewpoint position of the remote user from the remote video 10V photographed by the photographing apparatus 11.

The photographing device 21 is composed of one or more general video cameras installed around the local user 20A, and has a function of photographing a local video 20V including the local user 20A sitting on a desk (second desk) 20T. have.
The viewpoint position detection unit 22 has a function of detecting a local viewpoint position 20P indicating the three-dimensional viewpoint position of the local user from the local video 20V captured by the imaging device 21.
Note that a technique for recognizing a person's eyes from an input video and calculating a three-dimensional coordinate value as a viewpoint position, which is necessary for the viewpoint position detection unit 12 and the viewpoint position detection unit 22, is known in Non-Patent Document 3 and the like. Technology can be used.

  The screen display device 23 includes a display device such as an LCD or a PDP disposed on the side of the desk 20T of the local user 20A, and has a function of displaying the adjusted remote video 30V reconstructed by the video reconstruction unit 30 on the screen. Have. In the example of FIG. 2, since the remote user 10A is photographed from the left hand side of the remote user 10A, the local user 20A of the desk 20T is matched so that the direction of the remote video 10V matches the line-of-sight direction of the local user 20A. A screen display device 23 is disposed on the right-hand side.

The video reconstruction unit 30 is configured to use the remote viewpoint 10P detected by the viewpoint position detection unit 12 and the local viewpoint position 20P detected by the viewpoint position detection unit 22, based on the remote video 10V captured by the imaging device 11. And the local video 20V are processed in a three-dimensional manner so that the remote user 10A is viewed from the local viewpoint position 20P, and the remote user side edge 10B of the desk 10T and the local user side edge of the desk 20T. The image having the edge 20B aligned on the same straight line has a function of reconstructing as an adjusted remote image 30V.
As a technique for reconstructing a video viewed from an arbitrary viewpoint position, which is necessary for the video reconstruction unit 30, a known technique such as Non-Patent Document 4 may be used.

  The viewpoint position detection unit 12, the viewpoint position detection unit 22, and the video reconstruction unit 30 may each be configured by a dedicated image processing circuit, or by an arithmetic processing unit that executes an image processing program on the CPU. Also good. In addition, the viewpoint position detection unit 12, the viewpoint position detection unit 22, and the video reconstruction unit 30 may be realized by individual devices, or may be realized by a single device by combining a plurality or all of them. Further, the viewpoint position detection unit 12, the viewpoint position detection unit 22, and the video reconstruction unit 30 may be arranged on either the remote side 10 or the local side 20, and an intermediate position connecting the remote side 10 and the local side 20, For example, you may arrange | position in a communication network.

[Operation of this embodiment]
Next, the operation of the video conference system according to the present embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing the operation of the video conference system according to the present embodiment.

On the remote side 10, after the video conference communication is started, the photographing device 11 always photographs the remote user 10A and outputs a remote video 10V (step 100). At this time, the remote video 10V includes not only the remote user 10A but also the desk 10T of the remote user 10A.
The viewpoint position detection unit 12 detects a remote viewpoint position 10P indicating the three-dimensional viewpoint position of the remote user 10A from the remote video 10V from the photographing apparatus 11 (step 101). At this time, it is assumed that processing information such as three-dimensional coordinates indicating the photographing position of the photographing apparatus 11 necessary for detecting the remote viewpoint position 10P is set in advance.

In parallel with this, on the local side 20, after the video conference communication is started, the photographing device 21 always photographs the local user 20A (step 110) and outputs the local video 20V (step 110). At this time, the local video 20V includes not only the local user 20A but also the desk 20T of the local user 20A.
The viewpoint position detection unit 22 detects a local viewpoint position 20P indicating the three-dimensional viewpoint position of the local user 20A from the local video 20V from the imaging device 21 (step 111). At this time, it is assumed that processing information such as three-dimensional coordinates indicating the photographing position of the photographing device 21 necessary for detecting the local viewpoint position 20P is set in advance.

  The video reconstruction unit 30 is configured to use the remote viewpoint 10P detected by the viewpoint position detection unit 12 and the local viewpoint position 20P detected by the viewpoint position detection unit 22, based on the remote video 10V captured by the imaging device 11. The adjusted remote video 30V is reconstructed by three-dimensionally processing the local video 20V (step 120).

  At this time, for example, the video reconstruction unit 30 extracts the remote user side edge 10B of the desk 10T from the remote video 10V, and extracts the local user side edge 20B of the desk 20T from the local video 20V. Subsequently, the remote user side edge 10B and the local user side edge 20B are aligned on the same straight line, in other words, the remote user side edge 10B is positioned on the extension line of the local user side edge 20B. The three-dimensional coordinates of the video 10V and the local video 20V are matched, and the local viewpoint position 20P is arranged in the three-dimensional space of the remote video 10V. At this time, information such as the positional relationship between the remote viewpoint position 10P and the local viewpoint position 20P is set in advance.

  Next, an image viewed from the local viewpoint position 20P is cut out from the three-dimensional space of the remote image 10V. Accordingly, the adjustment remote is an image when the remote user 10A is viewed from the local viewpoint position 20P, and the remote user side edge 10B of the desk 10T and the local user side edge 20B of the desk 20T are aligned on the same straight line. Video 30V is reconstructed.

The screen display device 23 receives the adjusted remote video 30V reconstructed by the video reconstruction unit 30 in this way and displays it on the screen (step 121).
In this way, steps 100 to 121 are repeatedly executed until the video conference communication ends.

[Effects of the present embodiment]
As described above, according to the present embodiment, the remote video 10V and the local video 20V are three-dimensionally processed based on the remote viewpoint position 10P and the local viewpoint position 20P, so that the remote user 10A from the local viewpoint position 20P. The remote user side edge 10B of the first desk 10T and the local user side edge 20B of the second desk 20T are aligned on the same straight line. Reconstructed by the reconstruction unit 30 and displayed on the screen display device 23 of the local user 20A.

Therefore, the edges of the desks of both users can be seen in a straight line, and images corresponding to changes in the viewpoint position can be displayed, and both parties share the same desk and feel as if they are sitting next to each other. A face-to-face environment can be established.
In addition, since the viewpoint position of the local user is taken into consideration, even when the position of the head of the local user moves, an image with little parallax with the actual facing environment can be reconstructed. Further, since the user's viewpoint position is detected from the video, it is not necessary to attach a special sensor or the like to the user's face, and the burden on the user can be avoided.

  DESCRIPTION OF SYMBOLS 1 ... Video conference system, 10 ... Remote side, 10A ... Remote user, 10B ... Remote user side edge, 10P ... Remote viewpoint position, 10V ... Remote image | video, 10T ... Desk, 11 ... Imaging device, 12 ... Viewpoint position detection part , 20 ... Local side, 20A ... Local user, 20B ... Local user side edge, 20P ... Local viewpoint position, 20V ... Local video, 20T ... Desk, 21 ... Shooting device, 22 ... Viewpoint position detection unit, 23 ... Screen display Device, 30 ... Video reconstruction unit, 30V ... Adjusted remote video.

Claims (2)

A first photographing device for photographing a remote image including a remote user sitting at a first desk;
A first viewpoint position detection unit that detects a remote viewpoint position indicating a three-dimensional viewpoint position of the remote user from the remote video;
A second photographing device for photographing a local image including a local user sitting at a second desk;
A second viewpoint position detection unit that detects a local viewpoint position indicating a three-dimensional viewpoint position of the local user from the local video;
Based on the remote viewpoint position and the local viewpoint position, the remote video and the local video are processed in a three-dimensional manner, so that the remote user is viewed from the local viewpoint position, and A video reconstruction unit for reconstructing an image in which the remote user side edge of the first desk and the local user side edge of the second desk are aligned on the same straight line as an adjusted remote video;
A video conferencing system comprising: a screen display device arranged on a side of the second desk and displaying the adjusted remote video on a screen. A first imaging step in which a first imaging device captures a remote video including a remote user sitting at a first desk;
A first viewpoint position detection unit, wherein a first viewpoint position detection unit detects a remote viewpoint position indicating a three-dimensional viewpoint position of the remote user from the remote video;
A second imaging step in which a second imaging device captures a local image including a local user sitting at a second desk;
A second viewpoint position detection step in which a second viewpoint position detection unit detects a local viewpoint position indicating a three-dimensional viewpoint position of the local user from the local video;
Video when the video reconstruction unit views the remote user from the local viewpoint position by processing the remote video and the local video three-dimensionally based on the remote viewpoint position and the local viewpoint position And a video reconstruction step of reconstructing a video in which the remote user side edge of the first desk and the local user side edge of the second desk are aligned on the same straight line as an adjusted remote video; ,
A screen conferencing method, comprising: a screen display step disposed on a side of the second desk to display the adjusted remote video on the screen.

Images