JP2011113206A - System and method for video image communication - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a user with a feeling of a distance and reality with a dialogue counterpart in a video image. <P>SOLUTION: An image acquisition unit (user B) 10 acquires a two-dimensional video image of the user B, and a camera focal distance and a camera view angle at the image acquisition. A viewpoint position detector (user A) 11 acquires the viewpoint position of the user A. A position controller (user B) 12 controls the position and the posture of the image acquisition unit (user B) 10 so as to make the viewpoint position of the image acquisition unit (user B) 10 of the user B coincident with the viewpoint position V of the user A. A viewpoint position image generator (user A) 13 generates the two-dimensional image of the user B corresponding to the viewpoint position of the user A from the two-dimensional image of the user B, the camera focal distance, the camera view angle and the viewpoint position of the user A. A video image display unit (user A) 14 displays the two-dimensional image of the user B corresponding to the viewpoint position of the user A. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to video communication, in which a two-dimensional display is viewed as a single window, and as if the users are looking through each other's space through the window according to the user's viewpoint position, The present invention relates to a video communication system and a video communication method for realizing a depth expression by realizing that a video of a conversation partner's space seen from the position is displayed on a display (motion parallax).

          In video communication, it is important to present the conversation partner as a video to the user as if it is facing the conversation partner. Conventionally, in video communication, in order to make it feel as if a real-size conversation partner is in front of you, there has been considered a method in which a person who is a conversation partner is projected on a two-dimensional display in real size (for example, patents). Reference 1).

Japanese Patent No. 3586126

  In face-to-face conversations, the sense of distance between humans is important because it affects the ease of speaking and tension (Reference 1: Edward Hall, Hidden Dimensions, 2000, Misuzu Shobo). However, in a conversation partner displayed on a conventional two-dimensional display, the image displayed on the two-dimensional display does not change regardless of the position at which the image is observed. Since there was no image change depending on the viewpoint position, depth information was missing and the distance to the conversation partner could not be expressed.

  By the way, the actual size display alone is not enough to make it feel like the other party is in front of you. For example, even if you see a person who is displayed on the display in a life-size manner, the person may feel that the person is in front of or behind the display, or the depth position in the first place. It feels like there is a larger or smaller person, and the conversation partner cannot be felt life-size. As described above, since the conventional technology cannot express a sense of distance, it cannot be expected to have an effect on conversation by distance as in face-to-face conversation.

  In addition, in the prior art, it was impossible to grasp how far the user was from the conversation partner in the video, so as with face-to-face conversation, the interpersonal distance could be adjusted appropriately, I couldn't feel the tension I could bring. Furthermore, in addition to not knowing where the conversation partner's line of sight and attitude is facing, the sense of sharing the same space with the conversation partner in the remote location (realism) is missing. It was.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a video communication system and a video communication method capable of giving a user a sense of distance and reality with a conversation partner in a video. It is to provide.

  In order to solve the above-described problem, the present invention provides a video transmission on the display side at least between a video transmission device on the imaging side and a video transmission device on the display side. An image communication system for transmitting to an apparatus, wherein the imaging-side video transmission apparatus captures a two-dimensional image including a person on the imaging side and acquires the two-dimensional image and a photographing condition at the time of photographing. The display-side video transmission device includes viewpoint position detection means for detecting the viewpoint position of the display-side person, and video display means for displaying the two-dimensional image on the imaging side. The video transmission apparatus according to the present invention is a position for controlling the position / posture of the image acquisition means so that the viewpoint position of the person on the display side detected by the viewpoint position detection means matches the line-of-sight position of the image acquisition means. The display-side video transmission apparatus includes the viewpoint position of the display-side person detected by the viewpoint position detection unit, the two-dimensional image acquired by the image acquisition unit, and the shooting conditions at the time of shooting. And a viewpoint position for generating a two-dimensional image on the imaging side corresponding to the viewpoint position of the person on the display side to display on the video display means based on the display information of the video display means on the display side An image communication system comprising image generation means.

  According to the present invention, in the above invention, the viewpoint position image generation unit includes a two-dimensional image including a person on the imaging side photographed by the image obtaining unit, photographing conditions at the time of photographing, and the video on the display side. An input means for inputting display information of the display means; and an actual size of a two-dimensional image including the person on the imaging side photographed by the image acquisition means based on the photographing conditions at the time of photographing input by the input means A two-dimensional image actual size calculating means for calculating the size information of the image, a two-dimensional image including the person on the imaging side photographed by the image acquiring means, and the two-dimensional image actual size calculating means Based on the actual size information, the shooting conditions at the time of shooting, and the display information of the video display means on the display side, to be displayed on the video display means on the imaging side, Characterized in that it comprises an image generating means for generating a two-dimensional image of the imaging-side in accordance with the person of the viewpoint position of 示側.

  According to the present invention, in the above-described invention, the image generation unit is an actual size of the two-dimensional image located at a position separated from a viewpoint position of the person on the display side by a focal length at the time of photographing by the image acquisition unit. A projection conversion process is performed on the display surface of the video display means on the display side by converting a two-dimensional image including the person on the imaging side photographed by the image acquisition means having the size indicated by the information information to the display side A two-dimensional image on the imaging side corresponding to the viewpoint position of the person is generated.

  The present invention is characterized in that, in the above invention, the photographing condition at the time of photographing includes at least a focal length at the time of photographing by the image obtaining unit and an angle of view at the time of photographing by the image obtaining unit.

  In the present invention according to the present invention, the display information includes at least the resolution of the video display means and the actual size of height and width.

  Further, in order to solve the above-described problem, the present invention provides an image captured by the imaging-side video transmission device at least between the imaging-side video transmission device and the display-side video transmission device. A video communication method for transmitting to a video transmission apparatus, wherein the display-side video transmission apparatus detects a viewpoint position of the display-side person by viewpoint position detection means, and the imaging-side video transmission apparatus includes: Taking a two-dimensional image including the person on the imaging side by the image obtaining means, obtaining the two-dimensional image and photographing conditions at the time of photographing, and the viewpoint position of the person on the display side and the photographing means In order to match the line-of-sight position, the step of controlling the position / posture of the image acquisition means, the viewpoint position of the person on the display side, the two-dimensional image acquired by the image acquisition means and Based on the shadow condition and the display information of the video display means on the display side, a two-dimensional image on the imaging side corresponding to the viewpoint position of the person on the display side to be displayed on the video display means is generated. And a step of displaying a two-dimensional image on the imaging side corresponding to the viewpoint position of the person on the display side on the video display means on the display side.

  According to the present invention, the motion parallax of changing the conversation partner's video in accordance with the user's viewpoint (eye position) is realized, and the conversation partner is displayed with a person size corresponding to the distance, so that the video is displayed to the user. Can give a sense of distance and reality with the conversation partner.

It is a conceptual diagram which shows the image of the video display by this invention. It is a conceptual diagram for demonstrating the outline | summary of this invention. It is a block diagram which shows the structure of the video communication apparatus by this embodiment. It is a conceptual diagram for demonstrating the production | generation method of the two-dimensional image of the user B according to the viewpoint position of the user A by this embodiment. It is a flowchart for demonstrating operation | movement in the viewpoint position image generation part (user A) 13 by this embodiment. In this embodiment, it is a conceptual diagram for demonstrating the calculation method of the magnitude | size information of the actual size of the two-dimensional image of the user B. FIG. It is a conceptual diagram for demonstrating perspective transformation by this embodiment.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

A. Principle of the Invention In the present invention, in video communication, the position / posture of a camera on the other side of the conversation is moved according to the position of the user's eyes relative to the display detected by a sensor (camera, etc.) ) Is projected on the display surface of the user side and displayed on the display. Thereby, since the image of the conversation partner according to the movement of the eye position of the user (head movement) can be seen, a high presence is obtained by motion parallax.

  That is, according to the present invention, in the video communication at two points, the user's viewpoint position is as if the user and the conversation partner are looking into each other's space through the window with the display as a single window. Accordingly, it is possible to display an image of the conversation partner's space seen from the position on the display (motion parallax) and to express the depth position of the conversation partner in the image.

  FIG. 1 is a conceptual diagram illustrating an image display image according to the present invention. In FIG. 1, when the user 1 changes the viewpoint (corresponding to three-axis movement), the depth position of the conversation partner 2 in the video (the conversation partner 2 and the background wall 3) is two-dimensionally changed according to the change of the viewpoint. By displaying on the display 4, the appearance of the conversation partner 2 is reproduced as if it were real.

  That is, when the user 1 faces the two-dimensional display 4 at the position P1, the conversation partner 2 is displayed as if viewed from the front, and when the user 1 is viewed from the left position P2, the conversation is performed. Reflecting the positional relationship (depth) with the partner 2 and the wall 3 in the background, displaying the conversation partner 2 as seen from the left side, and when the user 1 is viewing from the right side position P3, The position (depth) of the conversation partner 2 and the wall 3 in the background is reflected, and the conversation partner 2 is displayed as if viewed from the right side. In order to realize the present invention, the viewpoint position of the user 1 with respect to the display 4 is detected, and an image of the conversation partner 2 corresponding to the detected viewpoint position is captured and displayed. The outline of the present invention will be described below.

  FIG. 2 is a conceptual diagram for explaining the outline of the present invention. In accordance with the viewpoint position V1 of the user 1 on the point A side, the position / posture of the camera 5 that captures the conversation partner 2 on the point B side is controlled, and the viewpoint position V1 of the user 1 and the camera 5 with respect to the conversation partner 2 are controlled. By matching the viewpoint position V <b> 2, an image that should be seen from the position of the user 1 is always captured by the camera 5 and the image is projected on the display 4. This realizes an image expression as if the display 4 is viewed as a single window, as if looking into the space of the conversation partner 2 through the window.

B. Embodiment FIG. 3 is a block diagram illustrating a configuration of a video communication apparatus according to the present embodiment. The illustrated video communication apparatus is prepared for each of a user 1 (hereinafter referred to as user A) and a conversation partner 2 (hereinafter referred to as user B), and is used via two-way communication. Hereinafter, a case where the user B's spatial video is displayed on the user A's display will be described.

The image acquisition unit (user B) 10 corresponds to the camera 5 in FIG. 2, acquires the two-dimensional video of the user B, the camera focal length and the camera angle of view at the time of image acquisition, and the viewpoint position image generation unit (user A) ) 13 through a communication line such as a network. The viewpoint position detection unit (user A) 11 uses an image processing technique using face recognition and stereo matching, an optical or magnetic motion tracking technique, and the like on the display 4 (see FIGS. 1 and 2). On the other hand, the viewpoint position V (V _x , V _y , V _z ) of the user A indicating where the viewpoint (eye) position of the user A is is obtained and supplied to the position control unit (user B) 12.

Next, the position control unit (user B) 12 determines the position of the image acquisition unit (user B) 10 of the user B, that is, the camera 5 with respect to the display 6 (see FIG. 2), the viewpoint position V (V _x , In order to match (follow) V _y , V _z ), the position / posture of the image acquisition unit (user B) 10 is controlled. At this time, the viewpoint position V (V _x , V _y , V _z ) of the user A is located on the back side of the display 6 when viewed from the user B, as shown in FIG. Therefore, ingenuity is necessary for photographing the user B.

  For example, the display 6 is a transmissive type. Alternatively, a half mirror (not shown) is installed between the display 6 and the user B so that the angle formed by the normal line of the plane of the image acquisition unit (user B) 10 and the normal line of the half mirror is 45 degrees. Then, the image acquisition unit (user B) 10 is arranged above or below the half mirror so as to be virtually the same position as the user viewpoint position. Further, in the position / posture control, as the movement in the front-rear direction, in addition to physically moving the image acquisition unit (user B) 10, an optical zoom or a digital zoom of the image acquisition unit (user B) 10 is also included. Is available.

  At this time, the orientation (posture) of the image acquisition unit (user B) 10 is directed toward the display. For example, the optical axis of the lens of the image acquisition unit (user B) 10 always passes through the center of the display 6. Hereinafter, description will be made under this condition.

Next, the viewpoint position image generation unit (user A) 13 captures the two-dimensional image of the user B taken by the image acquisition unit (user B) 10, that is, the camera 5, the camera focal length, the camera angle of view, and Using the viewpoint position V (V _x , V _y , V _z ) of the user A relative to the display position detected by the viewpoint position detection unit (user A) 11 as an input, the video display unit (user A) 14 on the user A side A two-dimensional image of the user B corresponding to the viewpoint position V (V _x , V _y , V _z ) of the user A to be output is generated.

The video display unit (user A) 14 is a two-dimensional user B according to the viewpoint position V (V _x , V _y , V _z ) of the user A generated by the viewpoint position image generation unit (user A) 13. Display an image.

FIG. 4 is a conceptual diagram for explaining a method for generating a two-dimensional image of the user B according to the viewpoint position of the user A according to the present embodiment. As shown in FIG. 4, on the user B side, an image acquisition unit whose position and orientation are controlled so as to coincide with the viewpoint position V (V _x , V _y , V _z ) of the user A through the display 6 of the user B. (User B) By projecting the two-dimensional image of user B taken by user 10 onto the four display surfaces of user A on user A side, user A's viewpoint position V (V _x , V _y , V _z ) A two-dimensional image of the user B is generated according to the above.

Next, the main operation of this embodiment will be described.
FIG. 5 is a flowchart for explaining the operation of the viewpoint position image generation unit (user A) 13 according to the present embodiment. First, the viewpoint position image generation unit (user A) 13 displays a two-dimensional image of the user B taken by the image acquisition unit (user B) 10, a camera focal length, a camera angle of view, and a video display on the user A side. Display information (resolution of display 4, actual size of height and width) of unit 14 is input (step S10). Next, the viewpoint position image generation unit (user A) 13 calculates and gives the actual size information to the two-dimensional image of the user B (step S11).

Here, a method for calculating the actual size information of the two-dimensional image of the user B will be described below.
FIG. 6 is a conceptual diagram for explaining a calculation method of the actual size information of the two-dimensional image of the user B in the present embodiment. The height α _h and width α _w of the two-dimensional image of the user B shown in FIG. 6 are expressed by the following equations (1) and (1) using the camera angle of view (vertical θ _h and horizontal θ _w ) and the camera focal length D. 2).

Next, the viewpoint position image generation unit (user A) 13 has a camera focal length D and size information, and has a two-dimensional image (height α _h and width α _w ) of the user B, and a video display unit 14 of the user A. The image to be displayed on the video display unit 14 of the user A is generated using the display information (step S12). As shown in FIG. 4, the generated image has size information at a position separated from the viewpoint position V (V _x , V _y , V _z ) (= camera viewpoint position) of the user A by the camera focal length. The two-dimensional image of the user B can be acquired by performing projection such as perspective projection on the display surface of the video display unit 14 of the user A.

  First, the coordinate system of the viewpoint position of the user A, the two-dimensional image of the user B, and the display position of the video display unit 14 for the user A is unified. At this time, the coordinate system has the actual size information using the display information of the video display unit 14 of the user A and the like. Here, size information on the display surface of the video display unit 14 for the user A is newly required.

Next, an example of actual perspective transformation using the size information of the display surface of the video display unit 14 for the user A is shown. FIG. 7 is a conceptual diagram for explaining perspective transformation according to the present embodiment. As shown in FIG. 7, the center of the display is the origin O (0, 0, 0) of the coordinate system, the x-axis is the horizontal direction of the coordinate display surface, the y-axis is the vertical direction, and the normal of the display surface is the z-axis. The reverse direction of the user position is the positive direction. User B's viewpoint position V (V _x , V _y , V _z ) (where V _z <0), user B should be projected to a point D (x, y, z) on the display surface. The point H (h _x , h _y , h _z ) of the two-dimensional image is calculated by the equation (3).

  In addition to the perspective projection, a general projective transformation method such as parallel projection can be used.

  According to the embodiment described above, the video changes according to the user's viewpoint (eye position), that is, realizes motion parallax, and displays the conversation partner with a person size according to the distance. Since the same information is obtained as when facing one window visually, the conversation partner in the video is given a clue about the depth, and the user grasps the sense of distance from the conversation partner in the video can do. Thereby, it is possible to adjust the distance appropriately as in the face-to-face conversation, and to bring about the effects of ease of talking and presence or absence of tension that the distance brings in the face-to-face conversation. In addition, as described above, since the video changes, it is possible to make it feel as if a full-size conversation partner is in front of you, compared to the case where a person is presented as a two-dimensional video as in the prior art. Non-linguistic information such as line of sight and posture is transmitted in the same way as in face-to-face conversation in real space, and it is possible to give a feeling of sharing the same space (realism) with a conversation partner in a remote place.

  A program for realizing each step by the designated position image generation unit 13 described above is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. Various processes may be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 10 Image acquisition part 11 Viewpoint position detection part 12 Position control part 13 Viewpoint position image generation part 14 Image | video display part

Claims (6)

A video communication system for transmitting video captured by the video transmission device on the imaging side to the video transmission device on the display side at least between the video transmission device on the imaging side and the video transmission device on the display side,
The imaging-side video transmission device is:
An image acquisition means for capturing a two-dimensional image including the person on the imaging side and acquiring the two-dimensional image and a photographing condition at the time of photographing;
The display-side video transmission device includes:
Viewpoint position detecting means for detecting the viewpoint position of the person on the display side;
Video display means for displaying a two-dimensional image on the imaging side,
The imaging-side video transmission device is:
A position control means for controlling the position / posture of the image acquisition means in order to match the viewpoint position of the person on the display side detected by the viewpoint position detection means with the line-of-sight position of the image acquisition means;
The display-side video transmission device includes:
The viewpoint position of the person on the display side detected by the viewpoint position detection means, the two-dimensional image acquired by the image acquisition means and the shooting conditions at the time of shooting, and the display information of the video display means on the display side; And a viewpoint position image generating means for generating a two-dimensional image on the imaging side corresponding to the viewpoint position of the person on the display side for displaying on the video display means. . The viewpoint position image generation means includes
An input means for inputting a two-dimensional image including a person on the imaging side photographed by the image acquisition means, a photographing condition at the time of photographing, and display information of the video display means on the display side;
Two-dimensional image actual size calculation means for calculating the actual size information of the two-dimensional image including the person on the imaging side photographed by the image acquisition means based on the photographing condition at the time of photographing input by the input means. When,
A two-dimensional image including the person on the imaging side photographed by the image acquisition means, the actual size information of the two-dimensional image calculated by the two-dimensional image actual size calculation means, and the photographing conditions at the time of photographing, Image generation for generating a two-dimensional image on the imaging side corresponding to the viewpoint position of the person on the display side to be displayed on the video display means on the imaging side based on display information of the video display means on the display side The video communication system according to claim 1, further comprising: means. The image generation means includes
By the image acquisition means having the size indicated by the actual size information of the two-dimensional image located at a position away from the viewpoint position of the person on the display side by a focal length at the time of shooting of the image acquisition means. The two-dimensional image including the imaged person on the imaging side is subjected to projective transformation processing on the display surface of the video display means on the display side, so that the two-dimensional image on the imaging side corresponding to the viewpoint position of the person on the display side is obtained. The video communication system according to claim 2, wherein the video communication system is generated. The shooting conditions at the time of shooting are as follows:
4. The video communication system according to claim 1, wherein the video communication system includes at least a focal length at the time of shooting by the image acquisition unit and an angle of view at the time of shooting by the image acquisition unit. The display information is
5. The video communication system according to claim 1, wherein the video communication system includes at least the resolution of the video display means and the actual size of height and width. A video communication method for transmitting video captured by the video transmission device on the imaging side to the video transmission device on the display side at least between the video transmission device on the imaging side and the video transmission device on the display side,
The display-side video transmission device detecting a viewpoint position of the display-side person by viewpoint position detection means;
The imaging-side video transmission apparatus captures a two-dimensional image including the imaging-side person by using an image acquisition unit, and acquires the two-dimensional image and a capturing condition at the time of capturing;
Controlling the position / posture of the image acquisition means in order to match the viewpoint position of the person on the display side and the line-of-sight position of the photographing means;
Displayed on the video display unit based on the viewpoint position of the display side person, the two-dimensional image acquired by the image acquisition unit, the shooting conditions at the time of shooting, and the display information of the video display unit on the display side Generating a two-dimensional image on the imaging side in accordance with a viewpoint position of the person on the display side,
Displaying the two-dimensional image on the imaging side corresponding to the viewpoint position of the person on the display side on the video display means on the display side.

Images