WO2016159166A1

WO2016159166A1 - Image display system and image display method

Info

Publication number: WO2016159166A1
Application number: PCT/JP2016/060533
Authority: WO
Inventors: 康夫高橋; 吏中野; 貴司折目; 雄一郎竹内; 暦本　純一; 宮島　靖
Original assignee: 大和ハウス工業株式会社; ソニー株式会社
Priority date: 2015-03-31
Filing date: 2016-03-30
Publication date: 2016-10-06
Also published as: JP6461679B2; JP2016192688A

Abstract

In a case where the installation height of a camera differs from the eye height of a user displayed on a display, the present invention improves the realistic sensation of a conversation to be carried out while an image of the user is being displayed on the display. This image display system obtains an image of the user captured by a camera, divides the image into a prescribed number of image pieces, obtains distance data indicating the distance between the camera and an object in an image piece for each of the image pieces, and executes a rendering process using the image of the user and the distance data to create a three-dimensional image of the user. The eye height of the user is detected, and when the detected eye height differs from the installation height of the camera, the rendering process which is for obtaining the three-dimensional image of the user when viewed from a virtual visual point located at the detected eye height is executed on the basis of the difference between the two heights and the distance between the camera and the user.

Description

Video display system and video display method

The present invention relates to an image display system and an image display method, and more particularly, to an image display system and an image display method for displaying an image of a conversation partner in a remote place on a display of a conversation person.

A communication system (hereinafter referred to as a video display system) that enables users in remote spaces to interact while watching each other's video is already known. In this system, video data of video is transmitted from one user side, and the video data is received and expanded on the other user side. Thereby, the video of one user comes to be displayed on the display of the other user. As a result, users watching each other's images on the display feel as if they are facing each other.

Also, among the video display systems described above, there is a system that displays captured video in three dimensions using texture mapping or the like (see, for example, Patent Document 1). By displaying the three-dimensional video (hereinafter, three-dimensional video) in this way, it is possible to further improve the realism of the dialogue performed while displaying the other party's video on the display.

Furthermore, among the above video display systems, there is a system that can match the eyes of the person who is looking at the display and the eyes of the person shown on the display for the purpose of further enhancing the realism of the dialogue. Exists (for example, see Patent Documents 1 to 3). Specifically, in the systems described in

Patent Documents

1 and 2, the installation position of the camera is appropriately determined in advance so that the positions of the line of sight coincide. Moreover, in the system described in Patent Document 3, the position of the camera that captures the image of the person shown on the display is moved up and down according to the height of the eyes of the person who is looking at the display, thereby matching the eyes of both. Let

JP 2014-86774 A JP 2000-32420 A Special table 2014-522622 gazette

However, in the systems described in

Patent Documents

1 and 2, the installation position of the camera is determined so that the positions of the eyes coincide with each other, so that the height of the eyes is limited. That is, since the installation position of the imaging camera is fixed, it becomes a system that is difficult for a person who has a line of sight at a height different from the installation position (specifically, the position of the line of sight does not match). On the other hand, in the system described in Patent Document 3, the position of the camera can be adjusted according to the height of the eyes of the person viewing the display. Since it is necessary to provide an adjustment mechanism, the system construction cost is expensive.

In addition, in order to further improve the realism of dialogue using the video display system, the movement of the person watching the display (especially the movement of the face) and the change in the position of the person shown on the display are followed. Thus, it is necessary to switch the video on the display. Specifically, when the face of the person watching the display moves sideways, it reflects the appearance of the person moving the face sideways in a situation where the person is actually facing the conversation partner. It is desirable to change the image.

Also, the farther the camera subject is from the camera, the smaller the display size of the subject image on the display. However, in a situation where the conversation is actually conducted while facing each other, the figure (size) of the other person when the other person is slightly separated from one of the parties is It seems that there is almost no change in how people look (look). In consideration of such appearance, it is desirable to adjust the display size of the image of the subject when the distance between the subject and the camera, that is, the depth distance changes.

Therefore, the present invention has been made in view of the above problems, and the object of the present invention is to display the user on the display when the height of the user's eyes projected on the display is different from the installation height of the imaging device. It is an object to provide a video display system and a video display method capable of improving the realistic sensation of a dialogue performed while displaying the video.
Another object of the present invention is to change the display image on the display to reflect the actual appearance when the face of the second user who is watching the user image displayed on the display moves sideways. It is. A third object of the present invention is to appropriately adjust the display size of the user's video displayed on the display when the distance between the user and the imaging device changes.

According to the video display system of the present invention, (A) a video acquisition unit that acquires a video of a user captured by an imaging device, and (B) when the video is divided into a predetermined number of video pieces. For each video piece, a distance data acquisition unit that acquires distance data indicating a distance between the imaging device and the object in the video piece, and (C) the user's video and the distance data are used. A 3D image generation unit that generates a 3D image of the user by executing a rendering process; and (D) a height detection unit that detects the height of the user's eyes, and (E) When both the height at which the imaging device is installed and the height of the eyes detected by the height detection unit are different, the 3D image generation unit determines the difference between the two and the imaging device and the user. Based on the distance between, the height detector It is solved by executing the rendering process for acquiring the three-dimensional image of the user when viewed from a virtual viewpoint in the eye of the height and knowledge.

According to the above configuration, the user's 3D video is generated by executing the rendering process using the user's video captured by the imaging device and the distance data acquired for the user's video. If the height of the user's eyes is different from the height at which the imaging device is installed, the user's three-dimensional image when viewed from a virtual viewpoint at the same height as the user's eyes Execute the rendering process so that In this way, by rendering processing as 3DCG technology, a 3D image of a user viewed virtually from the same height as the user's eyes is obtained, so that even if the heights of both are different, the display can be viewed. It is possible to match the eyes of the person who is present and the eyes of the person shown on the display. As a result, it is possible to improve the realism of the dialogue performed while displaying the user's video on the display.

In the video display system, the video acquisition unit acquires the user's video captured by the imaging device and the background video captured by the imaging device, and the distance data acquisition unit includes: The distance data is acquired for each of the user image and the background image, and the 3D image generation unit uses the distance data acquired for the user image and the user image. Generating the 3D image of the user by executing a process, and executing the rendering process using the background data and the distance data acquired for the background image, Generating an original video, combining the 3D video of the user and the 3D video of the background, When having a combined image display unit for displaying a combined image in which the user is positioned in front of the background to the display, which is preferable.
In the above configuration, the 3D video of the user and the 3D video of the background are synthesized, and the synthesized video in which the user is positioned in front of the background is displayed. By displaying the composite video having such a feeling of depth, the realism of the dialogue performed while displaying the user's video on the display is further improved.

In the video display system, the video acquisition unit further acquires a foreground video captured by the imaging device, and the distance data acquisition unit further acquires the distance data regarding the foreground video. The 3D image generation unit further generates the 3D image of the foreground by executing the rendering process using the distance data acquired for the foreground image and the foreground image, and the composition The video display unit synthesizes the 3D video of the user, the 3D video of the background, and the 3D video of the foreground, the user is positioned in front of the background, and It is more preferable to display the composite image in which the foreground is located on the display.
In the above configuration, in addition to the 3D video of the user and the 3D video of the background, the 3D video of the foreground is further synthesized, and the synthesized video with the foreground positioned in front of the user is displayed. As a result, a composite image having a greater sense of depth is displayed. As a result, the realism of the dialogue performed while displaying the user's video on the display is further improved.

The video display system may further include a determination unit that determines whether the distance between the imaging device and the user has changed based on the distance data, and the imaging device captures an image of the user. When the determination unit determines that the distance between the imaging device and the user has changed, the composite video display unit displays the display size of the user video in the composite video, It is more preferable to adjust the display size before the distance between the imaging device and the user is changed.
According to the above configuration, even when the distance between the imaging device and the user, that is, the depth distance changes, the display displays the user's three-dimensional video with the display size before the change. . That is, when the user's depth distance changes, the composite image after the change reflects how it looks when the user actually sees the user (ie, the size of the user recognized through his / her own vision). The 3D image of the user is displayed at the display size. As a result, the realism of the dialogue performed while displaying the user's video on the display is further improved.

The video display system may further include a face movement detection unit that detects that the face of a second user who views the composite video displayed on the display has moved in the width direction of the display. When the detection unit detects the movement of the face, the composite video display unit transitions the composite video displayed on the display from a state before the face movement detection unit detects the movement of the face. A transition process is performed, and in the transition process, one of a display position of the 3D video of the user in the composite video and a range included in the composite video in the background 3D video It is even more preferable that the composite video is shifted to a state in which it is shifted along the width direction by any amount larger than the other shift amount.
According to the above configuration, in the synthesized video obtained by synthesizing the user video and the background video, it is possible to individually adjust the display position, the display size, and the like of each of the user video and the background video. . Then, when the second user's face moves laterally, one of the display position of the user's 3D video and the range included in the synthesized video in the background 3D video is changed to the other shift amount. The synthesized video is transitioned to a state that is shifted in the horizontal direction by an amount larger than that. As a result, on the display after the second user's face has moved sideways, an image reproducing the appearance when the user is actually seen from the position of the face after moving is displayed. Become so. As a result, the realism of the dialogue performed while displaying the user's video on the display is further improved.

In the video display system, the video acquisition unit acquires the video of the user captured by the plurality of imaging devices that capture the video of the user in different imaging directions, for each imaging device, The distance data acquisition unit acquires the distance data regarding the video of the user for each imaging device, and the 3D video generation unit acquires the video of the user acquired for the imaging device and the imaging device. An image piece generating step for generating the user's 3D image piece for each image pickup device based on the distance data, and for generating the 3D image of the user for each image pickup device. A step of combining each of the user's three-dimensional video pieces so that common video regions included in the respective pieces overlap each other, and the video piece generation step When the three-dimensional image piece of the portion including the user's eyes is generated, if the two are different, the virtual image is based on the difference between the two and the distance between the imaging device and the user. It is more preferable that the rendering process for acquiring the 3D video piece when viewed from the viewpoint is executed.
According to the above configuration, when a user's video is captured by a plurality of imaging devices having different imaging directions, a 3D video piece is generated for each imaging device, and finally the 3D video pieces are connected to each other. Get 3D video. On the other hand, among the 3D image pieces generated for each imaging device, when generating a 3D image piece of a part including the user's eyes, it is as seen from a virtual viewpoint at the height of the user's eyes. A rendering process for acquiring a 3D image is executed. As a result, if a 3D video of a user formed by combining 3D video pieces is displayed on the display, the user's line of sight and the line of sight of the person watching the display can be matched.

In the video display system, when the imaging direction is different from the normal direction of the reference plane, the 3D video generation unit generates the video piece based on the video captured in the imaging direction in the video piece generation step. It is more preferable to convert the 3D image piece of the user into the 3D image piece when virtually viewed from the normal direction.
In the above configuration, when a user's video is captured in an imaging direction different from the normal direction of the reference plane, and a 3D video piece is generated from the video, it is generated based on the video captured in the imaging direction. The user's 3D video piece is converted into a 3D video piece viewed virtually from the normal direction. And a user's 3D image | video is acquired using the converted 3D image piece. The three-dimensional image thus obtained is an image when viewed from the normal direction, and is appropriately displayed when displayed on the display. More specifically, it becomes possible to suppress the vicinity of the portion where the 3D video pieces are joined in the 3D video of the user from appearing to be bent.

In addition, according to the video display method of the present invention, the above-described problem is that (A) the computer acquires the video of the user captured by the imaging device, and (B) the computer displays the video for a predetermined number of times. Obtaining distance data indicating a distance from the object in the video piece from the imaging device for each video piece when divided into video pieces; And generating a 3D video of the user by executing a rendering process using the distance data, and (D) a computer detecting the eye height of the user, E) When both the height at which the imaging device is installed and the detected eye height are different, the computer detects based on the difference between the two and the distance between the imaging device and the user. did It is solved by executing the rendering process for acquiring the three-dimensional image of the user when viewed from a virtual viewpoint at the height of the serial eyes.
According to the above method, even if the eye height of the user is different from the installation height of the imaging device, the line of sight of the person watching the display and the line of sight of the person shown on the display (that is, the user) are matched. Is possible. As a result, it is possible to improve the realism of the dialogue performed while displaying the user's video on the display.

According to the video display system and the video display method of the present invention, even if the eye height of the user is different from the installation height of the imaging device, the eye and the display of the person viewing the display (that is, the second user) It is possible to match the line of sight of the person (that is, the user) projected on the screen. In addition, when the face of the second user moved sideways, the image displayed on the display was reproduced from the position of the face after moving, actually facing the user and looking at the user Transition to video is possible. Furthermore, when the distance (depth distance) between the imaging device and the user changes, the display size of the user's 3D image in the composite image displayed on the display is changed to the display size before the change of the depth distance. Adjust so that Thus, it is possible to display the 3D video of the user at a size that is felt when the user actually faces the user and sees the conversation partner (that is, the size of the conversation partner recognized by the user through his / her own vision). It becomes possible.
With the above operation, according to the video display system and the video display method of the present invention, it is possible to improve the realism of the dialogue performed while displaying the user's video on the display.

It is the figure which showed the structure of the video display system which concerns on one Embodiment of this invention. It is the figure which showed the arrangement position of the system component apparatus installed in the room of each user. 3A and 3B are diagrams showing an example of the display of the present invention. It is explanatory drawing about the procedure of an image composition. It is explanatory drawing about the procedure which extracts a person image | video from a real image | video. It is explanatory drawing about the procedure which produces | generates a three-dimensional image | video. It is the figure which showed the structure of the home server which each user holds from a functional surface. It is explanatory drawing about the procedure which matches the height of a eyes | visual_axis about a user's three-dimensional image, (A) is an image | video when it images from an actual camera position, (B) is a position of a camera and a user's eyes | visual_axis. (C) shows the relationship between the images taken from the virtual camera position. It is the figure which showed the structural example of the conventional video display system, and shows a mode that a display video changes in response to the movement of the person who is looking at a display. It is the figure which showed typically the condition where the 2nd user's face moved sideways. It is explanatory drawing about the depth distance of each of a user, a background, and a foreground. It is explanatory drawing which showed the change of the synthetic | combination video when a transition process is performed, (A) has shown the synthetic | combination video before a transition process, (B) has each shown the synthetic | combination video after a transition process. It is the figure which showed the structural example of the conventional video display system, and has shown a mode that the display size of the said user's image | video changes according to a user's depth distance. It is explanatory drawing about adjustment of a video display size, (A) is a synthetic | combination image | video before a user's depth distance changes, (B) is the synthesis | combination of the stage in which size adjustment was performed after the depth distance changed. Each video is shown. It is the figure which showed the flow of the image | video display flow (the 1). It is the figure which showed the flow of the video display flow (the 2). It is the figure which showed the procedure which acquires the three-dimensional image | video of a person. It is the figure which showed typically a mode that a user's image | video is imaged with a some camera. It is the figure which showed the 3D image piece produced | generated for each camera, and the 3D image formed by combining 3D image pieces. It is the figure which showed the procedure which acquires the three-dimensional image | video of a person in a modification. It is explanatory drawing regarding a 2nd transition process, (A) has shown the synthetic | combination image | video before a 2nd transition process, (B) has each shown the synthetic | combination image | video after a 2nd transition process.

Hereinafter, an embodiment of the present invention (hereinafter, this embodiment) will be described with reference to the drawings. The video display system according to the present embodiment (hereinafter, system S) is used for users who are in rooms separated from each other to interact with each other while watching each other's appearance (video). More specifically, a display as a video display is installed in a room where each user is present, and the other party's video is displayed (displayed) on this display. As a result, each user feels that the display is viewed as glass (for example, window glass or door glass) and interacts with the other party through the glass.

The system S is to be used when each user is at his / her home. That is, this system S is used for each user to have a conversation with a conversation partner (a pseudo face-to-face conversation, hereinafter simply referred to as “face-to-face conversation”) while at home. However, the present system S is not limited to this, and the system S is a place where the user is not at home, such as a meeting place, a commercial facility, a school classroom, a school, a public facility such as a hospital, a company, an office, etc. May be used when in Moreover, you may use this system S in order for the person who is in the room apart from each other in the same building to have a face-to-face conversation.

Hereinafter, in order to explain the system S in an easy-to-understand manner, a case where two users have a face-to-face conversation using the system S will be described as an example. One user is Mr. A and the other user is Let's say B. In the following, the configuration of the system S will be described from the viewpoint of Mr. B, that is, from the viewpoint of viewing Mr. A's video. That is, Mr. A corresponds to a “user”, and Mr. B corresponds to a “second user”. However, “user” and “second user” are relative concepts that are switched according to the relationship between the person who sees the image and the person who sees the image. Therefore, when the viewpoint of Mr. A is used as a reference, Mr. B corresponds to “user” and Mr. A corresponds to “second user”.

<< Basic configuration of this system >>
First, the basic configuration of the system S will be described. This system S is used for two users (namely, Mr. A and Mr. B) to conduct a face-to-face conversation while watching each other's images, and more specifically, the life-size of the conversation partner for each user. Is displayed, and the other party's voice is played back. In order to obtain such an audiovisual effect, each user has a communication unit 100. That is, this system S is comprised by the communication unit 100 which each user possesses.

Next, the configuration of the communication unit 100 will be described with reference to FIG. FIG. 1 is a diagram showing the configuration of the system S, more specifically, the configuration of each communication unit 100. Each communication unit 100 includes a home server 1, a camera 2 as an imaging device, a microphone 3 as a sound collection device, an infrared sensor 4, a display 5 as a video display, and a speaker 6 as main components. Among these devices, the camera 2, the microphone 3, the infrared sensor 4, the display 5, and the speaker 6 are arranged in a predetermined room at the home of each user (for example, a room used when performing a face-to-face conversation).

The home server 1 is a central device of the system S, and includes a computer, specifically, a server computer constituting a home gateway. The configuration of the home server 1 is publicly known and includes a CPU, a memory such as a ROM and a RAM, a communication interface, a hard disk drive, and the like.

The home server 1 is installed with a program for executing data processing necessary for realizing the face-to-face conversation (hereinafter referred to as a dialogue program). This interactive program incorporates a program for displaying 3D images. This program is a program for constructing and displaying a 3D image by 3D computer graphics (hereinafter, 3DCG), and is a so-called renderer. The 3DCG renderer has a function of synthesizing a plurality of 3D images. Then, when an image formed by combining a plurality of 3D images, that is, a combined image is displayed on the display 5, the combined 3D images are arranged at different positions in the depth direction of the display 5. It looks like this.

Further, the home server 1 is connected in a communicable state with a communication device via an external communication network GN such as the Internet. That is, the home server 1 belonging to the communication unit 100 owned by Mr. A communicates with the home server 1 belonging to the communication unit 100 owned by Mr. B via the external communication network GN, and transmits and receives various data between the two servers. I do. Note that the data transmitted and received by the home server 1 is data necessary for a face-to-face conversation, for example, video data indicating video of each user and audio data indicating audio.

The camera 2 is a known network camera and captures an image of a subject within an imaging range (angle of view). Here, the “video” is constituted by an aggregate of a plurality of continuous frame images (RGB images), but in the following description, in addition to including an aggregate of frame images, individual frames Including images. In the present embodiment, the imaging range of the camera 2 is fixed. For this reason, the camera 2 always captures an image of a predetermined area of the space in which the camera 2 is installed during its activation.

The camera 2 outputs a signal indicating the captured video (video signal) to the home server 1 belonging to the same unit as the communication unit 100 to which the camera 2 belongs. Although the number of cameras 2 installed is not particularly limited, in the present embodiment, only one camera 2 is provided in each communication unit 100 in consideration of cost.

Further, the lens of the camera 2 faces the display screen forming surface of the display 5. Here, the panel of the display 5 (strictly speaking, it is the touch panel 5a and corresponds to the mirror surface portion) constituting the formation surface is made of transparent glass. Therefore, as shown in FIG. 2, the camera 2 captures an image of a person located in front of the panel through the panel. FIG. 2 is a diagram showing arrangement positions of various devices arranged in the rooms of Mr. A and Mr. B as the components of the system S. Note that the position of the camera 2 may be a position away from the display 5.

Here, when the person who is the subject is away from the display 5 by a predetermined distance at the front position of the display 5, the camera 2 can capture the whole body image from the person's face to the foot. The “whole body image” may be a whole body image in a standing posture or a whole body image in a sitting posture. The “whole body image” includes an image in which a part of the body is hidden by an object placed in front.

In this system S, the camera 2 is installed at a height of about 1 m above the floor. For this reason, when the height (strictly speaking, the height of the eyes) of the person standing in front of the display 5 is higher than the installation position of the camera 2, the camera 2 moves the face of the person who is the subject from below. The image will be taken. Here, the height at which the camera 2 is installed (in other words, the position of the camera 2 in the vertical direction) is not particularly limited, and can be set to an arbitrary height.

The microphone 3 collects sound in the room in which the microphone 3 is installed, and the sound signal is sent to the home server 1 (strictly, the home server 1 belonging to the same unit as the communication unit 100 to which the microphone 3 belongs). Output. In the present embodiment, the microphone 3 is installed at a position directly above the display 5 as shown in FIG.

The infrared sensor 4 is a so-called depth sensor, and is a sensor for measuring the depth of a measurement object (corresponding to the object) by an infrared method. Specifically, the infrared sensor 4 irradiates infrared rays from the light emitting unit 4a toward the measurement object, and measures the depth by receiving the reflected light at the light receiving unit 4b. More specifically, the light emitting unit 4 a and the light receiving unit 4 b of the infrared sensor 4 face the display screen forming surface of the display 5. On the other hand, a film capable of transmitting infrared light is attached to a portion of the touch panel 5a of the display 5 constituting the forming surface at a position immediately before the infrared sensor 4. The infrared light reflected from the measurement object after being irradiated from the light emitting unit 4a passes through the film and is received by the light receiving unit 4b.

In this system S, as the “depth”, the distance from the camera 2 (strictly, the lens surface of the camera 2) to the measurement object, that is, the depth distance is measured. For this reason, in the present system S, the light receiving position by the light receiving unit 4b of the infrared sensor 4 is the same position as the surface position of the lens of the camera 2 in the depth direction of the display 5 (strictly, the normal direction of the display screen). It is set to be.

Further, in the present system S, the depth measurement result is obtained for each pixel when the image captured by the camera 2 is divided into a predetermined number of image pieces (pixels). When the depth measurement results obtained for each pixel are collected in units of video, depth data (corresponding to distance data) for the video can be obtained. This depth data defines the measurement result of the infrared sensor 4, that is, the depth, for each pixel of the captured image (strictly, each frame image) of the camera 2. In other words, the depth data for the video is a depth map of the video, and the depth of the target is included in the pixel group corresponding to the video of the target in the video captured by the camera 2 in the depth data. Distance (depth value) is specified. Specifically, as shown in FIG. 5 to be described later, since the depth distance is different between the background image and the image in front thereof, the corresponding pixels are clearly different as shown in FIG. . In FIG. 5, the black pixels correspond to the background image, the hatched pixels correspond to the image of the object in front of the background, and the white pixels are those of the person in front. Corresponds to video.

By using the depth data as described above, it is possible to extract a person's video from the video. A person video extraction method using depth data will be described later. In the present system S, the position of a person can be specified from the depth data. However, the present invention is not limited to this. For example, a position detection sensor may be installed separately from the infrared sensor 4, and the position of the person may be specified from the detection result of the position detection sensor.

The speaker 6 emits sound (reproduced sound) that is reproduced when the home server 1 develops the sound data, and is configured by a known speaker. In the present embodiment, as shown in FIG. 2, a plurality of speakers 4 (four in FIG. 2) are installed at positions sandwiching the display 5 in the horizontal width direction of the display 5.

The display 5 forms a video display screen. More specifically, the display 5 has a panel made of transparent glass, and forms a display screen on the front surface of the panel. In the present system S, the above-described panel is the touch panel 5a and receives an operation (touch operation) performed by the user.

Furthermore, the above panel has a size sufficient to display a whole body image of a person. In the face-to-face conversation by the system S, the whole body image of the conversation partner is displayed in a life-size size on the display screen formed on the front surface of the panel. That is, it is possible to display Mr. A's whole body image in a life-size size on the display 5 on the Mr. B side. As a result, Mr. B who is looking at the display screen feels as if he is meeting Mr. A, in particular, the feeling of facing through the glass.

Furthermore, the display 5 of the present system S normally functions as furniture arranged in the room, specifically as a look, and forms a display screen only during face-to-face conversation. Hereinafter, the configuration of the display 5 will be described in detail with reference to FIGS. FIGS. 3A and 3B are diagrams showing a configuration example of the display 5 used in the present system S. FIG. 3A shows a non-interactive state, and FIG. 3B shows a face-to-face conversation. Each state is shown.

The touch panel 5a included in the display 5 constitutes a part of the appearance arranged in the room where the face-to-face conversation is performed, specifically, a specular part. Then, as shown in FIG. 3A, the touch panel 5a does not form a display screen during a non-dialogue when no dialogue is performed, that is, while no video is displayed. In other words, the display 5 of the present system S shows an appearance as a figure at the time of non-dialogue. On the other hand, when the face-to-face conversation is started, the touch panel 5a forms a display screen on the front surface. Thereby, as shown in FIG. 3B, the display 5 displays the conversation partner and the video of the background on the front surface of the touch panel 5a.

Incidentally, the home server 1 is to switch the display screen on and off according to the measurement result of the infrared sensor 4. More specifically, when the user stands at the front position of the display 5 in starting the face-to-face conversation, the camera 2 captures an image including the user (hereinafter referred to as an actual image) and the infrared sensor 4 measures the depth. . Thereby, the depth data about the actual video is acquired, and the home server 1 specifies the distance between the user and the camera 2, that is, the depth distance based on the depth data. When the depth distance is equal to or less than the predetermined distance, the home server 1 controls the display 5 to form a display screen on the front surface of the touch panel 5a. As a result, the touch panel 5a of the display 5 that has been functioning as a figure until then functions as a screen for displaying images. On the other hand, when the depth distance is equal to or greater than the predetermined distance, the home server 1 controls the display 5 and turns off the display screen that has been formed so far. As a result, the display 5 functions as a figure again.

As described above, in the present system S, the display 5 is used as an appearance when non-interactive. This makes it difficult to notice the presence of the display screen during non-interaction. On the other hand, during a face-to-face conversation, a display screen is formed and the image of the conversation partner is displayed, so that the user can obtain a visual effect as if he is interacting with the conversation partner through the glass. Become. In addition, about the structure which uses both a video display screen and appearance, a well-known structure can be utilized like the structure described in the international publication 2009/122716, for example. Further, the display 5 is not limited to a configuration that is also used as a figure. The device used as the display 5 only needs to have a size sufficient to display the whole body image of the conversation partner. And from the viewpoint of making it difficult to notice the presence of the display screen during non-interaction, furniture and building materials installed in a room for face-to-face interaction and having a mirror surface portion are suitable. For example, a door (glass door) A window (glass window) may be used as the display 5. In addition, about the display 5, it is not limited to what is used also as furniture, such as building materials, such as a door and a window, or a figure, etc., The normal display which forms a display screen constantly during starting may be used.

<< About video composition >>
In the face-to-face conversation using the system S, Mr. A's video and its background video are displayed on the Mr. B's display 5, and Mr. B's video and his background video are displayed on the Mr. A's display 5. . Here, the person image and the background image displayed on each display 5 are not captured simultaneously by the camera 2 but are captured at different timings. That is, each display 5 displays a composite video obtained by synthesizing a human video and a background video captured at different timings. Further, in the present system S, in addition to the person video and the background video, a synthesized video obtained by further synthesizing the foreground video is displayed.

Hereinafter, the video composition procedure will be outlined with reference to FIG. FIG. 4 is an explanatory diagram of the video composition procedure. In the following description, a case where Mr. A's video, background video, and foreground video are combined will be described as a specific example.

Among the synthesized videos, the background video (indicated by the symbol Pb in FIG. 4) is a video of an area within the imaging range of the camera 2 in the room used when Mr. A performs a face-to-face conversation. . And in this embodiment, when Mr. A is not in the said room, the camera 2 is supposed to image a background image | video. That is, the background video is to be taken alone. In addition, about the imaging | photography timing of a background image | video, it is possible to set arbitrarily if it is in the period when Mr. A is not in the said room.

On the other hand, a person image (specifically, Mr. A's image, represented by the symbol Pu in FIG. 4) is captured when Mr. A is in the room, strictly speaking, within the imaging range of the camera 2. Is done. Here, the video (that is, the real video) captured by the camera 2 includes a background video and a foreground video in addition to a human video. In the system S, a person video is extracted from the actual video and used. A method for extracting a person image from a real image is not particularly limited, and an example is a method for extracting a person image using the above-described depth data. Hereinafter, a person video extraction method using depth data will be described with reference to FIG. FIG. 5 is an explanatory diagram of a procedure for extracting a person video from a real video. In FIG. 5, for the convenience of illustration, the pixels constituting the depth data are coarser than the actual pixels.

During the period when the camera 2 is capturing an image, the infrared sensor 4 measures the depth of the measurement object within the angle of view of the camera 2. As a result, depth data about the actual video is obtained. The depth data for the real video is obtained by defining the measurement result of the infrared sensor 4, that is, the depth for each pixel when the frame image constituting the real video is divided into a predetermined number of pixels. In the depth data for the actual video, as shown in FIG. 5, the pixels belonging to the human video (the white pixels in the figure) and the pixels belonging to the other video (the black pixels and the hatched hatching in the figure). The depth is clearly different from that of (pixel).

Then, Mr. A's skeleton model is identified based on the depth data and the captured image of the camera 2 (strictly, information for identifying the position of the image of Mr. A's face in the captured image). The skeletal model is a simple model of position information about Mr. A's skeleton (specifically, in the body, head, shoulders, elbows, wrists, upper body center, waist, knees, ankles) as shown in FIG. It is what. As a method for acquiring the skeleton model, a known method can be used, for example, the same as the method employed in the invention described in Japanese Patent Application Laid-Open No. 2014-155893 and Japanese Patent Application Laid-Open No. 2013-116311. The method may be used.

Then, after specifying the skeleton model, the person image is extracted from the actual image based on the skeleton model. In the present specification, a description of a technique for extracting a person image from a real image based on a skeleton model will be omitted. However, a rough procedure will be described. In the depth data based on the identified skeleton model, A Identify the pixel group that belongs to the person's video. Thereafter, an area corresponding to the specified pixel group is extracted from the actual video. The image extracted by such a procedure corresponds to the person image of Mr. A in the actual image.

Further, in the present system S, the foreground video (indicated by the symbol Pf in FIG. 4) is extracted from the actual video and used in the same manner as the person video. The method for extracting the foreground video from the actual video is not particularly limited. However, for example, a method for extracting the foreground video using the depth data as in the case of the person video can be considered. More specifically, a pixel group having a depth depth smaller than the pixels belonging to the person video is specified in the depth data of the real video. Then, in the actual video, the video corresponding to the specified pixel group is extracted as the foreground video.

After extracting the person image and the foreground image from the actual image according to the procedure described above, the background image, the person image and the foreground image are synthesized. More specifically, in the background image captured by the camera 2, an image of a portion actually displayed on the display 5 (a range surrounded by a broken line in FIG. 4, hereinafter referred to as a display range) is set. . Here, the display range corresponds to a portion included in the synthesized video in the background video captured by the camera 2. Note that the size of the display range is determined according to the size of the display 5. In the present embodiment, the initial (default) display range is set at the center of the background video. However, the initial display range is not particularly limited, and may be a portion other than the central portion of the background video.

Then, the above-described display range in the background video, the extracted person video, and the extracted foreground video are synthesized to obtain a synthesized video (indicated by symbol Pm in FIG. 4). As a result, as shown in FIG. 4, an image in which Mr. A is positioned in front of the background and the foreground is positioned in front of Mr. A is displayed on the display 5 on the Mr. B side.

As described above, in this system S, the composite video is displayed as the display video on the display 5. In the configuration for displaying the composite video, the display position, the display size, and the like can be individually adjusted for each of the human video, the background video, and the foreground video. Specifically, for example, the display size of the video of Mr. A, which is a person video, can be adjusted without changing the display size of the background video and the foreground video.

In this system S, the display size of Mr. A's video is adjusted to coincide with Mr. A's actual size (life size). As a result, Mr. A's video is displayed in a life-size size on the display 5 on the Mr. B side, and the realism of the face-to-face conversation using the system S is further improved. However, the display size of the person video is not limited to the life size. Here, the life-size size means that the camera 2 is located at a position away from the camera 2 by a predetermined distance (specifically, a distance d1 in FIG. 10B to be described later, hereinafter referred to as a reference distance). It means the size when the captured human image is displayed as it is. The reference distance d1 is set in advance and stored in the memory of the home server 1.

<< About 3D image generation >>
In the system S, a 3D image is to be displayed on the display 5. More specifically, as described in the previous section, the display 5 is configured to display a composite video obtained by combining the background video, the human video, and the foreground video. 3D video (3D video). This 3D image is obtained by executing a rendering process by 3DCG using a 2D image (specifically, an image composed of frame images in RGB format) captured by the camera 2 and depth data about the image. It is obtained by. Here, strictly speaking, the rendering process is a surface rendering video display process, which is a process for generating a 3D video when viewed from a virtually set viewpoint.

And in this system S, the process which adopted the texture mapping as a rendering process is performed. Hereinafter, a procedure for generating a 3D image will be described with reference to FIG. FIG. 6 is an explanatory diagram of a procedure for generating a 3D video. Note that the mesh model in the figure is coarser than the actual mesh size for convenience of illustration. In the following, a case where a 3D video of Mr. A is generated will be described as an example.

The video of Mr. A taken by the camera 2 (strictly speaking, the video of Mr. A extracted from the actual video) is a two-dimensional video and is used as a texture in texture mapping. On the other hand, the depth data (that is, the depth map) acquired for the actual video including the video of Mr. A is used to construct a mesh model that forms the skeleton of the 3D video. Here, the mesh model represents a person (Mr. A) with a polygon mesh. As a method of constructing a mesh model from depth data (depth map), a known method can be used.

Then, after the mesh model is obtained, as shown in FIG. 6, a stereoscopic image of Mr. A is obtained by pasting a two-dimensional image (specifically, an image of Mr. A) as a texture to the mesh model. That is, it becomes possible to generate a 3D image having a sense of depth. A 3D image is generated by such texture mapping, and further, by performing processing such as movement and rotation, it is possible to acquire a 3D image when the viewpoint is changed. As a result, it is also possible to acquire a 3D image when the face of Mr. A is viewed from below and a 3D image when the face of Mr. A is viewed from the side.

Also, for the background and foreground, it is possible to generate a 3D image by the same procedure as for a person. That is, a background three-dimensional image is acquired by executing a rendering process by texture mapping using the background image captured by the camera 2 and the depth data acquired for the background image. In addition, the foreground video captured by the camera 2 (strictly, the foreground video extracted from the real video), the depth data acquired for the foreground video (strictly, the depth data for the real video including the foreground video), A foreground 3D image is acquired by executing a rendering process by texture mapping using.

In this system S, texture mapping is used. However, the rendering process for acquiring a 3D image is not limited to the one using texture mapping. For example, the rendering process uses bump mapping. Also good.

Further, in the depth data, there is a possibility that a defective portion, that is, a pixel for which a depth measurement result cannot be obtained for some reason may occur. In particular, a missing portion is likely to occur near the boundary between the person image and the background image (near the edge). In this way, when a missing part is generated, if the position of the missing part can be specified, a two-dimensional image that is a texture may be pasted as it is on the missing part in texture mapping. Alternatively, the surrounding video may be pasted. In addition, among the pixels constituting the depth data, in the pixel group corresponding to the person video, when a missing portion is generated near the edge, a pixel group that is slightly larger than the above pixel group in texture mapping is selected. What is necessary is just to extract and paste the two-dimensional image | video corresponding to the said pixel group.

<< About home server functions >>
Next, functions of the home server 1, particularly functions related to video display processing will be described. Note that both the Mr. A's home server 1 and Mr. B's home server 1 have the same function, and execute the same data processing through two-way communication when performing the face-to-face conversation. For this reason, only the function of one home server 1 (for example, Mr. B's home server 1) will be described below.

The home server 1 functions as the home server 1 when the CPU of the apparatus executes a dialogue program, and specifically executes a series of data processing related to a face-to-face dialogue. Here, with reference to FIG. 7, the configuration of the home server 1 will be described from the viewpoint of its function, particularly the video display function. FIG. 7 is a diagram showing the configuration of the home server 1 in terms of functions.

As shown in FIG. 7, the home server 1 includes a data transmission unit 11, a data reception unit 12, a background video storage unit 13, a first depth data storage unit 14, a real video storage unit 15, a human video extraction unit 16, a skeleton model. A storage unit 17, a second depth data storage unit 18, a foreground video extraction unit 19, a height detection unit 20, a 3D video generation unit 21, a composite video display unit 22, a determination unit 23, and a face movement detection unit 24 are provided. Each of these data processing units is realized by a hardware device (specifically, CPU, memory, communication interface, hard disk drive, etc.) of the home server 1 cooperating with a dialogue program as software. . Hereinafter, each data processing unit will be described.

The data transmission unit 11 digitizes the video signal captured by the B-side camera 2 and transmits it to the A-side home server 1 as video data. Here, the types of video data transmitted by the data transmission unit 11 are classified into two types. One is the video data of the background video, specifically, the video of the same room captured when Mr. B is not in the room corresponding to the background (strictly, it is within the imaging range of the camera 2). This is data indicating the image of the area. The other is actual video data, which is an image captured while Mr. B is in the room, more specifically, data indicating Mr. B and its background and foreground images. .

Further, when transmitting the video data of the background video, the data transmission unit 11 generates depth data about the background video based on the measurement result of the infrared sensor 4 and transmits the depth data together with the video data of the background video. . This depth data is used when executing a rendering process for acquiring a 3D image of the background, and also when specifying a distance (depth distance) between the background and the camera 2. Similarly, when transmitting the video data of the real video, the data transmission unit 11 generates depth data for the real video based on the measurement result of the infrared sensor 4 and transmits the depth data together with the video data of the real video. To do. This depth data is used when extracting a person image (specifically, an image of Mr. B) and a foreground image from an actual image. The depth data is used at the time of each of the rendering process for acquiring Mr. B's 3D video and the rendering process for acquiring the 3D video of the foreground. Furthermore, the depth data is also used when specifying the distance (depth distance) between Mr. B and the camera 2.

The data receiving unit 12 receives various data transmitted from the home server 1 on the A side. The data received by the data receiving unit 12 includes video data of the background video, depth data about the background video, and video data of the real video and depth data about the real video. Here, the video data of the background video received by the data receiving unit 12 is data indicating the video of the same room captured when Mr. A is not in the room corresponding to the background. In this way, the data receiving unit 12 receives the background video data, and thereby acquires the background video captured by the camera A's camera 2. In this sense, it can be said that the data receiving unit 12 corresponds to a video acquisition unit.

Further, the depth data regarding the background video received by the data receiving unit 12 is used when performing rendering processing for acquiring the background three-dimensional video, and the distance (depth distance) between the background and the camera 2 is used. ) Is also used to specify. Hereinafter, the depth data regarding the background video received by the data receiving unit 12 is referred to as “first depth data”.

The video data of the actual video received by the data receiving unit 12 is data indicating the video of Mr. A, the background, and the foreground captured while Mr. A is in the room. The depth data about the actual video received by the data receiving unit 12 is used when extracting the video of A and the foreground video from the actual video. The depth data is used at the time of each of the rendering process for acquiring Mr. A's 3D video and the rendering process for acquiring the 3D video of the foreground. Further, the depth data is also used when specifying the distance between A and the camera 2 (depth distance) and the distance between the foreground and the camera 2 (depth distance). Hereinafter, the depth data regarding the actual video received by the data receiving unit 12 is referred to as “second depth data”.

As described above, the data receiving unit 12 receives the first depth data and the second depth data from the home server 1 on the A side, so that the depth data for the background video, the depth data for the person video, and the foreground Obtain depth data for each video. In this sense, it can be said that the data receiving unit 12 corresponds to a distance data acquiring unit that acquires depth data that is distance data.

The background video storage unit 13 stores the video data of the background video received by the data receiving unit 12. The first depth data storage unit 14 stores depth data regarding the background video received by the data receiving unit 12, that is, first depth data. The real video storage unit 15 stores the video data of the real video received by the data receiving unit 12.

The person video extracting unit 16 expands the video data of the real video received by the data receiving unit 12, and extracts the human video (that is, the video of Mr. A) from the real video. The skeleton model storage unit 17 stores a skeleton model (specifically, Mr. A's skeleton model) used when the person video extraction unit 16 extracts a person video. The second depth data storage unit 18 stores depth data on the actual video received by the data receiving unit 12, that is, second depth data.

The person video extraction unit 16 reads the real video from the real video storage unit 15 and the second depth data about the real video from the second depth data storage unit 18 in extracting the video of Mr. A from the real video. Then, the person video extraction unit 16 specifies Mr. A's skeleton model from the read second depth data and the captured video of the camera 2. The identified skeleton model of Mr. A is stored in the skeleton model storage unit 17. Thereafter, the person video extraction unit 16 reads out Mr. A's skeleton model from the skeleton model storage unit 17 and extracts a person video, that is, Mr. A's video from the actual video based on the skeleton model. In this way, the person video extraction unit 16 extracts the person video from the actual video, thereby acquiring the video of Mr. A captured by the camera 2 on the A side. In this sense, it can be said that the person video extraction unit 16 corresponds to a video acquisition unit.

The foreground video extracting unit 19 develops the video data of the real video received by the data receiving unit 12 and extracts the foreground video from the real video. More specifically, the foreground video extraction unit 19 extracts the real video from the real video storage unit 15 and the second depth data about the real video from the second depth data storage unit 18 when extracting the foreground video from the real video. Are read out respectively. Then, the foreground video extraction unit 19 extracts a pixel group corresponding to the foreground video from the read second depth data. Here, the pixel group corresponding to the foreground image is a pixel group having a depth distance smaller than the pixel group extracted from the second depth data by the person image extraction unit 16 (that is, the pixel group corresponding to the person image). It is. Thereafter, the foreground video extraction unit 19 extracts a video of a portion corresponding to the pixel group from the real video read from the real video storage unit 15 as a foreground video. In this way, the foreground video extraction unit 19 extracts the foreground video from the actual video, thereby acquiring the foreground video captured by the camera A's camera 2. In this sense, it can be said that the foreground video extraction unit 19 corresponds to a video acquisition unit.

The height detection unit 20 detects the eye height of Mr. A based on the data received from the home server 1 on the Mr. A side. More specifically, the height detection unit 20 reads the second depth data from the second depth data storage unit 18, and extracts a pixel group corresponding to the person video from the read second depth data. Thereafter, the height detection unit 20 specifies a pixel corresponding to the eye from the extracted pixel group, and calculates the eye height from the position of the specified pixel. Then, the detection result regarding the eye height is transferred to the 3D video generation unit 21. The 3D video generation unit 21 generates a 3D video (particularly, a 3D video of a person) according to the detection result. Will come to do. This will be explained in detail in the next section.

It should be noted that the method for specifying the eye height is not particularly limited, and a known method can be used. Specifically, in the present system S, the eye height is detected based on the second depth data. However, the present invention is not limited to this. For example, the system S analyzes the actual video including the video of Mr. May be detected.

The 3D video generation unit 21 executes 3DCG rendering processing to acquire a 3D video. Specifically, the 3D video generation unit 21 uses the background video stored in the background video storage unit 13 and the first depth data regarding the background video stored in the first depth data storage unit 14. The 3D image of the background is generated by executing the rendering process. Note that the 3D video generation unit 21 uses the most recently acquired background video among the background videos stored in the background video storage unit 13 when generating the background 3D video. Similarly, the first depth data acquired most recently is used for the first depth data stored in the first depth data storage unit 14.

In addition, the 3D video generation unit 21 extracts the person video extracted by the person video extraction unit 16 (specifically, the video of Mr. A) and the second depth data (strictly, stored in the second depth data storage unit 18). Performs the rendering process using the pixel data corresponding to the person image in the second depth data to generate a 3D image of the person (Mr. A). Similarly, the 3D video generation unit 21 uses the foreground video extracted by the foreground video extraction unit 19 and the second depth data stored in the second depth data storage unit 18 (strictly speaking, the foreground in the second depth data). A foreground 3D image is generated by executing a rendering process using pixel group data corresponding to the image). Note that, in the present system S, as described above, processing using texture mapping is executed as rendering processing.

The synthesized video display unit 22 synthesizes the 3D videos of the background, the person, and the foreground generated by the 3D video generation unit 21 and displays the synthesized video on the display 5 on the B side. The composite video display unit 22 selects a video to be included in the composite video, that is, a display range, from the background 3D video generated by the 3D video generation unit 21. Then, the composite video display unit 22 displays the composite video in which Mr. A is positioned in front of the selected display range and the foreground is positioned in front of Mr. A on the B-side display 5.

During the period when the composite video display unit 22 displays the composite video on the display 5 (in other words, during the period when the camera 2 on the A side captures the video of Mr. A), It is determined whether the distance between the camera 2 and Mr. A (that is, the depth distance of Mr. A) has changed. Such a determination is made based on the second depth data stored in the second depth data storage unit 18. When the determination unit 23 determines that the depth distance has changed, the determination result is transferred to the composite video display unit 22, and the composite video display unit 22 displays a composite video corresponding to the determination result on the display 5. This will be explained in detail in the next section.

Based on the measurement result of the infrared sensor 4, the face movement detection unit 24 generates depth data about the actual image captured by the Mr. B-side camera 2, and from this depth data, the face movement of Mr. B's face is laterally moved. Detect the presence or absence. More specifically, during the period in which the composite video is displayed on the display 5 by the composite video display unit 22, the face movement detection unit 24 specifies a pixel group corresponding to the video of Mr. B from the depth data, A change in the position of the pixel group is monitored. The face movement detection unit 24 detects that the face of Mr. B has moved sideways when recognizing the change in the position of the pixel group. The lateral movement means that the face of Mr. B moves in the left-right direction (the width direction of the display 5) with respect to the display 5 on the Mr. B side.

About the detection result that Mr. B's face moved sideways, it is handed over to the synthetic | combination video display part 22, and the synthetic | combination video display part 22 displays the synthetic | combination video according to the said detection result on the display 5. FIG. This will be explained in detail in the next section.

<< About the process to improve the realism of face-to-face dialogue >>
In this system S, in order to improve the realism of the face-to-face conversation using the system, the video displayed on the display 5 and the display size thereof are adjusted / changed according to each user's line of sight and the position of the face. Yes. Specifically, the following video display processes (R1) to (R3) are performed.
(R1) Eye height adjustment process (R2) Face movement process (R3) Depth distance change process

Hereinafter, each of the above three video display processes will be described individually. In the following description, a case in which a composite image including a 3D image of Mr. A is displayed on the display 5 on the side of Mr. B will be described as an example.

<About the process for adjusting the eye height>
In the present system S, as described above, the camera 2 is installed at a height of about 1 m from the floor. Therefore, depending on the height of Mr. A, the height of Mr. A's eyes differs from the height at which the camera 2 is installed. In such a case, the image of Mr. A displayed on the display 5 on the Mr. B side is different from the appearance (image) of Mr. A seen when actually facing Mr. A.

More specifically, when the eye height of Mr. A is higher than the installation height of the camera 2, the camera 2 takes an image of the face of Mr. A from below. During this time, since Mr. A is looking at the display 5 on the side of Mr. A in front, Mr. A's eyes are facing the front. Under the above circumstances, as shown in FIG. 8A, Mr. A's video displayed on the Mr. B side display 5 (strictly, it is a three-dimensional video, but is simplified in FIG. 8A). Becomes an image as if looking up at Mr. A's face. FIG. 8 is an explanatory diagram of the process for adjusting the eye height, and (A) in the figure shows an image of Mr. A taken from the actual camera position.

As described above, when an image as if looking up at Mr. A's face is displayed on the display 5, the face of Mr. A in the displayed image has a line of sight toward the front as shown in FIG. It will be projected in a state of facing upwards. In such a case, it is difficult to match Mr. A's line of sight displayed on the display 5 with Mr. B's line of sight looking at the display 5, and there is a possibility that the sense of reality of the face-to-face dialogue will be impaired.

Therefore, in the present system S, when the height of Mr. A's eyes and the installation height of the camera 2 are different, the eyes of Mr. A displayed on the display 5 and the eyes of Mr. B looking at the display 5 are matched. In addition, a process for adjusting the eye height is performed. The process will be described. A 3DCG rendering process is executed to acquire a three-dimensional image of Mr. A viewed from a virtual viewpoint at the same height as the height of Mr. A's eyes. More specifically, when performing the eye height adjustment process, Mr. B's home server 1 (strictly speaking, the height detection unit 20 described above) detects Mr. A's eye height. On the other hand, Mr. B's home server 1 stores information about the height at which Mr. A's camera 2 is installed. When both the height of Mr. A's eyes and the installation height of the camera 2 on the A's side are different, the home server 1 on the B's side (strictly speaking, the 3D video generation unit 21 described above) Then, a rendering process for acquiring a 3D video of Mr. A when viewed from a virtual viewpoint at the detected eye level is executed.

The rendering process will be described with reference to FIG. FIG. 8B is a diagram showing the positional relationship between the camera 2 and the eyes of Mr. A. When the height of the eyes of Mr. A and the installation height of the camera 2 of Mr. A are different, the home server 1 on the side of Mr. B has a difference between the two (the symbol H in FIG. 8B). Specified). Further, the home server 1 on the B side is based on the stored second depth data, and the distance between the A and the A camera 2 (that is, the depth distance of the A, FIG. (B) is indicated by the symbol L). Then, Mr. B's home server 1 determines the imaging direction of a virtual camera (shown by a broken line in FIG. 8B) installed at the same height as the detected eye of Mr. A. The difference between the actual imaging direction of the camera 2 is specified. Specifically, the angle α obtained by the following equation (1) is calculated as the difference.
α = arctan (H / L) (1)

Then, Mr. B's home server 1 executes a rendering process for acquiring Mr. A's video (three-dimensional video) captured from the virtual camera using the calculation result of the angle α. Specifically, the texture mapping using the video of Mr. A taken by the camera 2 (strictly speaking, the video of Mr. A extracted from the real video) and the second depth data which is depth data about the real video. Furthermore, video processing is performed for displacing the viewpoint by a height corresponding to the calculated angle α. As a result, Mr. A's 3D image captured from a virtual camera, in other words, Mr. A's 3D image with his eyes facing the front, as shown in FIG. 8C, is acquired. . FIG. 8C shows a video of Mr. A taken from a virtual camera position.

Thereafter, Mr. B's home server 1 (strictly speaking, the above-described synthesized video display unit 22) synthesizes the 3D video of Mr. A acquired by the above procedure and the 3D video of the background and foreground. Then, the synthesized video is displayed on the display 5 on the B side.

<About the process of moving the face>
When Mr. B's face moves sideways in a scene where Mr. A and Mr. B are actually facing each other, Mr. B's field of view (image reflected in Mr. B's eyes) changes as the face moves. In order to reproduce such a change in appearance caused by the movement of the face by the video display system, it is necessary to change the video displayed on the display 5 in conjunction with the movement of the face of the person who is viewing the display 5. . For this reason, in the conventional video display system, as shown in FIG. 9, when the face of a person (for example, Mr. B) who is watching the display 5 moves sideways, the video displayed on the display 5 is centered on the vertical axis. It was supposed to switch to rotate. Specifically, as shown in the figure, as the display image, an image in which the depth distance is different between the left part and the right part is displayed on the display 5. FIG. 9 is a diagram illustrating a configuration example of a conventional video display system, and illustrates how the display video changes in conjunction with the movement of Mr. B who is looking at the display 5.

However, when Mr. B's face moves sideways in the scene where Mr. A and Mr. B are actually facing each other, Mr. A's appearance of Mr. B is rotated as described above. There is nothing but horizontal movement. In the video display system shown in FIG. 9, when Mr. B's face moves sideways, both Mr. A's image and the background image are rotated by the same rotation amount (rotation angle). For this reason, in the video display system illustrated in FIG. 9, when the face of Mr. B moves sideways, the picture of Mr. A displayed on the display 5 is different from the way it looks when actually facing each other. End up.

On the other hand, in this system S, when Mr. B's face moves sideways, the process at the time of the face movement is performed, and the appearance when actually looking at Mr. A while facing Mr. A is accurate. As a result, the video (synthesized video) displayed on the display 5 is changed. Hereafter, the process at the time of a face movement is demonstrated, referring FIG. 10A, FIG. 10B, and FIG. FIG. 10A is a diagram schematically illustrating a situation in which Mr. B's face has moved laterally. FIG. 10B is an explanatory diagram regarding the depth distance of each of Mr. A, the background, and the foreground. FIG. 11 is an explanatory diagram showing changes in the composite video when a transition process described later is executed. (A) shows the composite video before the transition process, and (B) shows the composite video after the transition process. , Respectively.

In the following, the case where Mr. B who was standing at the approximate center of the display 5 has moved laterally will be described as an example. In the following description, one of the two directions opposite to each other in the width direction (that is, the left-right direction) of the display 5 is referred to as “first direction”, and the other is referred to as “second direction”. Here, the relationship between the first direction and the second direction is relative, and when one direction in the left-right direction is the first direction, the other direction is the second direction. Therefore, when the left direction is the first direction when the display 5 is viewed from the front, the right direction is the second direction. Conversely, when the right direction is the first direction, the left direction is the second direction.

Mr. B's home server 1 (strictly speaking, the aforementioned face movement detection unit 24) detects the presence or absence of movement of Mr. B's face while displaying the composite video on the display 5 of Mr. B. When the lateral movement of Mr. B's face is detected, Mr. B's home server 1 simultaneously detects the direction of movement and the amount of movement. Furthermore, Mr. B's home server 1 (strictly speaking, the above-described composite video display unit 22) executes a transition process according to the detection result related to Mr. B's face movement. The transition process is a process for transitioning the composite video displayed on the display 5 on the Mr. B side from the state before detecting the lateral movement of the Mr. B face. Specifically, both the display position of Mr. A's 3D video and foreground 3D video in the composite video, and the range (ie, display range) included in the composite video in the background 3D video are displayed. The composite video is shifted to a state shifted in the horizontal direction.

The transition process will be described in detail. In this process, first, a shift amount is set for each of the display position of the 3D video of Mr. A and the foreground 3D video in the synthesized video, and the display range of the background 3D video. . Here, assuming that Mr. B's face has moved in the first direction by the amount of movement x, the amount of each shift is the amount of movement x of Mr. B, the camera 2 and its subject (Mr. A and its background). And the foreground) (ie, the depth distance). In this system S, when setting the shift amount, the movement amount x of Mr. B's face is converted into a movement angle. The movement angle is an angle indicating the amount of change in the line of sight of Mr. B. The line of sight is a virtual straight line from the center position of Mr. B's eyes toward the center of the display 5.

Referring to FIG. 10A, the line illustrated by the one-dot chain line corresponds to the line of sight before the face of Mr. B moves, and the line illustrated by the two-dot chain line corresponds to the line of sight after movement. To do. The acute angle formed by the two line-of-sight lines, that is, the angle θ in FIG. 10A corresponds to the movement angle. It is assumed that the line of sight before Mr. B's face moves is a line along the normal direction of the display screen of the display 5 as shown in FIG. 10A.

Also, when setting the amount of displacement, the depth distance of each of Mr. A, the background (for example, the wall), and the foreground (for example, the box in front of Mr. A) is specified. Here, during the face-to-face conversation, the depth distance of Mr. A is maintained at a position separated from the camera 2 on the Mr. A side by the reference distance d1, as shown in FIG. 10B. On the other hand, as shown in FIG. 10B, the depth distance of the wall of the room as the background is separated from the camera 2 on the A side by a distance dw. Naturally, this distance dw is longer than the reference distance d1 which is the depth distance of Mr. A. Further, the depth distance of the box placed in front of Mr. A, which is the foreground, is separated from the camera 2 on the Mr. A side by a distance df, as shown in FIG. 10B. Naturally, this distance df is shorter than the reference distance d1 which is the depth distance of Mr. A.

Then, after the movement angle θ and the depth distances d1, dw, df of each of Mr. A, the background, and the foreground are specified, the display position of Mr. A's 3D image in the composite image, the display position of the foreground 3D image And a deviation amount are set for each of the display ranges of the background 3D video. More specifically, assuming that the shift amount with respect to the display position of Mr. A's 3D image is t1, the shift amount t1 is calculated by the following equation (2).
t1 = d1 × sin θ (2)

Also, assuming that the amount of deviation from the background 3D video display range is t2, the amount of deviation t2 is calculated by the following equation (3).
t2 = dw × sin θ (3)

Also, assuming that the amount of deviation with respect to the display position of the foreground 3D image is t3, the amount of deviation t3 is calculated by the following equation (4).
t3 = df × sin θ (4)

After setting the above-described shift amounts t1, t2, and t3, the display position of Mr. A's 3D image is shifted by the shift amount t1, the display range of the background 3D image is shifted by the shift amount t2, and the 3D video of the foreground is displayed. The composite image is shifted to a state where the display position is shifted in the second direction by the shift amount t3. Thus, when the synthesized video shown in FIG. 11A was initially displayed on the display 5 on the B-san side, the synthesized video is linked to the lateral movement of Mr. B's face. The state gradually changes to the state shown in FIG.

As described above, in the present system S, when Mr. B's face moves in the first direction during the period in which the synthesized video is displayed on the Mr. B display 5, the three-dimensional image of Mr. A in the synthesized video is displayed. The display position of the video, the display position of the foreground 3D video, and the display range of the background 3D video are all shifted in the second direction. Further, the shift amount t2 with respect to the display range of the background 3D video is larger than the shift amount t1 with respect to the display position of the 3D video of Mr. A. Further, the shift amount t3 with respect to the display position of the 3D video of the foreground is smaller than the shift amount t1 with respect to the display position of the 3D video of Mr. A. In this way, by transitioning the composite video to a state in which the display position of Mr. A's 3D video, the display position of the 3D video in the foreground, and the display range of the background 3D video are shifted from each other by different shift amounts. On the display 5 on the side of Mr. B, an image reflecting the appearance when Mr. A actually sees Mr. A from the position of the face after the movement is displayed.

To explain in an easy-to-understand manner, if Mr. B actually interacts with Mr. A, when Mr. B's face moves sideways, what is visible from Mr. B's position after the movement is shifted from the original position. Looks like it's in position. Here, the closer to Mr. B, the smaller the amount of deviation appears from the original position, and the farther away, the larger the amount of deviation appears from the original position. It becomes like this. In this system S, when it is detected that Mr. B's face has moved sideways in order to reproduce the above-described appearance, the display position of Mr. A's 3D image, the display position of the foreground 3D image, and The composite video is shifted so that the display range of the background video is shifted by different shift amounts. At this time, the displacement amount t2 with respect to the display range of the background 3D image is larger than the displacement amount t1 with respect to the display position of Mr. A's 3D image. As a result, in the composite video after the transition process, it is possible to see the video in the range that was not displayed in the original composite video (composite video before Mr. B's face moved) in the background, so-called peeking is possible It becomes.

<About the process when the depth distance changes>
When performing the face-to-face conversation, Mr. A usually stands at a position separated from the camera 2 on the Mr. A side by the reference distance d1. At this time, when Mr. A's image captured by the camera 2 is displayed on the display 5, the image is displayed in a life-size size as shown in FIG. On the other hand, when Mr. A moves rearward from the above position, when the captured image of the camera 2 is displayed on the display 5 in the same size, the image is smaller than the life size as shown in FIG. A small size is displayed. Such a change in display size is unavoidably caused by the optical characteristics of the lens of the camera 2. FIG. 12 is a diagram showing a configuration example of a conventional video display system, illustrating that the display size of the video of Mr. A displayed on the display 5 becomes smaller as the depth distance of Mr. A increases. Yes.

However, even when Mr. A and Mr. A are actually facing each other, even if Mr. A is slightly closer to or away from Mr. B, the appearance (size) of Mr. A is as seen from Mr. B. The appearance (appearance) looks almost unchanged. Therefore, in the present system S, a process at the time of changing the depth distance is performed in order to reproduce the actual appearance when the depth distance of Mr. A changes. As a result, the display size of Mr. A's video (strictly, a three-dimensional image) displayed on Mr. B's display 5 is maintained at a life-size size even after Mr. A's depth distance has changed. become.

Hereinafter, the process when the depth distance is changed will be described. In the following description, it is assumed that the depth distance of Mr. A has changed from the reference distance d1 to a distance d2 that is larger than the reference distance d1. The process at the time of changing the depth distance is the depth of Mr. A during the period in which the composite image is displayed on the Mr. B side display 5 (in other words, the period in which Mr. A's camera 2 captures the image of Mr. A). This is done when the distance changes. Specifically, Mr. B's home server 1 (strictly, the determination unit 23 described above) determines whether there is a change in the depth distance during the period. When it is determined that the depth distance has changed, Mr. B's home server 1 uses this as a trigger to start the process when the depth distance changes.

In the process of changing the depth distance, Mr. B's home server 1 (strictly speaking, the composite video display unit 22) executes an adjustment process for adjusting the display size of Mr. A's 3D video in the composite video. In the adjustment process, first, the depth distance d2 after the change is specified. Thereafter, based on the specified depth distance d2 after the change, the display size before the position of Mr. A in the depth direction is changed, that is, the display size of the video of Mr. A is adjusted to be a life size. Specifically, when the depth distance of Mr. A changes from d1 to d2, in the adjustment process, the display distance of the Mr. A's video (strictly speaking, the vertical size and horizontal size of the video) is changed to the depth distance. The display size is corrected by multiplying by the ratio (d1 / d2).

Thereafter, Mr. B's home server 1 synthesizes the 3D video of Mr. A whose size has been corrected, and the 3D video of the background and foreground, and displays the synthesized video on the display 5. Accordingly, as shown in FIGS. 13A and 13B, even if the depth distance of A changes, the video of Mr. A is displayed at the display size before the depth distance changes. Become. When the depth distance of Mr. A changes in this way, the display size of Mr. A's 3D image is adjusted to reflect the appearance when actually facing Mr. A. As a result, this system S is used. The realism of the face-to-face dialogue will be further improved. FIG. 13 is an explanatory diagram of the execution result of the adjustment process. FIG. 13A shows a composite image before the depth distance changes, and FIG. 13B shows a result after the depth distance changes. The composite images that have undergone the adjustment process are respectively shown. Further, in FIG. 13B, for comparison of the display size, the video of Mr. A after the depth distance has changed and before the adjustment process is performed is indicated by a broken line.

<< About the video display flow >>
Next, in the face-to-face conversation using the system S, a flow of a series of data processing related to video display, that is, a video display flow will be described. Here, in the video display follow described below, the video display method of the present invention is applied. That is, in the following, the flow of a video display flow to which the video display method is applied will be described as an explanation regarding the video display method of the present invention. In other words, each step in the video display flow described below corresponds to a component of the video display method of the present invention.

In the following, a case where a composite image including a 3D image of Mr. A is displayed on the display 5 on the side of Mr. B will be described as an example. Incidentally, the procedure for displaying the synthesized video including the 3D video of Mr. B on the display 5 on the side of Mr. A is substantially the same as the following procedure.

The video display flow proceeds when the home server 1 on the side of Mr. B, which is a computer, performs the steps shown in FIGS. 14 and 15 are diagrams showing the flow of the video display flow. Specifically, first, the Mr. B's home server 1 communicates with the Mr. A's home server 1 to receive the video data of the background video and the depth data (first depth data) about the background video. (S001). Thereby, Mr. B's home server 1 acquires a video of a room used when Mr. A performs a face-to-face conversation as a background video. At the same time, Mr. B's home server 1 acquires first depth data as data (distance data) indicating the distance between the background and the camera 2. Note that this step S001 is performed while the A-side camera 2 captures only the background video, that is, during a period when there is no A in the room where the face-to-face conversation is performed. Further, the acquired background video and first depth data are stored in the hard disk drive or the like of the home server 1 on the side of Mr. B.

Then, Mr. B's home server 1 reads the most recently acquired background video and first depth data out of the stored background video and first depth data, and performs processing by texture mapping as rendering processing using them. Execute. Thereby, Mr. B's home server 1 acquires a background three-dimensional image (S002).

On the other hand, when Mr. A enters the room for face-to-face conversation and starts the face-to-face conversation, the camera 2 installed in the room picks up an image including Mr. A, the background and the foreground, that is, a real image. Then, Mr. A's home server 1 transmits the video data of the actual video captured by the camera 2, and Mr. B's home server 1 receives the video data. Thereby, Mr. B's home server 1 acquires the above-mentioned actual video. In addition, the home server 1 on the Mr. A side transmits depth data (second depth data) on the real video simultaneously with the transmission of the video data of the real video, and the home server 1 on the B side transmits the second depth data. Receive. Thereby, Mr. B's home server 1 acquires the second depth data in a state of being set with the actual video (S003). The acquired actual video and second depth data are stored in the hard disk drive or the like of Mr. B's home server 1.

Thereafter, Mr. B's home server 1 extracts a person image, specifically, Mr. A's image from the acquired actual image (S004). Specifically, Mr. B's home server 1 specifies Mr. A's skeleton model based on the second depth data acquired in the previous step S002 and the captured video of the camera 2, and then A's video is extracted from the actual video based on the model.

Then, Mr. B's home server 1 executes rendering processing using the video of Mr. A extracted in the previous step S004 and the second depth data, and specifically executes processing by texture mapping. Thereby, Mr. B's home server 1 acquires a 3D image of the person (Mr. A) (S005).

Further, Mr. B's home server 1 extracts the foreground video from the actual video acquired in step S003 based on the second depth data (S006). Thereafter, Mr. B's home server 1 executes a rendering process by texture mapping using the extracted foreground video and the second depth data. Thereby, Mr. B's home server 1 acquires a 3D image of the foreground (S007).

After acquiring the 3D images of Mr. A and the foreground, the home server 1 on the side of Mr. B, the images within the predetermined range in these 3D images and the background 3D images acquired in step S002. (Display range) is synthesized (S008). Then, Mr. B's home server 1 displays the synthesized video on the Mr. B's display 5 (S009). Accordingly, Mr. A's 3D image is displayed in a life-size size in front of the background 3D image on the display 5 on the side of Mr. B, and in front of the 3D image of Mr. A. A foreground 3D image is displayed.

Here, among the series of steps related to the video display flow described above, step S005 for acquiring a 3D video of a person will be described in more detail with reference to FIG. FIG. 16 is a diagram illustrating a procedure for acquiring a 3D video of a person. In step S005, first, texture mapping is performed using the video of Mr. A extracted in the previous step S004 and the second depth data (S011). As a result, an image viewed from the position where the camera 2 on the A's side is installed is acquired as a 3D image of the A's.

Next, based on the first depth data stored in Mr. B's home server 1, the height of Mr. A's eyes is detected (S012). Thereafter, the home server 1 on the B side compares the detected height of the eyes of the A with the set height of the camera 2 on the A side (S013). If the heights of the two are different, the home server 1 on the side of Mr. B performs a process for adjusting the eye height (S014). In this process, Mr. B's home server 1 performs a rendering process for acquiring a three-dimensional image of Mr. A viewed from a virtual viewpoint at the same height as the detected eye height of Mr. A. Strictly speaking, the 3D image acquired in step S011 is subjected to image processing for displacing the viewpoint by a height corresponding to the angle α calculated by the above-described equation (1). As a result, it is possible to acquire the 3D video of Mr. A viewed from the above-described virtual viewpoint, that is, the 3D video of Mr. A with the line of sight facing the front (S015).

On the other hand, when the detected eye height of Mr. A coincides with the set height of the camera 2 on the A's side, the home server 1 on the B's side uses the 3D video acquired in step S011 as it is. Used in subsequent steps in state.

By the way, in the video display flow, Mr. B's home server 1 acquires the actual image (Mr. B, the background and the foreground image) captured by Mr. B's camera 2 and also displays the measurement result from the infrared sensor 4. Based on this, the depth data (second depth data) of the actual video is acquired. Based on such depth data, Mr. B's home server 1 determines whether Mr. B's face has moved laterally during the period in which the composite video is displayed on the Mr. B display 5 (S021). . When it is determined that the face of Mr. B has moved sideways, the home server 1 on the side of Mr. B specifies the direction and amount of movement of the face based on the depth data before the movement and the depth data after the movement. (S022).

Furthermore, Mr. B's home server 1 specifies the depth distance of each of Mr. A, the background, and the foreground based on the second depth data acquired in step S003 (S023). Thereafter, the home server 1 on the B-side side calculates a deviation amount used in the transition process executed in the next step S025 based on the values specified in steps S022 and S023 (S024). More specifically, in step S024, the shift amount t1 with respect to the display position of Mr. A's 3D video in the composite video, and the range (display range) included in the composite video in the background 3D video. The shift amount t2 and the shift amount t3 with respect to the display position of the foreground 3D video in the composite video are respectively calculated according to the equations (2) to (4) described above.

Then, Mr. B's home server 1 executes the transition process after calculating the deviation amount (S025). By executing this transition process, the composite image displayed on the display 5 transitions from a state before detecting the lateral movement of Mr. B's face. Specifically, when it is detected that Mr. B's face has moved sideways in the first direction, Mr. B's home server 1 displays the display position of Mr. A's 3D image in the composite image in the transition process, The composite video is shifted to a state in which the display position of the foreground 3D video and the display range of the background 3D video are shifted in the second direction by the shift amount calculated in the previous step S024. At this time, the shift amount with respect to the display range of the background 3D video is larger than the shift amount with respect to the display position of the 3D video of Mr. A. Further, the shift amount of the foreground 3D video display position is smaller than the shift amount of Mr. A with respect to the 3D video display position.

When the transition processing is completed, Mr. B's home server 1 displays the composite video after the transition processing, that is, the display position of Mr. A's 3D video, the display position of the foreground 3D video, and the background 3D video. The composite image with the display range shifted from the initial state is displayed on the display 5 (S026). As a result, the display 5 displays an image that reproduces the appearance when seen from the position of the face of Mr. B after the lateral movement. As described above, in the synthesized video after the transition process, the shift amount with respect to the display range of the background 3D video is larger than the shift amount of Mr. A with respect to the display position of the 3D video. For this reason, Mr. B can peek at the image that was not initially displayed on the display 5 among the three-dimensional images of the background by moving his / her face to the left and right.

In addition, the home server 1 on the B side has changed the depth distance of the A during the period in which the composite video is displayed on the display 5 on the B side based on the second depth data acquired in step S003. It is determined whether or not (S027). When it is determined that the depth distance of Mr. A has changed, the home server 1 on the side of Mr. B identifies the depth distance after the change based on the second depth data after the change (S028). Thereafter, the home server 1 on the B side adjusts the display size of the 3D image of the A in accordance with the identified depth distance after the change (S029). At this time, Mr. B's home server 1 adjusts the display size so that the three-dimensional image of Mr. A after the depth distance change is displayed in the display size before the depth distance change, that is, the life-size size. After the adjustment of the display size is completed, Mr. B's home server 1 synthesizes the 3D video of Mr. A after the size adjustment and the 3D video of the background and foreground, and displays the synthesized video on the display 5. (S030). Thereby, even after Mr. A's depth distance changes, Mr. A's three-dimensional image displayed on the display 5 continues to be displayed in a life-size size.

<< Variation of video display system >>
In the configuration of the system S described above, one camera 2 that captures each user's video is provided. That is, in the above embodiment, a single camera 2 captures a user's video, and the display 5 displays a 3D video based on the video captured by the single camera 2. did. On the other hand, if a user's image | video is imaged with the some camera 2 from which an imaging direction mutually differs, it will become possible to acquire a user's image | video from more viewpoints. As a result, for the user's 3D image generated by the rendering process using the captured image of the camera 2, the blind spot area that cannot be visually recognized only by the single camera 2 is reduced, and the 3D image is viewed. The degree of freedom with respect to the setting position of the viewpoint (virtual viewpoint) is also increased.

Hereinafter, a configuration (hereinafter, modified example) in which a plurality of cameras 2 capture a user's video will be described. In the following description, description of the same configuration as that described above will be omitted, and only a different configuration will be described. In the following, a case where the image of Mr. A is captured by the upper and lower two cameras 2 will be described as an example. In addition, about the number of cameras 2, an installation location, and each imaging direction, it is not limited to the content demonstrated below, It is possible to set arbitrarily.

In the modified example, as shown in FIG. 17, the image of Mr. A is picked up by the upper and lower two cameras 2. FIG. 17 is a diagram schematically illustrating a state in which the image of Mr. A is captured by the two upper and lower cameras 2. Further, the upper and lower two cameras 2 respectively capture the image of Mr. A at different positions. Specifically, the upper camera 2 is installed at a position somewhat higher than Mr. A's height, and the lower camera 2 is installed slightly above the floor surface.

In the modification, the video display screen of the display 5 (strictly, the front surface of the touch panel 5a) is used as a reference plane, and the imaging directions of the two upper and lower cameras 2 are perpendicular to the normal direction of the reference plane. Tilt in the direction. The imaging direction is the optical axis direction of the lens of the camera 2, and the imaging direction of the upper camera 2 is set to a direction that descends as approaching Mr. A. That is, the upper camera 2 images Mr. A's body from above. On the other hand, the imaging direction of the lower camera 2 is set to a direction that rises as it approaches Mr. A. That is, the lower camera 2 images Mr. A's body from below.

Also, in the face-to-face conversation according to the modification, Mr. A stands at a position separated from the reference position by the reference distance d1. When Mr. A stands at such a position, the upper camera 2 captures an image from the head of A to the waist (hereinafter referred to as an upper body image), and the lower camera 2 captures the abdomen from the foot of Mr. A. The previous video (hereinafter, lower body video) is captured. Furthermore, in the modification, an infrared sensor 4 is provided for each camera 2. Thereby, it is possible to individually acquire the depth data (strictly, the second depth data) for the video (actual video) captured by each of the upper and lower cameras 2.

On the other hand, in the modified example, Mr. B's home server 1 acquires Mr. A's video for each camera 2. More specifically, the home server 1 on the side of Mr. A has the video data of the real video including the upper body video captured by the upper camera 2 and the video data of the real video including the lower body video captured by the lower camera 2. And send. Mr. B's home server 1 acquires these video data, and extracts the video of Mr. A, specifically the upper body video and the lower body video, from the actual video indicated by each video data.

Further, in the modification, the Mr. B side home server 1 receives the depth data about the actual image captured by each camera 2 from the Mr. A side home server 1 for each camera. That is, in the modified example, the home server 1 on the side of Mr. B acquires the depth data for the real video including the upper body video and the lower body video of Mr. A for each camera. Further, in the modified example, Mr. B's home server 1 (strictly, the 3D video generation unit 21) generates a 3D video piece for each camera based on the actual video and depth data acquired for each camera. A process, that is, a video piece generation process is performed.

More specifically, in the video piece generation process, the home server 1 on the B side uses the upper body video of Mr. A obtained from the actual video captured by the upper camera 2 and the actual video captured by the upper camera 2. The depth data is used to perform rendering processing. As a result, a 3D image piece viewed from the imaging direction of the upper camera 2, specifically, a 3D image piece of Mr. A's upper body shown in FIG. 18 is acquired. Similarly, in the video piece generation process, the home server 1 on the B side has a lower body video obtained from the real video captured by the lower camera 2 and an actual video captured by the lower camera 2. A rendering process using depth data is performed. As a result, a 3D image piece viewed from the imaging direction of the lower camera 2, specifically, a 3D image piece of Mr. A's lower body shown in FIG. 18 is acquired. FIG. 18 is a diagram illustrating a 3D image piece generated for each camera and a 3D image of Mr. A generated in a combining step described later.

In addition, in the modification, the installation height of the camera 2 (that is, the upper camera 2) that captures the image of the part including the eyes of Mr. A is different from the height of the eyes of Mr. A. For this reason, in the modified example, during the above-described video piece generation process, when generating the 3D video piece of the portion including Mr. A's eyes (specifically, the 3D video piece of the upper body), A process for leveling is to be performed. In other words, in the modified example, a rendering process for acquiring a 3D image piece of the upper body when viewed from a virtual viewpoint at the height of Mr. A's eyes is executed. Hereinafter, a procedure for acquiring the 3D video of Mr. A in the modification will be described with reference to FIG. FIG. 19 is a diagram showing a procedure for acquiring the 3D video of Mr. A in the modification.

In the video display flow according to the modified example, Mr. B's home server 1 first performs a video piece generation process (S041) in generating a 3D video of Mr. A. In the video piece generation step, Mr. B's home server 1 generates a 3D video piece for each of the upper body and the lower body of Mr. A by executing a rendering process using texture mapping (S042, S043). Specifically, Mr. B's home server 1 generates a 3D image piece of the upper body viewed from the upper camera 2 when generating the 3D image piece of the upper body during the image piece generating process. After that, Mr. B's home server 1 specifies the difference between the installation height of the upper camera 2 and the eyes of Mr. A, and determines the distance (depth distance) between Mr. A and the upper camera 2. Identify. Furthermore, Mr. B's home server 1 obtains the rotation angle α used in the subsequent video processing based on these identification results. Then, Mr. B's home server 1 performs video processing for displacing the viewpoint by a height corresponding to the rotation angle α on the 3D video piece of the upper body generated in the previous step. As a result, a 3D image piece when viewed from a virtual viewpoint at the height of A's eyes is acquired as a 3D image piece of Mr. A's upper body. That is, a 3D image piece of the upper body of Mr. A with the line of sight facing the front is acquired.

In addition, the home server 1 on the B-side rotates the image with respect to the 3D image piece obtained from the actual image captured by the lower camera 2 when generating the 3D image piece of the lower body during the image piece generating process. Execute the process. More specifically, the lower camera 2 captures an image of the lower body of Mr. A from an imaging direction different from the normal direction of the display screen of the display 5 that is the reference plane. Then, Mr. B's home server 1 uses the texture using the actual video captured by the lower camera 2 (that is, the video captured in the above imaging direction) and the depth data of the actual video. Mapping is performed to generate a 3D image piece of Mr. A's lower body. The 3D image piece generated at this stage is a 3D image piece when viewed from the imaging direction of the lower camera 2.

On the other hand, Mr. B's home server 1 executes a video rotation process on the 3D video fragment when viewed from the imaging direction of the lower camera 2. In this video rotation process, a three-dimensional image fragment when the three-dimensional image fragment when viewed from the imaging direction of the lower camera 2 is virtually viewed from the normal direction of the display screen of the display 5 serving as a reference plane. It is a process for making it convert into. Specifically, the inclination degree of the imaging direction of the lower camera 2 with respect to the normal direction is specified by an angle (inclination angle), and the 3D image piece is rotated by the inclination angle. As a result, the 3D image piece when viewed from the normal direction of the reference plane is acquired as the 3D image piece of the lower half of Mr. A. Note that the above video rotation processing is realized by known video processing.

After acquiring the 3D image pieces of the upper and lower bodies, the Mr. B's home server 1 performs a combining step of combining the 3D image pieces to generate the 3D image of Mr. A (S044). ). In this joining step, the 3D image pieces of the upper body and the lower body are overlapped with a common image area (specifically, an area showing the image of Mr. A's abdomen) included in each 3D image piece. To join. When combining the video pieces, the upper part of the three-dimensional video piece is cut off the video below the abdomen, and the lower half of the three-dimensional video piece is cut off the video above the abdomen.

Then, when the joining process is completed, the 3D image of Mr. A is completed (S045). Such a 3D image is a 3D image when A is viewed from the front (in other words, the normal direction of the reference plane) as shown in FIG. Thereafter, Mr. B's home server 1 (strictly speaking, the synthesized video display unit 22) synthesizes the 3D video of Mr. A obtained by the above procedure and the 3D video of the background and the foreground. Then, the synthesized video is displayed on the display 5. At this time, in Mr. A's 3D image, an image in the vicinity of the joined portion of the 3D image pieces (specifically, near the abdomen) is displayed without a sense of incongruity.

To explain in an easy-to-understand manner, a three-dimensional image piece of the upper body obtained by using the actual image captured by the upper camera 2 and its depth data as it is, and an actual image captured by the lower camera 2 and its depth data as it is. Assume that the acquired 3D image piece of the lower body is simply combined. When the 3D image of Mr. A obtained in this case is displayed on the display 5, it appears as if the vicinity of the portion where the 3D image pieces are joined is bent in the 3D image (that is, upright). It looks as if it is slightly bent forward with respect to the posture). On the other hand, in the present modification, a process for adjusting the eye height is performed when generating a 3D image piece of the upper body. In addition, when generating a 3D image piece of the lower body, the depth data is converted into data about an image viewed from the normal direction of the reference plane, and a 3D image piece is generated based on the converted depth data. . As a result, for Mr. A's 3D video acquired by joining the 3D video pieces, it is possible to suppress a sense of incongruity that the vicinity of the joined portion of the 3D video pieces appears to be bent. .

In this modification, a plurality of cameras 2 (specifically, two cameras 2) are arranged side by side, but the present invention is not limited to this. For example, two cameras 2 may be arranged side by side. In such a case, a 3D image piece (specifically, a 3D image piece for each of the left and right bodies) is generated in the same procedure as described above, and the 3D image pieces are combined with each other. 3D images will be generated.

<< Other Embodiments >>
In the above embodiment, the video display system and the video display method of the present invention have been described with specific examples. However, said embodiment is only an example for making an understanding of this invention easy, and does not limit this invention. That is, the present invention can be changed and improved without departing from the gist thereof, and the present invention includes its equivalents.

In the above embodiment, the case where two users (Mr. A and Mr. B) have a face-to-face conversation through the system S has been described as an example. However, the present invention is not limited to this, and the face-to-face conversation is performed simultaneously. The number of people who can do this may be three or more.

In the above embodiment, a series of steps relating to video display, strictly speaking, a step of generating a 3D video for each of the user (for example, Mr. A) and the background and foreground and synthesizing the 3D video. The home server 1 on the side of the second user (for example, Mr. B) is supposed to be implemented. However, the present invention is not limited to this, and the series of steps described above may be performed by the home server 1 on the user (Mr. A) side.

In the above-described embodiment, the image of the space captured when there is no user in the space corresponding to the background is used as the background image. However, the present invention is not limited to this. For example, when the camera 2 captures the user and the background at the same time, that is, the person video and the background video are separated from the actual video, and the separated background video is used. May be. In such a case, since a portion of the background video that overlaps the human video is missing, it is necessary to complement the background video. On the other hand, if the background video captured when there is no user is used, there is no omission of the video as described above, so it is not necessary to perform video complementation, so the background video can be acquired more easily. It becomes.

In the above-described embodiment, in the transition process executed when the movement of the second user's face is detected, the display position of the user's 3D video in the composite video and the composite video in the background 3D video It was decided to shift both the range (display range) included in the inside. However, the present invention is not limited to this, and only one of the display position of the user's 3D video and the display range of the background 3D video may be shifted, and the other may be fixed (not shifted).

In the above embodiment, in the transition process, the shift amount increases in the order of the display position of the foreground 3D image, the display position of Mr. A's 3D image, and the display range of the background 3D image. . However, the magnitude relationship of the shift amounts may be different from the above magnitude relationship. That is, the shift amount may increase in the order of the display range of the background 3D video, the display position of Mr. A's 3D video, and the display position of the foreground video. More specifically, when Mr. B's face moves sideways while the composite video shown in FIG. 20A is initially displayed on the display 5 on the B side, the second transition process is performed. As a result, the synthesized video gradually transitions to the state shown in FIG. 20A and 20B are explanatory diagrams relating to the second transition process, in which FIG. 20A shows a composite video before the second transition process, and FIG. 20B shows a composite video after the second transition process.

By the way, in the transition process described above (that is, the transition process illustrated in FIG. 11) and the second transition process illustrated in FIG. 20, the direction of the line of sight of Mr. The subject whose line of sight is facing is different. To make it easier to understand, if Mr. B is actually facing Mr. A, if Mr. B's face moves sideways with Mr. B's line of sight facing Mr. A, he will be farther away from Mr. B. The larger the amount of shift, the more the position is shifted from the original position. In order to reproduce such an appearance, in the transition process described above, that is, in the transition process illustrated in FIG. 11, the display position of the foreground 3D image, the display position of Mr. A's 3D image, and the tertiary of the background The amount of deviation increases in the order of the display range of the original video. On the other hand, when Mr. B's face moves sideways with Mr. B's line of sight facing Mr. A's background, the position closer to Mr. B is more displaced from the original position by the amount of deviation. Become visible. In order to reproduce such an appearance, in the second transition process, the amount of deviation increases in the order of the background 3D video display range, Mr. A's 3D video display position, and the foreground video display position. ing.

Regarding the execution mode of the transition process, the mode that maximizes the amount of shift in the display range of the background 3D video (corresponding to the transition process described above) and the amount of shift in the display position of the foreground 3D video It may be possible to switch between the largest mode (corresponding to the second transition process). In such a case, the transition process is appropriately executed according to the direction of the line of sight of Mr. B at that time.

1 Home server 2 Camera (imaging device)
3 Microphone 4 Infrared sensor 4a Light emitting unit 4b Light receiving unit 5 Display 5a Touch panel 6 Speaker 11 Data transmitting unit 12 Data receiving unit 13 Background video storage unit 14 First depth data storage unit 15 Real video storage unit 16 Human video extraction unit 17 Skeletal model Storage unit 18 Second depth data storage unit 19 Foreground image extraction unit 20 Height detection unit 21 3D image generation unit 22 Composite image display unit 23 Determination unit 24 Face movement detection unit 100 Communication unit GN External communication network S This system

Claims

A video acquisition unit that acquires the video of the user captured by the imaging device;
A distance data acquisition unit that acquires distance data indicating a distance from the object in the video piece from the imaging device for each video piece when the video is divided into a predetermined number of video pieces;
A 3D video generation unit that generates a 3D video of the user by executing a rendering process using the video of the user and the distance data;
A height detection unit that detects the height of the eyes of the user,
When both the height at which the imaging device is installed and the height of the eyes detected by the height detection unit are different, the 3D image generation unit determines the difference between the two and the imaging device and the user. The rendering process for acquiring the 3D video of the user when viewed from a virtual viewpoint at the eye height detected by the height detection unit is executed based on the distance between A video display system characterized by that.
The video acquisition unit acquires the video of the user captured by the imaging device and the background video captured by the imaging device, respectively.
The distance data acquisition unit acquires the distance data for each of the user's video and the background video,
The 3D video generation unit generates the 3D video of the user by executing the rendering process using the video of the user and the distance data acquired for the video of the user. Generating the 3D video of the background by performing the rendering process using the distance data acquired for the video and the video of the background;
The composite video display unit comprising: a composite video display unit configured to combine the 3D video of the user with the 3D video of the background and display a composite video positioned by the user in front of the background. The video display system according to 1.
The video acquisition unit further acquires a foreground video captured by the imaging device,
The distance data acquisition unit further acquires the distance data for the foreground video,
The 3D image generation unit further generates the 3D image of the foreground by executing the rendering process using the distance data acquired for the foreground image and the foreground image;
The synthesized video display unit synthesizes the 3D video of the user, the 3D video of the background, and the 3D video of the foreground, the user is positioned in front of the background, and the user 3. The video display system according to claim 2, wherein the composite video in which the foreground is positioned in front of is displayed on the display.
A determination unit that determines whether a distance between the imaging device and the user has changed based on the distance data;
When the determination unit determines that the distance between the imaging device and the user has changed while the imaging device is capturing the video of the user, the composite video display unit 4. The video display system according to claim 2, wherein the display size of the video of the user is adjusted to be the display size before the distance between the imaging device and the user is changed. .
A face movement detection unit that detects that the face of the second user who sees the composite image displayed on the display has moved in the width direction of the display;
When the face movement detection unit detects the movement of the face, the composite image display unit displays the composite image displayed on the display, and the state before the face movement detection unit detects the movement of the face. In the transition process, the display position of the 3D video of the user in the composite video and the range included in the composite video in the 3D video of the background 5. The video according to claim 2, wherein the composite video is transitioned to a state in which one of them is shifted in the width direction by an amount larger than the other shift amount. 6. Display system.
The video acquisition unit acquires the video of the user captured by a plurality of the imaging devices that capture the video of the user in different imaging directions, for each imaging device,
The distance data acquisition unit acquires the distance data about the user's video for each imaging device,
The 3D video generation unit
A video piece generating step for generating a 3D video piece of the user for each imaging device based on the video of the user acquired for the imaging device and the distance data acquired for the imaging device;
In order to generate the 3D video of the user, a combining step of combining each of the 3D video pieces of the user for each imaging device so that common video areas included in the respective 3D video images overlap each other,
When generating the 3D image piece of the portion including the user's eyes in the image piece generating step, if the two are different, the difference between the two and the distance between the imaging device and the user are set. The video display system according to any one of claims 2 to 5, wherein the rendering processing for acquiring the 3D video piece when viewed from the virtual viewpoint is executed based on the rendering process. .
When the imaging direction is different from the normal direction of the reference plane, the 3D video generation unit generates the 3D video piece of the user generated based on the video captured in the imaging direction in the video piece generation step. The video display system according to claim 6, wherein the video is converted into the three-dimensional video piece when viewed virtually from the normal direction.
A computer acquiring an image of a user imaged by an imaging device;
Obtaining distance data indicating a distance from the imaging device to the object in the video piece for each video piece when the computer divides the video into a predetermined number of video pieces;
A computer generates a 3D video of the user by executing a rendering process using the video of the user and the distance data;
Detecting a height of the user's eyes,
When both the height at which the imaging device is installed and the detected eye height are different, the computer detects the detected based on the difference between the two and the distance between the imaging device and the user. An image display method comprising: executing the rendering process for acquiring the 3D image of the user when viewed from a virtual viewpoint at an eye level.