JP2011097447A - Communication system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a communication system capable of bringing the visual lines of conversation persons into a line even if the conversation persons do not face display surfaces. <P>SOLUTION: The communication system includes: an image output unit for outputting an image produced by viewing a first viewer, being a conversation person, approximately from the front side; a display unit including the display surface for displaying the image of the first viewer; and an image enlarging and reducing unit for calculating the image produced by performing, to the image output by the image outputting unit, a transmission and projection conversion which is directed to a display surface from a virtual display surface facing a straight line connecting the second viewer reference position which is a position of a visual point of a second viewer conversing with the first viewer and the center of the display surface and centers on a second viewer reference position. Then, the display unit displays the image produced by the image enlarging and reducing unit. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a communication system.

In a communication tool that uses video, such as a video conference system that displays each other's video for a plurality of conference participants located at different points, the talker is the person who is displayed on the display device, especially the face of the other party's talker Have a conversation while watching. Here, the talker means the person who is speaking and the person with whom the person speaks. For example, in a video conference system, when a conference participant who is speaking (hereinafter also referred to as a “speaker”) or when this speaker speaks to a single conference participant, the conference participation to be spoken Refers to the person. In the following, a person who views an image displayed by the communication system, for example, a participant in a video conference is also referred to as a “viewer”. In this way, it is important for conversations to have a conversation while looking at each other's faces. Furthermore, it is important for the conversations to have the same line of sight. is there.
In Patent Document 1, a front image of the first user is captured by providing a plurality of imaging devices around the display surface of the display device viewed by the first user (conversator), and the first user is selected by the first user. There has been proposed a method of matching the line of sight of the first user and the second user by transmitting a front image of the user and transmitting a profile image of the first user to other users. As a result, an image whose line of sight matches that of the other user can be displayed for the user who is directly facing the display device.

JP 2001-136501 A

  However, in the above method, when the talkers are not directly facing the display surface of the display device, the lines of sight of the talkers do not match. For example, when there are a plurality of viewers in one conference room and each of them views one display device, when a front image of a conversation person is displayed on the display surface, An image whose line of sight matches the conversation person can be seen, but a viewer who is in a seat not facing the display surface sees an image whose line of sight does not coincide with the conversation person. Therefore, when a viewer located at a seat not facing the display device has a conversation, the conversation is performed while viewing an image in which the other conversation person faces in a different direction. For this reason, face-to-face conversation, which is important in non-verbal communication, especially communication by line of sight cannot be sufficiently achieved.

  The present invention has been made in view of such circumstances, and a purpose thereof is a communication system capable of displaying an image whose line of sight coincides with that of a conversational partner even when the conversational person is not facing the display surface. Is to provide.

[1] The present invention has been made to solve the above-described problems, and a communication system according to an aspect of the present invention captures an image of a first viewer and views the first viewer from substantially the front. An image output unit for outputting, a reference position output unit for detecting a second viewer reference position, which is a position of the viewpoint of the second viewer who talks with the first viewer, and an image of the first viewer are displayed. From the virtual display surface facing the straight line connecting the second viewer reference position and the center of the display surface to the display surface with respect to the display unit including the display surface and the image output from the image output unit An image expansion / contraction unit that calculates an image subjected to perspective projection conversion centered on the second viewer reference position, and the display unit displays the image calculated by the image expansion / contraction unit. And
In this communication system, the image expansion / contraction unit generates an image based on the second viewer reference position, which is the position of the conversation person's viewpoint. Even if the conversation person is not facing the display screen, an image whose line of sight matches that of the other conversation person can be displayed. It is possible to display a natural image that can be seen when the other party's conversation person is viewed from the front, instead of the displayed image.

[2] A communication system according to an aspect of the present invention is the communication system described above, wherein the reference position output unit includes a reference position that is the position of the viewer's viewpoint, including the second viewer. A reference position detecting unit for detecting the viewer, and a reference position selecting unit for selecting the second viewer reference position from the reference positions detected by the reference position detecting unit. To do.
This communication system selects a second viewer reference position that is a reference position of a conversation person from a plurality of viewer reference positions, and the image expansion / contraction unit generates an image based on the second viewer reference position. Even if there are two or more viewers viewing the part, even if the conversation person is not directly facing the display surface for the conversation person who is viewing the display part from the second viewer reference position, Can be displayed, and the other party's talker seen when viewing the display from an angle is not an image that is deformed vertically, but is natural when the other party's talker is seen from the front. An image can be displayed.

[3] A communication system according to an aspect of the present invention is the communication system having the above-described communication system and the first terminal device and the second terminal device connected via a communication path, wherein the first terminal The device includes the image output unit and the reference position selection unit, and the second terminal device includes the reference position detection unit, the display unit, and the image expansion / contraction unit. Features.
In this communication system, the second viewer reference position, which is the reference position of the conversation person, is selected from the reference positions of the plurality of viewers as in the above communication system, and the image expansion / contraction unit is based on the second viewer reference position Since the image is generated, even when there are two or more viewers viewing the display unit, even if the conversation person is not directly facing the display surface with respect to the conversation unit viewing the display unit from the second viewer reference position It is possible to display an image whose line of sight matches that of the other party's talker, and the other party's talker is not a vertically deformed image when the display surface is viewed obliquely. It is possible to display a natural image that can be seen when entangled.

[4] A communication system according to an aspect of the present invention is the communication system described above, and includes a second display unit having a second display surface that displays images of one or more viewers including the second viewer. The image output unit further includes: a first imaging unit that captures a first stereo image with the first viewer as a subject; and an image including the face of the first viewer from the first stereo image. Detecting the direction of the line of sight of the first viewer; generating a three-dimensional model including the face of the first viewer based on the first stereo image; A free viewpoint image generation unit that generates and outputs an image including a viewer's face, and the reference position detection unit captures a second stereo image having the one or more viewers as subjects. Two imaging units and the second A reference position calculation unit for detecting a face of each of the one or more viewers or a part of the image from a Leo image and calculating a reference position of each of the one or more viewers, and The reference position selection unit detects the first viewer's line-of-sight direction from the first stereo image, and selects the viewer displayed at the intersection of the line-of-sight direction and the second display surface as the second viewer. The reference position of the viewer is selected as the second viewer reference position from the reference positions calculated by the reference position calculation unit.
In this communication system, the first imaging unit and the free viewpoint image generation unit output an image obtained by viewing the first viewer from the front (line of sight), and the second imaging unit and the reference position calculation unit detect the reference position. In addition, since the reference position selection unit selects and outputs the second viewer reference position from the reference positions, the second viewing is performed even when there are two or more viewers viewing the display unit, as in the communication system described above. Even if this conversation person is not directly facing the display surface, the image whose line of sight coincides with the other party's conversation person can be displayed, and the display surface It is possible to display a natural image that can be seen when the other party's conversation person is viewed from the front, instead of an image in which the other person's conversation person who is viewed from an oblique direction is deformed vertically.

[5] A communication system according to an aspect of the present invention is the communication system described above, wherein the display surface displays different images in at least two directions, and the reference position selection unit includes the second viewer reference position. And the second viewer reference position is associated with any one of display directions in which the display surface displays the different images, and the display unit displays an image input from the image expansion / contraction unit. And displaying in a direction associated with the second viewer reference position.
This communication system can display an image in which the line of sight of the conversation person coincides with the direction of the selected second viewer reference position, as with the communication system described above. , Natural images can be displayed.

[6] A communication system according to an aspect of the present invention is the communication system described above, and includes a second display unit having a second display surface that displays images of one or more viewers including the second viewer. The image output unit further includes: a first imaging unit that captures a first stereo image with the first viewer as a subject; and an image including the face of the first viewer from the first stereo image. Detecting the direction of the line of sight of the first viewer; generating a three-dimensional model including the face of the first viewer based on the first stereo image; A free viewpoint image generation unit that generates and outputs an image including a viewer's face, and the reference position detection unit captures a second stereo image having the one or more viewers as subjects. Two imaging units and the second A reference position calculation unit for detecting a face of each of the one or more viewers or a part of the image from a Leo image and calculating a reference position of each of the one or more viewers, and The reference position selection unit detects the first viewer's line-of-sight direction from the first stereo image, and selects the viewer displayed at the intersection of the line-of-sight direction and the second display surface as the second viewer. The reference position of the viewer is selected as the second viewer reference position from the reference positions calculated by the reference position calculation unit, and the positional relationship between the center of the display surface and the second viewer reference position Based on the above, any one of the display directions is associated with the second viewer reference position.
In this communication system, the first imaging unit and the free viewpoint image generation unit output an image obtained by viewing the first viewer from the front (line of sight), and the second imaging unit and the reference position calculation unit detect the reference position. In addition, since the reference position selection unit selects and outputs the second viewer reference position from the reference positions, as in the communication system described above, the conversation person is displayed in the direction of the selected second viewer reference position. Can be displayed, and a natural image can be displayed as in the communication system described above.

[7] A communication system according to an aspect of the present invention is the communication system described above, wherein the display surface displays different images in at least two directions, and the reference position selection unit has the display surface different in image. Any one of the reference positions is associated with each display direction that is a direction for displaying the image, and the image output unit is based on a reference position associated with a direction not associated with the second viewer reference position. Generating the first viewer image viewed from the position determined, and associating the direction with the image and outputting the image to the image expansion / contraction unit, the image expansion / contraction unit for each of the input directions From the virtual display surface facing the straight line connecting the reference position associated with the direction and the center of the display surface to the image associated with the direction from the virtual display surface to the display surface An image obtained by performing perspective projection transformation centered on the reference position associated with the direction is calculated, the calculated image is output in association with the direction, and the display unit is input from the image expansion / contraction unit. The image is displayed in a direction associated with the image.
This communication system outputs an image viewed from a position determined based on the reference position, with respect to a reference position that is not selected by the reference position selection unit. By displaying the image of the conversation person facing the direction of the second viewer reference position as this image, a more natural image in which the conversation persons face each other can be displayed to a non-conversation viewer.

[8] A communication system according to an aspect of the present invention is the communication system described above, and includes a second display unit having a second display surface that displays an image of one or more viewers including the second viewer. The image output unit further includes a first imaging unit that captures a first stereo image with the first viewer as a subject, and an image including the face of the first viewer from the first stereo image. And detecting a line-of-sight direction of the first viewer, generating a three-dimensional model including the face of the first viewer based on the first stereo image, and the reference position selection unit in the display direction For each of the viewpoint positions excluding the second viewer reference position among the associated viewpoint positions, the second viewer reference is determined from the viewpoint position centered on the center position of the display surface. Orientation to position And generating an image viewed from a position moved in the detected direction from the line-of-sight direction of the first viewer, and associating the generated image with the display direction associated with the position of the viewpoint A free viewpoint image generation unit that outputs to an image expansion / contraction unit, wherein the reference position detection unit captures a second stereo image having the one or more viewers as subjects, and the first imaging unit A reference position calculation unit for detecting a face of the one or more viewers or a part of the image from two stereo images and calculating a reference position of each of the one or more viewers, and the reference The position selection unit detects the first viewer's line-of-sight direction from the first stereo image, and calculates the reference position of the viewer displayed at the intersection of the line-of-sight direction and the second display surface. The second viewer from the reference position calculated by the department The second viewer is selected as a quasi-position, and any one of the display directions is associated with the second viewer reference position based on the positional relationship between the center of the display surface and the second viewer reference position. A reference position other than the second viewer reference position is associated with each display direction other than the direction associated with the reference position based on the positional relationship between the display surface and each reference position. And
This communication system displays an image of a conversation person facing the second viewer reference position for each of the viewpoint positions excluding the second viewer reference position. Therefore, it is possible to display a more natural image in which the talkers face each other for viewers who are not talkers.

[9] Also, a communication system according to an aspect of the present invention is connected to each other via the communication system described above, and a first terminal device used by the first viewer and a second viewer used by the second viewer. A communication system having a second terminal device and a third terminal device used by a third viewer, wherein any two of the first viewer, the second viewer, and the third viewer are conversational persons. The image output unit further includes a selection unit for selecting a conversation person. When the conversation person selection unit selects the first viewer and the second viewer, the image output unit substantially faces the first viewer. When the first viewer and the third viewer are selected, an image of the first viewer viewed from outside the front is generated when the first viewer and the third viewer are selected. Outputting to the image expansion / contraction part And features.
In this communication system, when the first viewer and the second viewer are conversational persons, an image whose line of sight matches the first viewer can be displayed to the second viewer, as in the above communication system. Moreover, a natural image can be displayed similarly to the communication system described above. Thereby, the second viewer can have a conversation while looking at a natural image whose line of sight matches that of the other party's conversation. Further, when the first viewer and the third viewer are the talkers and the second viewer is not the talker, an image obtained by viewing the first viewer from a position other than the front is displayed to the second viewer. By displaying an image of the first viewer facing the direction of the display unit on which the third viewer is displayed, the second viewer who is not a conversation person is the first viewer who is the conversation person. You can see a more natural image facing the direction.

[10] A communication system according to an aspect of the present invention is the above communication system, and a conversation person connected to the first terminal device, the second terminal device, and the third terminal device via a communication path. A communication system having a selection device, wherein the first terminal device includes the image output unit, and the second terminal device includes the reference position output unit, the display unit, the image expansion / contraction unit, And the talker selection device comprises the talker selection unit.
Similar to the communication system described above, this communication system can display an image whose line of sight matches the first viewer to the second viewer when the first viewer and the second viewer are conversational persons. Moreover, a natural image can be displayed like the above-mentioned communication system. Thereby, the second viewer can have a conversation while looking at a natural image whose line of sight matches that of the other party's conversation. Further, when the first viewer and the third viewer are the talkers and the second viewer is not the talker, an image obtained by viewing the first viewer from a position other than the front is displayed to the second viewer. By displaying an image of the first viewer facing the direction of the display unit on which the third viewer is displayed, the second viewer who is not a conversation person is the first viewer who is the conversation person. You can see a more natural image facing the direction.

[11] A communication system according to an aspect of the present invention is the communication system described above, further including a third viewer imaging unit that images a third viewer, and the image output unit includes the first viewing unit. A first viewer imaging unit that captures a first stereo image of a person as a subject, and an image including the face of the first viewer is detected from the first stereo image, and the line-of-sight direction of the first viewer is detected And when a three-dimensional model including the face of the first viewer is generated based on the first stereo image, and the talker selection unit selects the first viewer and the second viewer. Generates and outputs an image in which the first viewer is viewed from substantially the front, and when the talker selection unit selects the first viewer and the third viewer, the center of the display surface Centered on the position of the second display surface Detecting the direction from the position of the heart in the direction of the reference position output by the reference position output unit, and generating an image viewed from the position moved in the detected direction from the viewing direction of the first viewer, A free viewpoint image generation unit that outputs the first viewer image as viewed from other than the front, and the reference position output unit captures a second stereo image with the second viewer as a subject. A second viewer imaging unit, and a reference position calculation unit that detects the second viewer's face or a part of the image from the second stereo image and calculates the second viewer reference position, The talker selection unit includes the first viewer, the image captured by the second viewer imaging unit, and the image captured by the third viewer imaging unit, respectively. The frequency of mouth movements and the frequency of mouth movements of the second viewer And the third viewer's mouth movement frequency, select a talker based on the detected frequency, detect a gaze direction from an image of the selected talker, and based on the detected gaze direction It is characterized by selecting a conversation person.
Similar to the communication system described above, this communication system can display an image whose line of sight matches the first viewer to the second viewer when the first viewer and the second viewer are conversational persons. Moreover, a natural image can be displayed like the above-mentioned communication system. Thereby, the second viewer can have a conversation while seeing a natural image whose line of sight coincides with the other conversation person. In addition, when the first viewer and the third viewer are the talkers and the second viewer is not the talker, the image of the first viewer facing the direction of the display unit on which the third viewer is displayed. Is displayed, the second viewer who is not a conversation person can see a more natural image in which the first viewer who is the conversation person faces the other conversation person.

  According to the present invention, it is possible to display an image whose line of sight matches that of the other party's conversation even when the conversation is not facing the display surface.

1 is a system configuration diagram showing a schematic configuration of a video conference system 1 in a first embodiment of the present invention. It is a top view in conference room R1 in which the video conference terminal device 11 in the same embodiment is installed. It is a top view in conference room R2 in which the video conference terminal device 12 in the same embodiment is installed. It is a block diagram which shows schematic structure of the video conference terminal devices 11 and 12 in the embodiment. It is the front view which looked at the display part 116 in the same embodiment from the near-right direction with respect to the display surface. It is the front view which looked at the display part 127 in the same embodiment from the orthogonal | vertical direction front with respect to the display surface. It is a figure which shows the relative coordinate of the reference position which the reference position calculation part 122 calculates in the same embodiment. 4 is a data configuration diagram illustrating a data configuration of data input to a reference position selection unit 112 by a reference position calculation unit 122 in the embodiment. FIG. It is a figure which shows the eyes | visual_axis which the viewer P11 sees the image of the viewer P24 on the display part 116 in the embodiment. 4 is a data configuration diagram illustrating a data configuration of information on a reference position input to an image expansion / contraction unit 126 by a reference position selection unit 112 in the embodiment. FIG. 5 is a flowchart illustrating a processing procedure for generating an image when the free viewpoint image generation unit 113 captures an image from the center front of the display surface of the display unit 116 in the embodiment. It is a figure which shows the expansion-contraction of the image which the image expansion-contraction part 126 performs in the embodiment. 5 is a flowchart illustrating a processing procedure in which an image expansion / contraction unit 126 expands / contracts an image in the embodiment. 5 is a diagram illustrating an example of an image output from a free viewpoint image generation unit 113 and an image in which an image expansion / contraction unit 126 performs image expansion / contraction in the embodiment. FIG. It is a system block diagram which shows schematic structure of the video conference system 2 in the 2nd Embodiment of this invention. In the embodiment, it is a figure which shows the example of the screen which each display part displays. In the same embodiment, it is a flowchart which shows the process sequence in which the conversation person selection part 241 selects a conversation person. In the same embodiment, it is a data block diagram of the data which the conversation person selection part 241 inputs into the free viewpoint image generation part 213, 223, and 233. FIG. It is a figure which shows the angle which the free viewpoint image generation part 223 and the free viewpoint image generation part 233 rotate an imaging position when it determines with the viewer P31 not being a talker in the embodiment. It is a figure which shows the imaging position of the image which the free viewpoint image generation parts 223 and 233 generate | occur | produce in the embodiment. It is a figure which shows the expansion-contraction of the image which the image expansion-contraction parts 214 and 215 perform in the embodiment. It is a system block diagram which shows schematic structure of the video conference system 3 in the 3rd Embodiment of this invention. It is a figure which shows the example of the image which the display part 328 displays in the embodiment. It is a figure which shows the example of the image which the display part 316 displays in the same embodiment. 4 is a data configuration diagram illustrating a configuration of data input to a free viewpoint image generation unit 313 by a reference position selection unit 312 in the embodiment. FIG. In the embodiment, when the viewer P51 is determined to be a conversation person, the free viewpoint image generation unit 313 shows an angle at which the imaging position is rotated.

<First Embodiment>
Embodiments of the present invention will be described below with reference to the drawings. Below, the case where this invention is applied to a video conference system as an example of a communication system is demonstrated, However, The application range of this invention is not restricted to a video conference system. The communication system here is a system that displays an image of a party of communication to the other party. In addition, there are educational systems for teaching lessons by displaying teacher images on a display.

  FIG. 1 is a system configuration diagram showing a schematic configuration of a video conference system 1 according to the first embodiment of the present invention. In FIG. 1, the video conference system 1 includes a video conference terminal device (first terminal device) 11 and a video conference terminal device (second terminal device) 12 connected to each other by a communication network 13. The image picked up by the video conference terminal device 11 and the collected voice are sent to the video conference terminal device 12 via the communication network 13 by the transmission device, and the image and the voice are transmitted by the display unit via the reception device. Reproduced and displayed. In addition, the image picked up by the video conference terminal device 12 and the collected sound are sent to the video conference terminal device 11 via the communication network 13 by the transmission device, and the image and the sound are collected by the display unit via the reception device. The sound is reproduced and displayed.

  FIG. 2 is a plan view of the conference room R1 in which the video conference terminal device 11 is installed. As will be described later, the video conference terminal device 11 includes imaging devices 111-1 and 111-2 and a display unit (second display unit) 116. In the figure, imaging devices 111-1 and 111-2, a display unit 116, and a desk T1 are arranged in a conference room R1, and a viewer (first viewer) P11 is present. In addition, illustration is abbreviate | omitted about arrangement | positioning of the other part which comprises the video conference terminal device 11. FIG. These may be arranged in the conference room R1, or may be arranged outside the conference room R1. Alternatively, the display unit 116 may be incorporated in the housing.

  FIG. 3 is a plan view of the conference room R2 in which the video conference terminal device 12 is installed. As will be described later, the video conference terminal device 12 includes imaging devices 121-1 and 121-2 and a display unit 127. In the figure, in the conference room R2, imaging devices 121-1 and 121-2, a display unit 127, and a desk T2 are arranged, and there are viewers P21, P22, P23, P24, and P25. In addition, illustration is abbreviate | omitted about arrangement | positioning of the other part which comprises the video conference terminal device 12. FIG. These may be arranged in the conference room R2, or may be arranged outside the conference room R2. Alternatively, the display unit 127 may be incorporated in the housing.

  FIG. 4 is a configuration diagram showing a schematic configuration of the video conference terminal devices 11 and 12 and a connection relationship therebetween via a communication network (FIG. 1). In FIG. 1, the video conference system 1 includes a video conference terminal device 11 and a video conference terminal device 12. The video conference terminal device 11 includes an imaging unit (first imaging unit) 111, a reference position selection unit 112, a free viewpoint image generation unit 113, and a display unit 116. The imaging unit (first imaging unit) 111 includes imaging devices 111-1 and 111-2. The video conference terminal device 12 includes an imaging unit (second imaging unit) 121, a reference position calculation unit 122, an image expansion / contraction unit 126, and a display unit 127. The imaging unit 121 includes imaging devices 121-1 and 111-2. Note that the imaging unit 111 may include three or more imaging devices. Similarly, the imaging unit 121 may include three or more imaging devices. By increasing the number of imaging devices included in the imaging unit 121 or the imaging unit 131 and capturing an image of a wide range of viewers as subjects, images of various orientations are generated when generating viewer images described later. be able to. Further, by using an image captured from a direction close to the direction of the image to be generated, when generating an image of a viewer described later, more accurate three-dimensional coordinate data is calculated, and a more accurate image is generated. can do.

  The imaging unit 121 and the reference position calculation unit 122 correspond to the reference position detection unit 123 of the present invention, and the viewer's viewpoint position (hereinafter also referred to as a reference position) includes a second viewer who is a talker. Detect more than viewers. In the present embodiment, the reference position detection unit 123 detects the reference position by calculating the reference position using the image captured by the imaging unit 121 and the reference position calculation unit 122. Note that the method by which the reference position detection unit detects the reference position is not limited to the method described above. For example, the reference position detection unit is composed of a reference position calculation unit, and the reference position detection unit detects the reference position by using the position sensor as described later, and the reference position detection unit detects the reference position. A method may be used.

Further, the reference position detection unit 123 and the reference position selection unit 112 correspond to the reference position output unit of the present invention, and a reference position of a second viewer who is a talker (hereinafter also referred to as a second viewer reference position). ) Is output. In the present embodiment, when the reference position selection unit 112 selects the reference position of the second viewer who is a conversation person from the reference positions detected by the reference position detection unit 123, the reference position output unit The viewer reference position is detected.
In addition, the imaging unit 111 and the free viewpoint image generation unit 113 correspond to the image output unit 114 of the present invention, and images the first viewer and views the first viewer from the front (hereinafter, the front image). (Also called). In the present embodiment, the free viewpoint image generation unit 113 generates a front image of the viewer P11 (first viewer) using the image captured by the imaging unit 111, and outputs the generated front image. The output unit 114 outputs a front image. Note that the method by which the image output unit outputs the front image is not limited to the method described above. For example, the image output unit includes an imaging device, and the image output device captures an image from the center front of the display surface of the display unit 116 by an existing method using a half mirror, as will be described later. Other methods may be used such that the unit outputs a front image.
Note that the image output by the image output unit 114 does not have to be an image obtained by viewing the first viewer from the front. If the viewer does not feel uncomfortable, an image viewed from a position shifted from the front. It may be.

  The imaging devices 111-1, 111-2, 121-1, and 121-2 are configured to include a CCD (Charge Coupled Device) and a lens, and are cameras that capture a moving image. The imaging device 111-1, 111-2, 121-1, or 121-2 includes a solid-state imaging device such as a CMOS (Complementary Metal Oxide Semiconductor) as an imaging element instead of a CCD. It may be configured. Further, the display unit 116 may incorporate a camera module as the imaging device 111-1 or 111-2, and the display unit 127 may incorporate a camera module as the imaging device 121-1 or 121-2. Also good.

The display units 116 and 127 are configured to include a display surface of the liquid crystal panel, and display an image such as a moving image on the display surface. The display units 116 and 127 may include a display surface other than a liquid crystal panel such as a plasma display panel.
The free viewpoint image generation unit 113 generates a front image based on the image captured by the imaging unit 111.
The reference position calculation unit 122 calculates, based on the image captured by the imaging unit 121, the center position of a line connecting the centers of both eyes of the viewer as the reference position. The reference position selection unit 112 selects the reference position of the conversation person from the reference positions calculated by the reference position calculation unit 122. Hereinafter, the selected conversation person is also referred to as a second viewer, and the reference position of the second viewer is also referred to as a second viewer reference position.
The image expansion / contraction unit 126 is displayed on the display unit 127 when the conversation person (second viewer) is positioned obliquely with respect to the display unit 127, and the viewer in the image is seen from the front from the position of the conversation person. Scale the image so that it looks like it looks.

  FIG. 5 is a front view of the display unit 116 as viewed from the front in the direction perpendicular to the display surface. The display unit 116 displays images including the viewers P21 to P25. Imaging devices 111-1 and 111-2 are installed at the left and right ends of the upper portion of the display unit 116.

  FIG. 6 is a front view of the display unit 127 as viewed from the front in the direction perpendicular to the display surface. The display unit 127 displays an image including the viewer P11. Imaging devices 121-1 and 121-2 are installed on the left and right ends of the upper portion of the display unit 127.

Next, the operation of the video conference system 1 will be described.
The imaging unit 121 captures a stereo image (second stereo image) including the viewers P21 to P25. Here, the stereo image refers to a plurality of images obtained by simultaneously imaging the subject from different angles. The imaging unit 121 captures a stereo image by simultaneously capturing the viewers P21 to P25, which are subjects, with the imaging unit 121-1 and the imaging unit 121-2. The imaging unit 121 inputs the captured image to the reference position calculation unit 122 and also inputs the image to the display unit 116 via the communication network 13 (FIG. 1). The display unit 116 displays an image captured by the imaging device 121-1 among images received from the imaging unit 121. Note that a free viewpoint image generation unit may be provided between the imaging unit 121 and the display unit 116, and the free viewpoint image generation unit may generate an image captured from the center of the display surface of the display unit 127. In this case, the display unit 116 displays an image captured from the center of the display surface of the display unit 127. Thus, when a conversation person among the viewers P21 to P25 gazes at the display unit 127, the viewer P11 can have a conversation while looking at an image whose line of sight coincides with the conversation person. This free viewpoint image generation unit generates an image captured from the center of the display surface of the display unit 127 by generating a three-dimensional model using the stereo matching method, as in the free viewpoint image generation unit 113 described later. To do.
The reference position calculation unit 122 calculates the reference positions of the viewers P21 to P25 based on the image received from the imaging unit 121. The reference position calculation unit 122 calculates the relative coordinates of each reference position with respect to the center of the display surface of the display unit 127. Then, the reference position calculation unit 122 outputs the calculated reference positions to the reference position selection unit 112 via the communication network 13 (FIG. 1).

  FIG. 7 is a diagram illustrating the relative coordinates of the reference position calculated by the reference position calculation unit 122. In the figure, the reference position P of the viewer P24 is shown. As shown in FIG. 7A, the reference position calculation unit 122 sets the center of the display surface of the display unit 127 as the origin O (0, 0, 0). Further, the reference position calculation unit 122 sets a straight line extending perpendicularly from the origin O to the display surface of the display unit 127 as the z axis, and sets the front direction of the display surface of the display unit 127 as the positive direction of the z axis. Further, the display surface of the display unit 127 is a horizontally long rectangle, and the reference position calculation unit 122 sets a straight line passing through the origin O and parallel to the long side of the display surface as the x axis, and the right direction when viewed from the front is a positive x axis. And the direction. Further, the reference position calculation unit 122 sets a straight line passing through the origin O and parallel to the short side of the display surface as the y-axis, and upward as a positive direction of the y-axis. The reference position calculation unit 122 calculates the coordinates (xp, yp, zp) of the reference position P of the viewer P24. The reference position calculation unit 122 calculates the reference position in the same manner for other viewers.

The reference position calculation unit 122 stores the angle of view of the imaging devices 121-1 and 121-2 and the relative position and direction with respect to the display unit 127 in an internal storage unit (not shown). Then, the reference position calculation unit 122 calculates the coordinates of the reference position of each viewer by the stereo matching method using the images captured by the imaging devices 121-1 and 121-2.
Specifically, the reference position calculation unit 122 first performs face detection on an image captured by the imaging device 121-1 and an image captured by the imaging device 121-2 by skin color and face shape pattern matching. I do. When a plurality of face images are detected, the reference position calculation unit 122 determines that the face images having the smallest deviation amount between the two images are the same face images. The same applies to the correspondence between a plurality of eyes. Then, the reference position calculation unit 122 further detects both eyes from the detected faces based on the color and shape of the black eyes and the color and shape of the eyebrows. The reference position calculation unit 122 calculates three-dimensional coordinates by triangulation based on the detected amount of the center of both eyes and the positional relationship between the cameras and the point of interest between the images. Further, the reference position calculation unit 122 calculates the three-dimensional coordinates of the center of the line segment connecting the eyes from the three-dimensional coordinates of both eyes, and uses the three-dimensional coordinates as the coordinates of the reference position.

The reference position calculation unit 122 detects the face by pattern matching based on the shape of the eye, detects the face or a part of the image such as a midpoint of a straight line connecting both eyes, and the like, The reference position may be calculated.
Note that the method by which the reference position calculation unit 122 calculates the reference position is not limited to the above-described stereo matching method. For example, the position of the viewers P21 to P25 may be detected using a position detection sensor, and the reference position may be calculated using another method, such as approximately obtaining the center of the detected position as the reference position. .
Alternatively, the reference position calculation unit 122 may calculate the coordinates of the reference position P in a coordinate format other than the above, for example, in the polar coordinate format illustrated in FIG. For example, as shown in FIG. 5B, the reference position calculation unit 122 sets the origin O in FIG. 5A as the origin and the distance from the origin O to the reference position P as r. Further, the reference position calculation unit 122 sets the angle from the z axis to the point (xp, 0, zp) around the origin O as φ, and the counterclockwise direction when viewed from the positive direction of the y axis as a positive angle of φ. And Further, the reference position calculation unit 122 rotates from the point (xp, 0, zp) to the positive side of the y-axis with θ being the angle from the point (xp, 0, zp) to the point P with the origin O as the center. The side is a positive angle. The reference position calculation unit 122 calculates the coordinates (r, φ, θ) of the reference position P.

The reference position calculation unit 122 calculates the area occupied by each face detected in the face detection in the image, and outputs it to the reference position selection unit 112 in association with the reference position obtained based on each face detection.
FIG. 8 is a data configuration diagram illustrating a data configuration of data output from the reference position calculation unit 122 to the reference position selection unit 112.
In the figure, the data output from the reference position calculation unit 122 to the reference position selection unit 112 includes information on the reference positions for the number of viewers and information representing the area occupied by each viewer's face in the image. Is done.
The reference position information is information indicating the coordinates of the viewer's reference position. The coordinate information on the display unit is information indicating the coordinates of the area occupied by the viewer's face in the image. The reference position calculation unit 122 outputs the minimum value and maximum value of the x coordinate and the minimum value and maximum value of the y coordinate of the area occupied by the viewer's face in the image. The information indicating the area occupied by the viewer's face in the image by the reference position calculation unit 122 is not limited to the minimum value and maximum value of the x coordinate and the minimum value and maximum value of the y coordinate. For example, the reference position calculation unit 122 may approximate the viewer's face with an ellipse and use the coordinates of the center point and the lengths of the major axis and the minor axis as information representing the area occupied by the face in the image. Alternatively, the reference position calculation unit 122 may use the coordinates of each vertex obtained by approximating the viewer's face at a polygonal position with more corners than the quadrangle as information representing the area occupied by the face in the image.
Further, the unit of the reference position data in the figure is millimeters, and the unit of coordinates on the display unit is pixels. Note that the unit of the data of the reference position is not limited to millimeters, and other units representing the length such as inches may be used. The unit of coordinates on the display unit is not limited to pixels, and for example, the horizontal side length of the display surface is set to 1, and the horizontal coordinate is expressed by the relative length from the left end of the display surface. Other units may be used.

The imaging unit 111 captures an image of the viewer P11 and inputs it to the reference position selection unit 112 and the free viewpoint image generation unit 113. The imaging unit 111 captures a stereo image (first stereo image) of the viewer P11 using the imaging device 111-1 and the imaging device 111-2. The reference position selection unit 112 selects one reference position corresponding to the talker from the reference positions received from the reference position calculation unit 122 via the communication network 13 (FIG. 1) and outputs the selected reference position to the image expansion / contraction unit 126. To do. The reference position selection unit 112 detects the line of sight of the viewer P11 from the image received from the imaging unit 111, determines that the viewer who the viewer P11 is paying attention to is a talker, and the reference position corresponding to the talker Select.
Specifically, the reference position selection unit 112 first detects the face of the viewer P11 from the image captured by the imaging unit 111, and detects eyes from the detected face. The reference position selection unit 112 detects the viewing direction of the viewer P11 from the detected face direction and eye position. The direction of the face is determined based on the size of the left and right eye areas, for example. The reference position selection unit 112 detects an intersection between the detected line-of-sight direction and the display surface of the display unit 116 as a point on the display unit 116 that is viewed by the viewer P11. Then, the reference position selection unit 112 receives points from the reference position calculation unit 122 on points on the display unit 116 that the viewer P11 pays attention to from the information about the area occupied by each face of the viewers P21 to P25 in the image. By selecting the information of the area including the viewer P11, the viewer who pays attention to the viewer P11 is selected as the talker. The reference position selection unit 112 outputs the reference position associated with the selected region information to the image expansion / contraction unit 126 via the communication network 13 as the reference position corresponding to the talker.

  FIG. 9 is a diagram illustrating a line of sight when the viewer P11 views the image of the viewer P24 on the display unit 116. In the figure, images of the viewers P21 to P25 are displayed on the display unit 116, and the viewer P11 is watching the viewer P24. The viewers P11 are imaged by the imaging devices 111-1 and 111-2 installed in the display unit 116. The reference position selection unit 112 selects the reference position by detecting the viewing direction of the viewer P11 as described above using the images captured by the imaging devices 111-1 and 111-2.

FIG. 10 is a data configuration diagram showing the data configuration of the reference position information output from the reference position selection unit 112 to the image expansion / contraction unit 126. In the figure, the reference position information output from the reference position selection unit 112 to the image expansion / contraction unit 126 includes coordinates of one reference position. As will be described later, the image expansion / contraction unit 126 expands / contracts the image received from the imaging unit 111 based on the reference position.
In addition, the method for the reference position selection unit 112 to select one reference position corresponding to the talker is not limited to the method for detecting the line of sight described above. For example, the display unit 116 may be a touch panel, and the viewer P11 may select the other party's talker by touching the position on the display unit 116 where the viewer is displayed. The reference position selection unit 112 selects a reference position corresponding to the selected viewer. Alternatively, when the reference position selection unit 112 displays a cursor on the display unit 116, and the viewer P11 operates the cursor position with a remote controller or the like to select a conversation partner, the reference position selection unit 112 is selected. A reference position corresponding to the talker may be selected. Further, when any of the viewers P21 to P25 speaks to the viewer P11, the reference position selection unit 112 identifies the viewer who is speaking by voice recognition or detection of mouth movements, and talks to the identified viewer. You may make it select as a person. Alternatively, microphones may be provided for each of the viewers P21 to P25, and the reference position selection unit 112 may detect that the viewer has switched on the microphone for speaking and select a talker.

The free viewpoint image generation unit 113 generates an image captured from the center front of the display surface of the display unit 116 based on the image captured by the imaging unit 111. The free viewpoint image generation unit 113 generates an image using an existing method for generating an image of an arbitrary viewpoint from a plurality of images.
Specifically, the free viewpoint image generation unit 113 uses the image captured by the image capturing device 111-1 and the image captured by the image capturing device 111-2 to perform each point on the viewer P11 (hereinafter, referred to as “stereo matching method”). 3D coordinates are also calculated. At this time, the free viewpoint image generation unit 113 sets a point of interest on the image captured by the imaging device 111-1, and a point corresponding to the point of interest (hereinafter, referred to as the point of interest) on the image captured by the imaging device 111-2 Is also determined by the following method. First, the free viewpoint image generation unit 113 sets a corresponding point search range on the image captured by the imaging device 111-2 based on the positional relationship between the imaging devices 111-1 and 111-2. For each pixel in the search range, the free viewpoint image generation unit 113 associates the pixel and its surrounding pixels with the attention point and its surrounding pixels, and calculates the sum of the brightness differences of the associated pixels. calculate. The free viewpoint image generation unit 113 sets, as a corresponding point, a point having the smallest sum of brightness differences among the respective pixels within the search range. Note that the free viewpoint image generation unit 113 may determine the point of interest and the corresponding point using other methods such as extracting features such as shading edges from the image.
The free viewpoint image generation unit 113 calculates the three-dimensional coordinates of the attention point by the stereo matching method.

  Next, the free viewpoint image generation unit 113 constructs a three-dimensional model of the viewer P11 based on the calculated three-dimensional coordinates. Further, like the reference position selection unit 112, the free viewpoint image generation unit 113 detects the viewing direction of the viewer P11. The free viewpoint image generation unit 113 generates an image of the viewer P11 viewed from the viewing direction of the viewer P11 based on the constructed three-dimensional model. At that time, the free viewpoint image generation unit 113 generates an image by matching the reference position of the viewer P11 with the center of the image. The free viewpoint image generation unit 113 inputs the generated image to the image expansion / contraction unit 126 via the communication network 13 (FIG. 1).

FIG. 11 is a flowchart illustrating a processing procedure in which the free viewpoint image generation unit 113 generates an image of the viewer P11 viewed from the viewing direction of the viewer P11. When the video conference terminal device 11 is activated, the free viewpoint image generation unit 113 starts processing for generating an image of the viewer P11 viewed from the viewing direction of the viewer P11.
In step S <b> 1, the free viewpoint image generation unit 113 determines whether an image is input from the imaging unit 111. For example, the imaging unit 111 inputs an image in the form of frame data having a predetermined header to the free viewpoint image generation unit 113, and the free viewpoint image generation unit 113 determines that an image has been input when this header is detected. If it is determined that an image has been input (step S1: YES), the process proceeds to step S2. If it is determined that an image has not been input (step S1: NO), step S1 is repeated.
In steps S2 to S4, the free viewpoint image generation unit 113 calculates the position of each part of the viewer P11 by the stereo matching method. In step S <b> 2, the free viewpoint image generation unit 113 extracts an attention point that is received from the imaging unit 111 and is common to images captured by the imaging devices 111-1 and 111-2. The free viewpoint image generation unit 113 extracts attention points including points on the image of the viewer P11. In step S <b> 3, the free viewpoint image generation unit 113 calculates the parallax between the image captured by the imaging device 111-1 and the image captured by the imaging device 111-2 for each target point. In step S4, the free viewpoint image generation unit 113 calculates the three-dimensional coordinates of each point of interest using triangulation based on the calculated parallax.

  In step S5, the free viewpoint image generation unit 113 generates a 3D model of the viewer P11 based on the calculated 3D image. In step S6, the free viewpoint image generation unit 113 performs known texture mapping for giving a texture to the surface of the generated three-dimensional model. In step S7, the free viewpoint image generation unit 113 detects the viewing direction of the viewer P11. In step S <b> 8, the free viewpoint image generation unit 113 generates an image when captured from the viewing direction of the viewer P <b> 11 based on the three-dimensional model subjected to texture mapping. In step S <b> 9, the free viewpoint image generation unit 113 inputs the generated image to the image expansion / contraction unit 126. Thereafter, the process proceeds to step S1.

The reference position selection unit 112 inputs the coordinates of a point on the display unit 116 that the viewer P11 is interested in to the free viewpoint image generation unit 113, and the free viewpoint image generation unit 113 images the viewer P11 from this point. An image may be generated. Thereby, the process in which the free viewpoint image generation unit 113 detects the viewing direction of the viewer P11 can be reduced. Alternatively, the free viewpoint image generation unit 113 may generate an image viewed from the center of the display surface of the display unit 116. Thereby, when the viewer P11 pays attention to the vicinity of the center of the display surface, the free viewpoint image generation unit 113 can generate an image viewed from the viewing direction of the viewer P11.
In addition, you may make it an imaging device image the image from the center front of a display surface by the existing method using a half mirror. For example, the display unit 116 includes a half mirror on the display surface. The half mirror reflects an image projected by a projection unit installed below the display surface in the front direction of the display surface. Thereby, the display unit 116 displays an image on the display surface. The display unit 116 includes an imaging device in the center of the display surface and behind the half mirror, and the imaging device captures an image from the center front of the display surface. Thus, when the viewer P11 is looking at the display surface, the imaging device captures an image from the front. In the case of the existing method using the half mirror, the video conference system 1 may not include the free viewpoint image generation unit 113.

The image expansion / contraction unit 126 expands / contracts the image received from the free viewpoint image generation unit 113 so that the image when facing the display surface of the display unit 127 can be seen from the position of the viewer who is the talker. First, the image expansion / contraction unit 126, based on the reference position (xp, yp, zp) received from the reference position selection unit 112, the distance r between the origin O and the reference position P shown in FIG. An angle θ around the x axis and an angle φ around the y axis are calculated. The image expansion / contraction unit 126 calculates the angle θ with the clockwise direction as viewed from the positive side of the x axis as a positive angle, and calculates the angle φ with the counterclockwise direction as viewed from the positive side of the y axis as a positive angle.
The image expansion / contraction unit 126 calculates r, θ, and φ based on Expression (1). The reference position P is a position that is rotated by θ in the vertical direction and φ in the horizontal direction with respect to the origin O at a distance r.

Next, the image expansion / contraction unit 126 expands / contracts the image received from the free viewpoint image generation unit 113.
FIG. 12 is a diagram illustrating image expansion / contraction performed by the image expansion / contraction unit 126. This figure is a plan view of the display unit 127 as viewed from above. The origin O, the x axis, the z axis, the angle φ, and the reference position P are the same as those in FIG. The virtual display surface 127 ′ is a plane perpendicular to the straight line OP. FIG. 12 shows a case where there is no deviation in the y direction between the reference position P and the origin O, and the virtual display surface 127 ′ is shown by a straight line. The point Q is one point on the virtual display surface 127 ′, and the straight line 1 is a straight line passing through the point Q and the point P. Point R is an intersection of the straight line 1 and the display surface of the display unit 127.
The image expansion / contraction unit 126 is a case where the display unit 127 is viewed from the reference position P based on the distance r and angle φ between the center of the display surface of the display unit 127 and the reference position P and the size of the display surface of the display unit 127. The viewing angle α of the display unit 127 is calculated.

Next, the image expansion / contraction unit 126 calculates the shape of the virtual display surface 127 ′ including the origin O and perpendicular to the straight line OP. The virtual display surface 127 ′ is a display surface whose viewing angle matches that of the display surface of the display unit 127 when viewed from the reference position P.
The image expansion / contraction unit 126 compares the length of each side of the display surface of the display unit 127 with the length of each side of the virtual display surface 127 ′, and displays the display of the display unit 127 among the sides of the virtual display surface 127 ′. The side that is most magnified with respect to the face is determined. In the case of FIG. 12, the image expansion / contraction unit 126 determines that the side having the positive x coordinate value among the sides parallel to the y axis is the side that is most enlarged with respect to the display surface of the display unit 127.
The image expansion / contraction unit 126 expands the image received from the free viewpoint image generation unit 113 while maintaining the aspect ratio according to the determined enlargement ratio of the side to be enlarged. The image expansion / contraction unit 126 matches the origin O and the center of the enlarged image, and matches the direction of the most enlarged side of the virtual display surface 127 ′ with the corresponding side in the enlarged image to enlarge the image. The obtained image is associated with the virtual display surface 127 ′, and an image obtained by performing transmission projection conversion centered on the reference position P from the virtual display surface 127 ′ to the display surface of the display unit 127 with respect to the associated image. Is generated. Specifically, the image expansion / contraction unit 126 calculates the position of each pixel (pixel) of the image received from the free viewpoint image generation unit 113 that is associated on the virtual display surface 127 ′ by the above association. Further, the image expansion / contraction unit 126 calculates a position associated with the display surface of the display unit 127 by the following mapping for each pixel of the image received from the free viewpoint image generation unit 113. The image expansion / contraction unit 126 determines the pixels of the free viewpoint image generation unit 113 and the pixels of the display unit 127 based on the positions where the pixels of the image received from the free viewpoint image generation unit 113 are associated with the display surface of the display unit 127. Is associated. The image expansion / contraction unit 126 generates an image to be displayed by the display unit 127 based on the pixel association, and inputs the image to the display unit 127.

  In the figure, the virtual display surface 127 ′ passes through the origin O and is inclined at an angle φ with respect to the display surface of the display unit 127. Therefore, the virtual display surface 127 ′ is a straight line represented by Expression (2) in the xz plane in the drawing.

  A straight line connecting the point Q (q, −q (tan φ)) and the point P (r (sin φ), r (cos φ)) on the virtual display surface is expressed by Expression (3).

The intersection of this straight line and the display surface of the display unit 127 is a point to be projected. Since the intersection is an intersection between the equation (2) and z = 0, the projection point R is R (rq / (r (cos ² φ) + q (sin φ)), 0).
The image expansion / contraction unit 126 projects a free viewpoint image generation unit by projecting an image from the point Q (q, −q (tan φ)) to the point R (rq / (r (cos ² φ) + q (sin φ)), 0). The image received from 113 is expanded or contracted. Specifically, the pixel value of the pixel associated with the point Q in the image received from the free viewpoint image generation unit 113 is input to the display unit 127 as the pixel value on the display unit 127 associated with the point R. To do.
The above is the processing in the case of two dimensions (y = 0). Similarly, in the case of the three-dimensional case, the image expansion / contraction unit 126 generates an image when an image is projected from the virtual display surface 127 ′ onto the display surface of the display unit 127.

FIG. 13 is a flowchart illustrating a processing procedure in which the image expansion / contraction unit 126 performs image expansion / contraction. When the video conference terminal device 12 is activated, the image expansion / contraction unit 126 starts processing for expanding / contracting the image.
In step S <b> 21, the image expansion / contraction unit 126 determines whether an image is input from the free viewpoint image generation unit 113. If it is determined that an image has been input (step S21: YES), the process proceeds to step S22. If it is determined that no image is input (step S21: NO), step S21 is repeated.
In step S <b> 22, the image expansion / contraction unit 126 calculates the viewing angle of the display surface of the image display unit 127 when viewed from the reference position P. In step S23, the image expansion / contraction unit 126 calculates the outer shape of the virtual display surface 127 ′. In step S24, the image expansion / contraction unit 126 enlarges the image received from the free viewpoint image generation unit 113 based on the calculated outer shape of the virtual display surface 127 ′.
In step S <b> 25, the image expansion / contraction unit 126 generates an image when projected from the virtual display surface 127 ′ to the display unit 127 around the reference position P. In step S <b> 26, the image expansion / contraction unit 126 inputs the generated image to the display unit 127.

FIG. 14 is a diagram illustrating an example of an image output from the free viewpoint image generation unit 113 and an image in which the image expansion / contraction unit 126 performs image expansion / contraction.
The free viewpoint image generation unit 113 inputs an image obtained by viewing the viewer P11 from the front as shown in FIG. When the image expansion / contraction unit 126 expands / contracts the image and the image displayed by the display unit 127 is viewed from the front of the display unit 127, the image viewed from the front of the viewer P11 appears to be deformed as shown in FIG. When this is viewed from the position of the viewer who is a conversation person, an image of the viewer P11 seen from the front can be seen as shown in FIG. This figure is a view of the image calculated by the image expansion / contraction unit 126 based on the reference position of the viewer who is a conversation person as seen from the reference position on the right side toward the screen as described with reference to FIG. For this reason, the right side of the display surface appears longer than the left side.
As shown in FIG. 14C, an image of the viewer P11 as seen from the front can be seen from the position of the viewer who is the conversation person, so that the viewer who is the conversation person is the viewer P11 who is the other conversation person. You can have a conversation while looking at the images with the same line of sight.

As described above, in the video conference system 1, the free viewpoint image generation unit generates a front image, and when the image expansion / contraction unit 126 views the front image from the second viewer viewpoint position, the viewer P11 is viewed from the front. Convert the image so that it can be seen. Therefore, a conversation person who views the display unit 127 from the second viewer reference position can have a conversation while viewing an image whose line of sight coincides with the viewer P11 who is the other conversation person. In addition, the conversation person has a conversation while viewing a natural image that can be seen when the display unit 127 is viewed from the front instead of an image obtained by vertically deforming the viewer P11 when the display unit 127 is viewed from an oblique direction. be able to.
The arrangement of each part in the video conference system 1 is not limited to that in FIG. 4, and the reference position selection unit 112, the free viewpoint image generation unit 113, the reference position calculation unit 122, and the image expansion / contraction unit 126 are included in the video conference terminal device 11. And the video conference terminal device 12, or the video conference terminal device 11 and the video conference terminal device 12 may be included in different devices.
For example, the reference position calculation unit 122 may be included in the video conference terminal device 11 and receive an image from the imaging unit 121 via a communication network.

<Second Embodiment>
FIG. 15 is a system configuration diagram showing a schematic configuration of the video conference system 2 in the second embodiment of the present invention. In the figure, a video conference system 2 includes a video conference terminal device (first terminal device) 21, a video conference terminal device (second terminal device) 22, a video conference terminal device (third terminal device) 23, and a talker selection device. 24. The video conference terminal device 21, the video conference terminal device 22, the video conference terminal device 23, and the talker selection device 24 are connected to each other by a communication network. The video conference terminal device 21 includes an imaging unit (first viewer imaging unit) 211, a reference position calculation unit 212, a free viewpoint image generation unit 213, image expansion / contraction units 214 and 215, and display units 216 and 217. The The video conference terminal device 22 includes an imaging unit (second viewer imaging unit) 221, a reference position calculation unit 222, a free viewpoint image generation unit 223, image expansion / contraction units 224 and 225, and display units 226 and 227. The The video conference terminal device 23 includes an imaging unit (third viewer imaging unit) 231, a reference position calculation unit 232, a free viewpoint image generation unit 233, an image expansion / contraction unit (first image expansion / contraction unit) 234, and an image expansion / contraction unit (second An image expansion / contraction part) 235, a display part (first display part) 236, and a display part (second display part) 237 are configured. The talker selection device 24 includes a talker selection unit 241. The imaging unit 211 includes imaging devices 211-1 and 211-2. The imaging unit 221 includes the imaging devices 221-1 and 221-2. The imaging unit 231 includes the imaging devices 231-1 and 231-2.

The imaging unit 221 and the reference position calculation unit 222 correspond to the reference position output unit of the present invention, and the reference position output unit outputs the second viewer reference position that is the reference position of the conversation person. In the present embodiment, the image capturing unit 221 captures an image of a second viewer who is a conversation person, and the reference position calculating unit 222 calculates the reference position of the second viewer using this image. The second viewer reference position is detected and output.
The imaging unit 211 and the free viewpoint image generation unit 213 correspond to the image output unit of the present invention, and capture a first viewer described later and output a front image. In the present embodiment, the free viewpoint image generation unit 213 generates a front image using an image captured by the imaging unit 211, and outputs the generated front image, so that the image output unit outputs the front image. Note that the method by which the image output unit outputs the front image is not limited to the method described above, and other methods such as the same method as in the video conference system 1 may be used.

The imaging devices 211-1, 211-2, 221-1, 221-2, 231-1, and 231-2 are the same as the imaging device 111-1 and the like in FIG.
The reference position calculation units 212, 222, and 232 calculate the reference position in the same manner as the reference position calculation unit 122 in FIG. However, in the video conference system 2, when one viewer uses one video conference terminal device, the reference position calculation units 212, 222, and 232 each calculate one reference position. The case where a plurality of viewers use one video conference terminal device will be described later.
The free viewpoint image generation unit 213 generates an image of the viewer viewed from the viewing direction of the viewer based on the image captured by the imaging unit 211. In addition, the free viewpoint image generation unit 213 generates an image of the viewer viewed from a position moved from the viewer's line-of-sight direction. Details will be described later. The free viewpoint image generation units 223 and 233 are the same as the free viewpoint image generation unit 213.
The image expansion / contraction unit 214 expands / contracts an image to be displayed on the display unit 216 in the same manner as the image expansion / contraction unit 126 of FIG. The image expansion / contraction units 215, 224, 225, 234, and 235 are the same as the image expansion / contraction unit 214.
The display units 216, 217, 226, 227, 236, and 237 are the same as the display unit 127 of FIG.
The talker selection unit 241 selects a talker based on images input from the imaging units 211, 221, and 231. Details will be described later.

FIG. 16 is a diagram illustrating an example of a screen displayed by each display unit.
In FIG. 6A, the viewer P31 (first viewer) is looking at the display units 216 and 217 installed in the conference room R21. A viewer P32 (second viewer) is displayed on the display unit 216, and a viewer P33 (third viewer) is displayed on the display unit 217. In FIG. 5B, the viewer P32 is viewing the display units 226 and 227 installed in the conference room R22. The viewer P33 is displayed on the display unit 226, and the viewer P31 is displayed on the display unit 227. In FIG. 6C, the viewer P33 is viewing the display units 236 and 237 installed in the conference room R23. The viewer P31 is displayed on the display unit 236, and the viewer P32 is displayed on the display unit 237.
The figure is an example in which viewers P32 and P33 are conversational persons. In FIG. 6A, the display parts 216 and 217 are displayed so that viewers P32 and P33 face each other. On the other hand, as in the first embodiment, the image of the conversation person P33 is displayed on the display unit 226 in FIG. 7B so that the line of sight coincides with the conversation person P32. Similarly, the image of the conversation person P32 is displayed on the display unit 237 in FIG. 8C so that the line of sight coincides with the conversation person P33.
The two display units 216, 217, 226, 227, 236, and 237 of the video conference terminal devices 21, 22, and 23 may be formed using separate liquid crystal display devices, or one liquid crystal display. You may form so that the screen of an apparatus may be divided and displayed.

Next, the operation of the video conference system 2 will be described. Hereinafter, a case where the viewers P32 and P33 are talkers will be described. The same applies to other cases.
The imaging unit 211 captures a stereo image (first stereo image) of the viewer P31 and inputs the stereo image to the reference position calculation unit 212, the free viewpoint image generation unit 213, and the talker selection unit 241. The same applies to the imaging units 221 and 231. Hereinafter, the stereo image of the viewer P32 captured by the imaging unit 221 is also referred to as a second stereo image.
When the image of the viewer P31 is input from the imaging unit 211, the reference position calculation unit 212 calculates the reference position of the viewer P31 and inputs it to the image expansion / contraction units 214 and 215. Further, the reference position calculation unit 212 inputs the calculated reference position to the free viewpoint image generation unit 213. The same applies to the reference position calculation units 222 and 232.
The talker selection unit 241 detects mouth movements of the viewers P31, P32, and P33 from the images received from the imaging units 211, 221, and 231 via the communication network. Select a talker based on the frequency of Furthermore, the talker selection unit 241 selects the other talker by detecting the line of sight of the selected talker. For example, the talker selection unit 241 detects that the viewer P32 frequently opens and closes his / her mouth and selects the viewer P32 as a talker. Furthermore, the talker selection unit 241 detects the line of sight of the viewer P32, detects that the line of sight is on the display unit 226 side, and selects the viewer P33 as the conversation partner. The talker selection unit 241 inputs a signal indicating the selected talker to the free viewpoint image generation units 213, 223, and 233 via the communication network.
The talker selection device 24 may be provided attached to the video conference terminal device 21 and may be connected to the other video conference terminal devices 22 and 23 via a communication line.
Note that the present embodiment can be easily expanded when there are four or more video conference terminal devices. For example, the video conference system includes four video conference terminal devices, and each video conference terminal device includes three display units. The talker selection unit selects viewers of two video conference terminal devices among the four video conference terminal devices as the talker in the same manner as described above. In the same manner as described above, the video conference terminal device of the viewer who has not been selected as the conversation person displays images in which the conversation persons face each other.

FIG. 17 is a flowchart showing a processing procedure in which the talker selection unit 241 selects a talker.
The conversation selector 241 starts a process of selecting a conversation when the conversation selector 24 is activated.
In step S <b> 41, the talker selection unit 241 selects a talker based on images received from the imaging units 211, 221, and 231. In step S42, the talker selection unit 241 detects the line of sight of the selected talker. In step S43, the talker selection unit 241 selects a partner talker based on the detected line of sight. In step S <b> 44, the talker selection unit 241 inputs a signal indicating the selected conversation to the free viewpoint image generation units 213, 223, and 233. Then step S41 is repeated.

FIG. 18 is a data configuration diagram of data input by the talker selection unit 241 to the free viewpoint image generation units 213, 223, and 233.
In FIG. 9A, the two conversation parties selected by the conversation person selection unit 241 are indicated by terminal numbers. The terminal number is an identification number of the video conference terminal device used by the conversation person. Terminal number 1 indicates a video conference terminal device 31, terminal number 2 indicates a video conference terminal device 32, and terminal number 3 indicates a video conference terminal device 33. Terminal number 0 indicates that no conversation person is selected.
FIG. 5B shows data that the conversational selection unit 241 inputs to the free viewpoint image generation units 213, 223, and 233 when the conversational selection unit 241 has not selected a conversational party.
The talker selection unit 241 stores a threshold value for the frequency of moving the mouth in an internal storage unit (not shown), and determines that there is no viewer whose frequency of moving the mouth exceeds the threshold value. Do not select. In this case, the talker selection unit 241 inputs the data in FIG. 5B to the free viewpoint image generation units 213, 223, and 233.

  The talker selection unit 241 may select a talker by a method other than detecting mouth movement such as voice recognition. Note that the display units 216, 217, 226, 227, 236, and 237 are touch panels, and the talker selection unit 241 detects that the viewer has touched any position on the display unit. The conversation person and the other conversation person may be selected by a method other than the above, such as selecting a conversation person according to the displayed display unit.

When a signal indicating the selected conversation person is input from the conversation person selection section 241, the free viewpoint image generation section 213 generates an image of the viewer P 31 based on the input signal, and sends it to the image expansion / contraction sections 225 and 234. input.
Here, when the terminal number 2 is included in the signal received from the talker selection unit 241, the viewer P32 of the video conference terminal device 22 is selected as the talker. Therefore, in the relationship with the video conference terminal device 22, this signal is a signal indicating that it is a talker. In this case, like the free viewpoint image generation unit 113 of the first embodiment, the free viewpoint image generation unit 213 generates an image captured from the viewing direction of the viewer P31 and inputs the image to the image expansion / contraction unit 225.
On the other hand, when the terminal number 2 is not included in the signal received from the talker selection unit 241, the viewer P32 of the video conference terminal device 22 is not selected as a talker. Therefore, in the relationship with the video conference terminal device 22, this signal is a signal indicating that it is not a talker. In this case, the free viewpoint image generation unit 213 generates an image of the viewer P31 viewed from a position moved from the viewing direction of the viewer P31 as will be described later, and inputs the generated image to the image expansion / contraction unit 225 via the communication network. .
Similarly, when the terminal number 3 is included in the signal received from the talker selection unit 241, an image of the viewer P31 viewed from the viewing direction of the viewer P31 is generated and input to the image expansion / contraction unit 235. On the other hand, when the terminal number 3 is not included in the signal received from the talker selection unit 241, the free viewpoint image generation unit 213 generates an image of the viewer P31 viewed from the position moved from the viewing direction of the viewer P31. To the image expansion / contraction unit 235. The same applies to the free viewpoint image generation units 223 and 233.

FIG. 19 shows images generated by the free viewpoint image generation unit 223 and the free viewpoint image generation unit 233 when the viewer P31 is not a conversation person, that is, when the terminal number 1 is not included in the signal received from the conversation person selection unit 241. It is a figure which shows the angle of the shift | offset | difference of the viewpoint position of a viewer, and a viewer's gaze direction.
In the figure, point P indicates the reference position of the viewer P31, point Q indicates the reference position of the viewer P32 in the image of the display unit 216, and point R indicates the reference position of the viewer P33 in the image of the display unit 217. The free viewpoint image generation unit 223 generates an image with the reference position of the viewer P32 as the center of the image. Thereby, the point Q is the center of the display surface of the display unit 216. Similarly, the point R is the center of the display surface of the display unit 217. As for the angle of each vertex of the triangle PQR, the angle of the vertex Q is α, the angle of the vertex R is β, and the angle of the vertex P is π−α−β. Here, π represents the circumference ratio.
The free viewpoint image generation unit 223 stores the position of the center of the display surface of the display unit 216 as the position of the point Q, and stores the position of the center of the display surface of the display unit 217 as the position of the point R. The position P is a reference position received from the reference position calculation unit 212.
The free viewpoint image generation unit 223 detects the direction from the point R to the point P with the point Q as the center, and the viewer P32 viewed from the position moved from the viewing direction of the viewer P32 by this amount. Generate an image.
Similarly, the free viewpoint image generation unit 233 detects the direction from the point Q to the point P with the point R as the center, and viewing from the position moved from the viewing direction of the viewer P33 by this amount. An image of the person P33 is generated.

FIG. 20 is a diagram illustrating the imaging positions of images generated by the free viewpoint image generation units 223 and 233. As shown in FIG. 19A, the free viewpoint image generation unit 223 moves in the left direction of the figure by the angle α from the viewing direction of the viewer P32, that is, from the direction of the point R in FIG. An image of the viewer P32 viewed from the position is generated. This angle α is an angle that rotates from the point R to the point P around the point Q. As a result, the image of the viewer P32 facing the point P is rotated so as to face the point R. Similarly, as shown in FIG. 20B, the free viewpoint image generation unit 233 looks from the position where the imaging position is moved from the direction of the point Q to the direction of the point P from the viewing direction L33 of the viewer P33. An image of the viewer P33 is generated.
The free viewpoint image generation unit 223 detects the viewing direction L32 of the viewer P32 and the viewing direction L33 of the viewer P33, respectively, similarly to the reference position selection unit 112 of FIG. And the free viewpoint image generation part 223 synthesize | combines said image using the image which the imaging devices 221-1 and 221-2 of the imaging part 221 imaged similarly to the free viewpoint image generation part 113 of FIG. At this time, like the free viewpoint image generation unit 113, the free viewpoint image generation unit 223 generates an image by matching the reference position of the viewer P32 with the center of the image.

Note that, depending on the angle captured by the imaging units 211 to 231, an image when the imaging position is rotated as described above may not be generated. For example, when the imaging device 221-1 and the imaging device 221-2 of the imaging unit 221 capture an image from near the front of the viewer P32, if the angle α is large and close to the viewer's lateral direction, a three-dimensional model is generated. Necessary image data cannot be obtained and an image cannot be generated. In this case, the imaging unit 221 further includes an imaging device near the direction in which the image is generated. The free viewpoint image generation unit 223 selects two images according to the angle α from the images received from the imaging unit 221, generates a three-dimensional model using the selected images, and rotates the imaging position by the angle α. The above image is generated using this three-dimensional model.
Note that a camera may be installed at the above imaging position, and imaging may be performed using this camera. For example, a movable camera that images the viewer P32 is prepared. After calculating the imaging position, the free viewpoint image generation unit 223 controls to move to the imaging position calculated by the movable camera and perform imaging. As a result, it is not necessary for the free viewpoint image generation unit 223 to synthesize images, and the amount of calculation can be reduced.
The same applies to the free viewpoint image generation units 213 and 233.

When an image is input from the free viewpoint image generation unit 223, the image expansion / contraction unit 214 performs image expansion / contraction similarly to the image expansion / contraction unit 126 of FIG. Similarly, when an image is input from the free viewpoint image generation unit 233, the image expansion / contraction unit 215 expands / contracts the image.
FIG. 21 is a diagram illustrating image expansion / contraction performed by the image expansion / contraction units 214 and 215. As shown in FIG. 5A, the reference position P of the viewer P31 is at a position rotated by an angle θ from the direction facing the display unit 216. Therefore, similarly to the first embodiment, the free viewpoint image generation unit 223 performs transmission projection conversion centered on the reference position P from the virtual display surface 216 ′ passing through the point Q and perpendicular to the straight line PQ to the display unit 216. The image received from the free viewpoint image generation unit 223 is expanded and contracted by calculating an image performed on the image received from the image. The image expansion / contraction unit 214 inputs the calculated image to the display unit 216.
Similarly, the image expansion / contraction unit 215 performs transmissive projection conversion around the reference position P from the virtual display surface 217 ′ to the display unit 217, as shown in FIG. The image performed on the image received from is calculated. The image expansion / contraction unit 215 inputs the calculated image to the display unit 217.
Display unit 216 displays an image received from image expansion / contraction unit 214, and display unit 217 displays an image received from image expansion / contraction unit 215.

As described above, the display unit 216 displays an image that looks as if the viewer P32 is viewing the viewer P33 displayed on the display unit 217. Similarly, the display unit 217 displays an image that looks like the viewer P33 is viewing the viewer P32 displayed on the display unit 216. Therefore, the viewer P31 can see a more natural image that looks like the line of sight of the two viewers P32 and P33 having a conversation. In the video conference system 2, when the conversation person changes, the conversation person selection unit 241 selects a new conversation person, so that an image can be displayed so that the lines of sight of the conversational audiences match. .
Further, since the video conference system 2 displays an image when viewed from the line of sight to the conversation person, the conversation person displays an image whose line of sight matches that of the other conversation person as in the case of the video conference system 1. You can talk while watching.
The arrangement of each part of the video conference system 2 is not limited to that shown in FIG. 15, but the reference position calculation units 212, 222, and 232, the free viewpoint image generation units 213, 223, and 233, and the image expansion / contraction units 214, 215, and 224. 225, 234, and 235 may be included in any of the video conference terminal devices 21, 22, 23, and the conversational selection device 24, or may be included in devices other than these. . For example, the talker selection unit 241 may be included in the video conference terminal device 21. This eliminates the need for the video conference system 2 to include the talker selection device 24.

A single video conference terminal device may be used by a plurality of viewers. For example, when the viewers A and B use the video conference terminal device 21, the viewers C and D use the video conference terminal device 22, and the viewers E and F use the video conference terminal device 23, the viewers When C and the viewer E are conversations, the display unit 216 displays the image of the viewer C, and the display unit 217 displays the image of the viewer E.
Specifically, the imaging unit 211 captures an image including all viewers who use the video conference terminal device 21. The same applies to the imaging units 221 and 231. The reference position calculation unit 212 calculates each reference position of the viewer included in the image input from the imaging unit 211, and associates the calculated reference position with the coordinates on the display unit as described with reference to FIG. In addition, in addition to the free viewpoint image generation unit 213 and the image expansion / contraction units 214 and 215, the input is also made to the talker selection unit 241. The same applies to the reference position calculation units 222 and 232.

In the case where one viewer uses the above-described one video conference terminal device among all viewers included in the images input from the imaging units 211, 221, and 231. Similarly, a conversation person is selected based on the movement of the viewer's mouth. And the conversation person selection part 241 detects a conversation person's gaze direction, and calculates the coordinate on the screen which a conversation person pays attention to. The talker selection unit 241 sets the position on the screen to which the talker pays attention based on the information that is input from the reference position calculation units 212, 222, and 232 and associates the reference position with the coordinates on the display unit. Select the viewer to be displayed as the conversation partner. The talker selection unit 241 further associates the selected talker with information that is input from the reference position calculation unit and associates the reference position with the coordinates on the display unit. This association is performed by, for example, determining whether or not the person is a conversation person as shown in FIG. This is done by adding a flag to indicate.
The talker selection unit 241 inputs the generated information related to all the video conference terminal devices to the free viewpoint image generation units 213, 223, and 233.

The free viewpoint image generation unit 213 generates a three-dimensional model for all viewers included in the stereo image input from the imaging unit 211.
And in the information input from the conversation person selection part 241, when the conversation person is not contained in the viewer of the video conference terminal device 21, all the viewers who generated the three-dimensional model face the front. Generate and output. At this time, for example, based on the reference position, the viewer is arranged on the screen according to the actual position of the viewer. The same applies to the free viewpoint image generation units 223 and 233.
On the other hand, the free viewpoint image generation units 213, 223, and 233 use the video conference terminal device including the free viewpoint image generation unit (for example, the video conference terminal device 21 in the case of the free viewpoint image generation unit 213). When a viewer includes a conversation person, an image of the conversation person viewed from the front is input to a video conference terminal apparatus including the other conversation person, and the conversation person is not included. , The viewer's image viewed from a position moved from the direction of the conversation's line of sight is input so as to appear to face the other party's conversation.

For example, when the viewer C who uses the video conference terminal device 22 and the viewer E who uses the video conference terminal device 23 are talkers, the free viewpoint image generation unit 223 sends the video conference terminal device 23 to the video conference terminal device 23. Inputs an image of the viewer C viewed from the front. On the other hand, with respect to the video conference terminal device 21, from the position moved from the viewing direction of the viewer C, similarly to the viewer P32 when one viewer uses the one video conference terminal device described above. An image of the viewed viewer C is generated.
Similarly, the free viewpoint image generation unit 233 looks from the position moved from the viewing direction of the viewer E, similarly to the viewer P33 when one viewer uses the one video conference terminal device. An image of the viewer E is generated.
As described above, the free viewpoint image generation unit 223 and the free viewpoint image generation unit 233 are configured so that the viewer C and the viewer E can be viewed from the viewer A and the viewer B, respectively. An image obtained by moving the imaging positions of C and viewer E is generated and output. Here, the reference position calculation unit 212 further calculates a midpoint between the viewpoint position of the viewer A and the viewpoint position of the viewer B as a reference position, and inputs it to the free viewpoint image generation unit 223 and the free viewpoint image generation unit 233. To do. The free viewpoint image generation unit 223 and the free viewpoint image generation unit 233 generate the viewer C image and the viewer E image based on the reference position received from the reference position calculation unit 212.
As a result, an image in which viewer C and viewer E appear to face each other when viewed from the midpoint between the reference position of viewer A and the reference position of viewer B is displayed. Since viewer A and viewer B are considered to be viewing display unit 216 and display unit 217 from a position relatively close to this midpoint, viewer A and viewer B are viewer C and viewer C. It is possible to see a natural image where the viewer E faces each other. The same applies when other viewers are conversational.

Alternatively, in the above case where a plurality of viewers use one video conference terminal device, the display unit 216 and the display unit 217 display different images when the screen is viewed from the right and when viewed from the left. You may make it comprise including the liquid crystal display to display. A liquid crystal display that displays different images when viewed from the right and when viewed from the left, for example, in a car navigation system, a display that displays different images when viewed from the driver's seat and when viewed from the passenger seat Has been put to practical use.
In this case, the reference position calculation unit 212 calculates the reference position of the viewer A and the reference position of the viewer B, and outputs them to the free viewpoint image generation unit 223 and the image generation unit 233 freely. The free viewpoint image generation unit 223 and the free viewpoint image generation unit 233 are viewed from the reference position of the viewer B and the image that the viewer C and the viewer E face each other when viewed from the reference position of the viewer A. In such a case, an image that the viewer C and the viewer E see to face each other is generated and output to the display unit 216 and the display unit 217, respectively. The display unit 216 and the display unit 217 are based on the reference position of the viewer A and the reference position of the viewer B. For the viewer A, when viewed from the reference position of the viewer A, the viewer C And an image that viewer E looks to face each other, and viewer B displays an image that viewer C and viewer E seem to face each other when viewed from the reference position of viewer B. Thereby, the viewer A and the viewer B can see a natural image in which the viewer C and the viewer E face each other. The same applies when other viewers are conversational.
The display units 216, 217, 226, 227, 236, and 237 may include a liquid crystal display that displays different images in three or more directions. As a result, a natural image can be displayed for a larger number of viewers than in the case where different images are displayed in two directions.

<Third Embodiment>
FIG. 22 is a system configuration diagram showing a schematic configuration of the video conference system 3 in the third embodiment of the present invention. In the figure, the video conference system 3 includes a video conference terminal device (first terminal device) 31 and a video conference terminal device (second terminal device) 32. The video conference terminal device 31 and the video conference terminal device 32 are connected to each other by a communication network. The video conference terminal device 31 includes an imaging unit (first imaging unit) 311, a reference position selection unit 312, a free viewpoint image generation unit 313, and a display unit (second display unit) 316. The video conference terminal device 32 includes an imaging unit (second imaging unit) 321, a reference position calculation unit 322, image expansion / contraction units 326 and 327, and a display unit 328. The imaging unit 311 includes the imaging devices 311-1 and 311-2. The imaging unit 321 includes an imaging device 321-1 and an imaging device 321-2.

The imaging unit 321 and the reference position calculation unit 322 correspond to the reference position detection unit of the present invention, and detect a reference position for one or more viewers including a second viewer who is a talker. In the present embodiment, the reference position calculation unit 322 calculates the reference position using the image captured by the imaging unit 321, so that the reference position detection unit detects the reference position. Note that the method of detecting the reference position by the reference position detection unit is not limited to the method described above, and other methods such as the same method as in the video conference system 1 may be used.
The reference position detection unit and the reference position selection unit 312 correspond to the reference position output unit of the present invention, and output a second viewer reference position that is a reference position of the second viewer. In the present embodiment, the second viewer reference position is detected by the reference position selection unit 312 selecting the reference position of the second viewer who is the talker from the reference positions detected by the reference position detection unit. .
In addition, the imaging unit 311 and the free viewpoint image generation unit 313 correspond to the image output unit of the present invention, image the first viewer, and output a front image. In the present embodiment, the free viewpoint image generation unit 313 generates a front image using the image captured by the imaging unit 311 and outputs the generated front image, so that the image output unit outputs the front image. Note that the method by which the image output unit outputs the front image is not limited to the method described above, and other methods such as the same method as in the video conference system 1 may be used.
Further, the image expansion / contraction units 326 and 327 correspond to the image expansion / contraction unit of the present invention, and transmission projection centered on the second viewer reference position from the virtual display surface facing the second viewer reference position to the display surface. An image obtained by performing conversion on the front image or an image obtained by rotating an imaging position described later is calculated. In the present embodiment, when a front image or an image obtained by rotating an imaging position and a second viewer reference position are input to the image expansion / contraction unit 326 or the image expansion / contraction unit 327, the image expansion / contraction unit 326 or the image expansion / contraction unit 327 performs transmission projection conversion centered on the second viewer reference position from the virtual display surface facing the second viewer reference position to the display surface with respect to the front image or the image obtained by rotating the imaging position. Calculate the image.

The display unit 316 is the same as the display unit 116 of FIG.
The display unit 328 includes a liquid crystal display that displays different images when the screen is viewed from the right and when viewed from the left. The imaging unit 311 captures a stereo image (first stereo image) of the viewer P41 (first viewer), and the imaging unit 321 includes stereo images (second stereo) of the viewer P51 (second viewer) and the viewer P52. Image). The imaging devices 311-1, 311-2, 321-1, and 321-2 are the same as the imaging device 111-1 and the like in FIG.
The free viewpoint image generation unit 313 generates an image of the viewer viewed from the viewing direction of the viewer based on the image captured by the imaging unit 311. In addition, the free viewpoint image generation unit 312 generates an image of the viewer viewed from a position moved from the viewer's line-of-sight direction. Details will be described later.
The reference position calculation unit 322 calculates the reference position in the same manner as the reference position calculation unit 122 in FIG.
The image expansion / contraction unit 326 is associated with the left direction toward the display unit 328 among the directions in which the display unit 328 displays an image. It expands and contracts in the same way. The image expansion / contraction unit 327 is associated with the display unit 328 in the right direction among the directions in which the display unit 328 displays an image. It expands and contracts in the same way.

FIG. 23 is a diagram illustrating an example of an image displayed by the display unit 328. In the figure, viewers P51 and P52 are viewing the display unit 328 installed in the conference room R32. In addition, viewers P41 and P51 are talkers. The display unit 328 in FIG. 6A displays an image when the display unit 328 is viewed from the left. The viewer P51 is a conversation person, and the viewer P41 is displayed on the display unit 328 so that the line of sight coincides with the viewer P51. Further, an image when the display unit is viewed from the right is displayed on the display unit 328 of FIG. The display unit 328 displays an image in which the viewer P41 is looking in the direction of the viewer P51.
FIG. 24 is a diagram illustrating an example of an image displayed on the display unit 316. In the figure, the viewer P41 is viewing the display unit 316 installed in the conference room R31.

Next, the operation of the video conference system 3 will be described. Below, the case where the viewer P41 and the viewer P51 are talkers is demonstrated. The same applies when the viewer P41 and the viewer P52 are talkers.
The imaging unit 321 captures images of the viewers P51 and P52, inputs the images to the reference position calculation unit 322, and inputs the images to the display unit 316 via the communication network. Display unit 316 displays an image received from imaging unit 321.
The reference position calculation unit 322 calculates the reference positions of the viewers P51 and P52 based on the image received from the imaging unit 321 similarly to the reference position calculation unit 122 in FIG. The reference position calculation unit 322 inputs the calculated reference position to the reference position selection unit 312.
The imaging unit 311 captures an image of the viewer P51 and inputs it to the reference position selection unit 312 and the free viewpoint image generation unit 313.
The reference position selection unit 312 selects one reference position corresponding to the talker from the reference positions received from the reference position calculation unit 322 via the communication network. Similarly to the reference position selection unit 112 in FIG. 1, the reference position selection unit 312 detects the line of sight of the viewer P41 and selects the reference position corresponding to the talker. The reference position selection unit 312 adds a flag indicating the selected reference position to the reference position received from the reference position calculation unit 322. Furthermore, the reference position selection unit 312 stores in advance the direction in which the display unit 328 displays an image, and associates the direction in which the display unit 328 displays an image with the reference position. The reference position selection unit 312 inputs a reference position associated with the direction in which the display unit 328 displays an image to the free viewpoint image generation unit 313.
At this time, when a plurality of reference positions can be associated with one direction, the reference position selection unit 312 inputs only one reference position to the free viewpoint image generation unit 313 in association with this direction. . When the reference position of the talker is included in these reference positions, only the reference position of the talker is input to the free viewpoint image generation unit 313 in association with this direction. When the reference position of the talker is not included, the free viewpoint image generation unit 313, for example, among the reference positions associated with the same direction, only the reference position that is input first from the reference position calculation unit 322, The free viewpoint image generation unit 313 is input in association with this direction.
The reference position selection unit 312 inputs the reference position input to the free viewpoint image generation unit 313 to the image expansion / contraction units 326 and 327. The reference position selection unit 312 inputs the reference position associated with the left direction toward the display unit 328 to the image expansion / contraction unit 326 according to the direction in which the display unit 328 displays the image, and the right side toward the display unit 328. A reference position associated with the direction is input to the image expansion / contraction unit 327.

FIG. 25 is a data configuration diagram illustrating a configuration of data input to the free viewpoint image generation unit 313 by the reference position selection unit 312.
In the figure, the data input by the reference position selection unit 312 to the free viewpoint image generation unit 313 includes the reference position of two viewers, the coordinates on the display unit, and a flag indicating the selected talker. The The reference position and the coordinates on the display unit are the same as the reference position and the coordinates on the display unit in FIG. The flag indicating the conversation person indicates the conversation person selected by the reference position selection unit 312 by the flag value “1”, and indicates the conversation person other than the selection by the reference position selection part 312 by the flag value “0”. .
Also, in the figure, “direction 1” indicates the left direction toward the display unit 328 among the directions in which the display unit 328 displays an image, and “direction 2” indicates the direction in which the display unit 328 displays an image. Among these, the direction on the right side toward the display unit 328 is shown. “Viewer 1” is associated with “direction 1”, and “viewer 2” is associated with “direction 2”.

Based on the data received from the reference position selection unit 312, the free viewpoint image generation unit 313 determines the reference position of the viewer located on the left side of the display surface front direction of the display unit 328 and the reference position of the viewer located on the right side. Distinguish.
The free viewpoint image generation unit 313 generates an image of the viewer P41 viewed from the viewing direction of the viewer P41 with respect to the reference position of the conversation person. For the reference positions of viewers other than the talker, the free viewpoint image generation unit 313 generates an image of the viewer P41 viewed from the position moved from the viewing direction of the viewer P41. Details will be described later. Of the generated images, the free viewpoint image generation unit 313 inputs an image corresponding to the reference position of the viewer located on the left side in the front direction of the display surface of the display unit 328 to the image expansion / contraction unit 326 via the communication network. An image corresponding to the reference position of the viewer located at is input to the image expansion / contraction unit 327 via the communication network. As described above, the free viewpoint image generation unit 313 associates the image with the display direction by inputting the image to the image expansion / contraction unit corresponding to the image display direction.

FIG. 26 is a diagram illustrating an angle at which the free viewpoint image generation unit 313 rotates the imaging position when the reference position selection unit 312 selects the viewer P51 as a talker.
In the figure, point S indicates the reference position of the viewer P52, point T indicates the reference position of the viewer P41 in the image of the display unit 328, and point U indicates the reference position of the viewer P51. The free viewpoint image generation unit 313 generates an image obtained when the imaging position is rotated from the direction of the point S to the direction of the point U by the angle γ in FIG.
Thereby, the viewer P52 sees the image of the viewer P41 in line of sight with the viewer P51.

Thus, since the video conference system 3 displays the image of the viewer P41 who is in line of sight with the conversation person P51 for the viewer P52 who is not a conversation person, the viewer P52 has a line-of-sight direction between the conversation persons. You can join the meeting while looking at the images that match nature. Further, for the viewer P51 who is a conversation person, an image whose line of sight matches the conversation person P41 is displayed, so the conversation person P51 talks more naturally while viewing the image whose line of sight matches the conversation person P41. I can do it.
Note that the liquid crystal display provided with the display unit 328 is not limited to displaying different images in two directions, and may display different images in three or more directions. In this case, the video conference terminal device 32 is configured to include the same number of image expansion / contraction units as the number of directions in which different images are displayed on the liquid crystal display, and the free viewpoint image generation unit 313 is a viewer corresponding to each image expansion / contraction unit. An image of P41 is generated and input to each image expansion / contraction unit. In this way, by displaying different images in more directions, more natural images can be displayed for more viewers.
The arrangement of each part in the video conference system 3 is not limited to that shown in FIG. 22, and the reference position selection unit 312, the free viewpoint image generation unit 313, the reference position calculation unit 322, and the image expansion / contraction units 326 and 327 are included in the video conference terminal. Either the device 31 or the video conference terminal device 32 may be included, or the video conference terminal device 31 and the video conference terminal device 32 may be included in different devices.
For example, the reference position calculation unit 322 may be included in the video conference terminal device 31 and receive an image from the imaging unit 321 via a communication network.

In addition, this invention can also be implemented with the following aspect.
(1) A first viewer and a second terminal device in a first conference room provided with a first terminal device and a second terminal device for video conference, which are connected to each other via a communication network. Is a communication system that displays each other's video to the second viewer in the second conference room where the first terminal device is configured to capture the first viewer, and the first imaging unit captures the first viewer. A free viewpoint image generation unit that generates an image of the first viewer based on the captured image and a first display unit that displays the second viewer, and the second terminal device captures the second viewer An image generated by the free viewpoint image generation unit and received via the communication network, and generating an image of the first viewer facing the second viewer's line of sight by expanding and contracting the image The expansion / contraction part and the image expansion part generated by the image expansion / contraction part Including a second display unit that displays an image of the first viewer facing the viewing direction of the two viewers, and transmitting the image captured by the second imaging device to the first terminal device via the communication network, A characteristic communication system.
In this communication system, since the second display unit displays the image of the first viewer facing the second viewer's line of sight, the second viewer can view the image whose line of sight matches the first viewer. You can have a conversation with the first viewer.

(2) There are a plurality of viewers in the second conference room, and a second viewer who is one of the viewers speaks by paying attention to the first viewer displayed on the second display unit. The communication system according to (1) above.
In this communication system, since the second display unit displays the image of the first viewer facing the second viewer's line of sight, the second viewer pays attention to the image whose line of sight matches the first viewer. To talk to the first viewer and speak to the first viewer.

(3) The first imaging unit includes a plurality of imaging devices, and the free viewpoint image generation unit generates a three-dimensional image of the first viewer based on the image captured by the first imaging unit. The communication system according to (1) or (2), characterized in that it is characterized in that
In this communication system, the free viewpoint image generation unit generates a three-dimensional image of the first viewer based on images captured by a plurality of imaging devices. Accordingly, as described above, the second viewer can have a conversation with the first viewer while viewing an image whose line of sight matches the first viewer.

(4) A first viewer and a second terminal device of a first conference room provided with a first terminal device and a second terminal device for video conference, which are connected to each other via a communication network. Is a communication system that displays images of each other to the second viewer in the second conference room where the first terminal device captures the first viewer, and the first image capturing unit captures the first viewer. A free viewpoint image generation unit that generates an image viewed from the line of sight of the first viewer based on the image and a first display unit that displays the second viewer, and the second terminal device displays the second viewer The display surface for displaying the image generated by the second imaging unit for imaging and the free viewpoint image generation unit and received via the communication network displays different images when viewed from the right and when viewed from the left. A second display unit, A communication system, wherein the imaged image is transmitted to the first terminal device via the communication network, and a different image is displayed on the second display unit corresponding to the viewer.
In this communication system, different images corresponding to viewers are displayed on the second display unit. Therefore, for viewers who are conversational users, images of the first viewer facing the viewer's line of sight By displaying, a viewer who is a conversation person can have a conversation with the first viewer while viewing an image whose line of sight matches that of the first viewer. For viewers who are not conversational users, it is possible to display a more natural image in which the lines of sight of the conversational persons match.

(5) A videoconferencing first terminal device, a second terminal device, and a third terminal device connected to each other via a communication network, and each viewer's video in each conference room in which each terminal device is placed Is connected to each conference room via a communication network, and includes a talker selection device that selects a talker from viewers of each conference room, and each terminal device includes The free viewpoint of each conference room, comprising: an imaging unit that images viewers in a conference room, a free viewpoint image generation unit that generates images from the imaging unit, and a display unit that displays images of viewers in other conference rooms A communication system, wherein the image generation unit generates a rotated viewer image as an image to be sent to a display unit in a conference room of a viewer who is not a talker.
In this communication system, a non-conversational viewer can generate a rotated viewer image and display a more natural image in which the conversing viewers face each other.

(6) A viewer display method for a communication system including a display unit including a display surface for displaying an image, imaging the first viewer, and outputting an image obtained by viewing the first viewer from a substantially front side. For an image output step, a reference position output step for detecting a second viewer reference position that is a position of a viewpoint of a second viewer who talks with the first viewer, and an image output by the image output step, Calculates an image obtained by performing perspective projection transformation centered on the second viewer reference position from the virtual display surface facing the straight line connecting the second viewer reference position and the center of the display surface to the display surface A viewer display method comprising: an image expansion / contraction step, and an image display step in which the display unit displays an image calculated by the image expansion / contraction unit.
In this viewer display method, since the image expansion / contraction step generates an image based on the second viewer reference position, which is the position of the conversation person's viewpoint, for the conversation person who views the display unit from the second viewer reference position. Even if this conversation person is not directly facing the display surface, it is possible to display an image whose line of sight matches that of the other conversation person. It is possible to display a natural image that can be seen when the other party's conversation person is seen from the front, instead of the image deformed into the shape.

(7) One person including an image output step of capturing an image of the first viewer on a computer and outputting an image of the first viewer viewed from substantially the front, and a second viewer having a conversation with the first viewer A viewer display program for executing a reference position selection step of selecting a second viewer reference position that is a reference position of the second viewer from a reference position that is a position of the viewer's viewpoint.
This program generates an image of the first viewer viewed from substantially the front, and selects the second viewer reference position, which is the position of the viewer's viewpoint, so that the image of the first viewer viewed from the front By converting the image that is visible when viewed from the front of the display surface so that it can be viewed from the second viewer reference position, even if the second viewer who is a conversation person is not facing the display surface For this conversation person, an image whose line of sight coincides with the other party's conversation person can be displayed, and the other person's conversation person who is seen when the display surface is viewed from an oblique direction is not an image that is deformed vertically. It is possible to display a natural image that can be seen when the other conversation person is viewed from the front.

(8) One or more viewers including a second viewer who talks to a reference position, which is the position of the viewer's viewpoint, on a computer having a display unit having a display surface for displaying images. A reference position detecting unit step for detecting the first viewer, and connecting the second viewer reference position, which is the reference position of the second viewer, to the center of the display surface with respect to the image viewed from the front of the first viewer An image expansion / contraction step for calculating a perspective projection conversion centered on the second viewer reference position from the virtual display surface facing the straight line to the display surface, and the image calculated in the image expansion / contraction step A viewer display program for executing a display step of displaying on the display unit.
This program converts the projection from the virtual display surface facing the second viewer reference position to the display surface with the second viewer reference position as the viewpoint, into an image obtained by viewing the first viewer from substantially the front. Since it is applied and displayed, even if the second viewer who is a conversation person is not facing the display surface, the conversation person can display an image whose line of sight matches that of the other conversation person, In addition, it is possible to display a natural image that can be seen when the other party's conversation person is viewed from the front, instead of an image in which the other person's conversation person who is seen when the display surface is viewed from an oblique direction is deformed vertically.

A program for realizing all or part of the functions of the video conference systems 1 to 3 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed. The processing of each unit may be performed as necessary. Here, the “computer system” includes an OS and hardware such as peripheral devices.
Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the specific structure is not restricted to this embodiment, The design change etc. of the range which does not deviate from the summary of this invention are included.

  The present invention is suitable for use in a communication system.

1-3 Video conference system 11, 12, 21-23, 31, 32 Video conference terminal device 24 Conversation selection device 111, 121, 211-231, 311, 321 Imaging unit 122, 212, 222, 232, 322 Reference position Calculation unit 112, 312 Reference position selection unit 113, 213, 223, 233, 313 Free viewpoint image generation unit 116, 127, 216, 217, 226, 227, 236, 237, 316, 328 Display unit 126, 214, 215, 224, 225, 234, 235, 326, 327 Image expansion / contraction part 241 Conversation selection part

Claims (11)

An image output unit that captures an image of the first viewer and outputs an image of the first viewer viewed from substantially the front;
A reference position output unit for detecting a second viewer reference position that is a position of a viewpoint of a second viewer who talks with the first viewer;
A display unit including a display surface for displaying the image of the first viewer;
With respect to the image output from the image output unit, the second viewer reference position from the virtual display surface to the display surface facing the straight line connecting the second viewer reference position and the center of the display surface An image expansion / contraction unit that calculates an image subjected to perspective projection conversion centered on
Comprising
The display unit displays the image calculated by the image expansion / contraction unit.
A communication system characterized by this. The reference position output unit
A reference position detector that detects a reference position that is the position of the viewer's viewpoint for one or more viewers including the second viewer;
A reference position selection unit that selects the second viewer reference position from the reference positions detected by the reference position detection unit;
The communication system according to claim 1, further comprising: A communication system having a first terminal device and a second terminal device connected via a communication path,
The first terminal device includes the image output unit and the reference position selection unit,
The communication system according to claim 2, wherein the second terminal device includes the reference position detection unit, the display unit, and the image expansion / contraction unit. A second display unit having a second display surface for displaying an image of one or more viewers including the second viewer;
The image output unit includes:
A first imaging unit that images a first stereo image of the first viewer as a subject;
An image including the face of the first viewer is detected from the first stereo image, the line-of-sight direction of the first viewer is detected, and the face of the first viewer is detected based on the first stereo image. A free viewpoint image generation unit that generates and outputs an image including the face of the first viewer viewed from the line-of-sight direction.
Comprising
The reference position detector is
A second imaging unit that captures a second stereo image having the one or more viewers as subjects;
A reference position calculation unit for detecting a face of each of the one or more viewers or an image of a part thereof from the second stereo image and calculating a reference position of each of the one or more viewers;
Comprising
The reference position selection unit detects the viewing direction of the first viewer from the first stereo image, and the viewer displayed at the intersection of the viewing direction and the second display surface is the second viewer. Selecting the reference position of the viewer as the second viewer reference position from the reference positions calculated by the reference position calculation unit;
The communication system according to claim 2 or claim 3, wherein The display surface displays different images in at least two directions;
The reference position selection unit selects the second viewer reference position and associates the second viewer reference position with any one of display directions in which the display surface displays the different images.
The display unit displays an image input from the image expansion / contraction unit in a direction associated with the second viewer reference position;
The communication system according to claim 2 or claim 3, wherein A second display unit having a second display surface for displaying an image of one or more viewers including the second viewer;
The image output unit includes:
A first imaging unit that images a first stereo image of the first viewer as a subject;
An image including the face of the first viewer is detected from the first stereo image, the line-of-sight direction of the first viewer is detected, and the face of the first viewer is detected based on the first stereo image. A free viewpoint image generation unit that generates and outputs an image including the face of the first viewer viewed from the line-of-sight direction.
Comprising
The reference position detector is
A second imaging unit that captures a second stereo image having the one or more viewers as subjects;
A reference position calculation unit for detecting a face of each of the one or more viewers or an image of a part thereof from the second stereo image and calculating a reference position of each of the one or more viewers;
Comprising
The reference position selection unit detects the viewing direction of the first viewer from the first stereo image, and the viewer displayed at the intersection of the viewing direction and the second display surface is the second viewer. A reference position of the viewer is selected as the second viewer reference position from the reference positions calculated by the reference position calculation unit, and the center of the display surface and the position of the second viewer reference position Based on the relationship, associate any one of the display directions with the second viewer reference position,
The communication system according to claim 5. The display surface displays different images in at least two directions;
The reference position selection unit associates any of the reference positions with each of the display directions in which the display surface displays the different images.
The image output unit generates an image of the first viewer viewed from a position determined based on a reference position associated with a direction not associated with the second viewer reference position, Associating the image and outputting it to the image expansion / contraction unit,
For each of the input directions, the image expansion / contraction unit is a virtual that directly faces a straight line connecting the reference position associated with the direction and the center of the display surface with respect to the image associated with the direction. Calculating an image obtained by performing perspective projection transformation centered on a reference position associated with the direction from the display surface to the display surface, and outputting the calculated image in association with the direction;
The display unit displays an image input from the image expansion / contraction unit in a direction associated with the image;
The communication system according to claim 2 or claim 3, wherein A second display unit having a second display surface for displaying an image of one or more viewers including the second viewer;
The image output unit includes:
A first imaging unit that images a first stereo image of the first viewer as a subject;
An image including the face of the first viewer is detected from the first stereo image, the line-of-sight direction of the first viewer is detected, and the face of the first viewer is detected based on the first stereo image. A center of the display surface for each of the viewpoint positions excluding the second viewer reference position among the viewpoint positions associated with the display direction by the reference position selection unit , The direction from the viewpoint position to the second viewer reference position is detected, and an image viewed from the position moved in the detected direction from the line of sight of the first viewer is generated. A free viewpoint image generation unit that associates a display direction with which the position of the viewpoint is associated with the generated image and outputs the image to the image expansion / contraction unit;
Comprising
The reference position detector is
A second imaging unit that captures a second stereo image having the one or more viewers as subjects;
A reference position calculation unit that detects a face of the one or more viewers or a part of the image from the second stereo image, and calculates a reference position of each of the one or more viewers;
Comprising
The reference position selection unit detects the first viewer's line-of-sight direction from the first stereo image, and sets the viewer's reference position displayed at the intersection of the line-of-sight direction and the second display surface as the reference A second viewer reference position is selected from the reference positions calculated by the position calculation unit, and the second viewer reference position is selected based on the positional relationship between the center of the display surface and the second viewer reference position. One of the display directions is associated, and each of the display directions other than the direction associated with the second viewer reference position is set to any one of the reference positions other than the second viewer reference position as the display surface. Corresponding based on the positional relationship with each reference position,
The communication system according to claim 7. A communication system having a first terminal device used by a first viewer, a second terminal device used by a second viewer, and a third terminal device used by a third viewer, which are connected to each other via a communication channel. And
Further comprising a talker selection unit for selecting any two of the first viewer, the second viewer, and the third viewer as a talker,
The image output unit generates an image obtained by viewing the first viewer from substantially the front when the talker selection unit selects the first viewer and the second viewer, and the image expansion and contraction is performed. When the first viewer and the third viewer are selected, an image of the first viewer viewed from other than the front is generated and output to the image expansion / contraction unit.
The communication system according to claim 1. A communication system having a conversational selection device connected to the first terminal device, the second terminal device, and the third terminal device via a communication path,
The first terminal device includes the image output unit,
The second terminal device includes the reference position output unit, the display unit, and the image expansion / contraction unit,
The talker selection device includes the talker selection unit.
The communication system according to claim 9. Further comprising a third viewer imaging unit that images the third viewer;
The image output unit includes:
A first viewer imaging unit that captures a first stereo image with the first viewer as a subject;
An image including the face of the first viewer is detected from the first stereo image, the line-of-sight direction of the first viewer is detected, and the face of the first viewer is detected based on the first stereo image. A three-dimensional model including the first viewer and the second viewer when the conversational selection unit selects the first viewer and the second viewer. , When the talker selection unit selects the first viewer and the third viewer, the reference position from the center position of the second display surface, centered on the center position of the display surface The direction in the direction of the reference position output by the output unit is detected, and an image viewed from a position moved in the detected direction from the direction of the first viewer's line of sight is generated, and the first view viewed from other than the front A free viewpoint image generator that outputs an image of a viewer;
Comprising
The reference position output unit
A second viewer imaging unit that captures a second stereo image with the second viewer as a subject;
A reference position calculation unit for detecting the second viewer's face or a part of the image from the second stereo image and calculating the second viewer reference position;
Comprising
The talker selection unit includes the first viewer, the image captured by the second viewer imaging unit, and the image captured by the third viewer imaging unit, respectively. Detecting the frequency of mouth movement, the frequency of mouth movement of the second viewer, and the frequency of mouth movement of the third viewer, selecting a talker based on the detected frequency, and selecting the selected talker Detect the line-of-sight direction from the image, and select the other party's talker based on the detected line-of-sight direction,
The communication system according to claim 10.

Images