JP2009147792A - Communication apparatus with image, communication display method with image, program and communication system with image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a communication apparatus with an image capable of calculating and displaying an eye direction of a participant at a communication partner side only through processing at a receiving side, without providing a dedicated device or processing section at the communication partner side. <P>SOLUTION: A communication apparatus with an image comprises: a transmitting/receiving section 11 for receiving an image from a communication partner; a participant detecting section 13 and a gaze direction detecting section 14 for detecting, from the image received by the transmitting/receiving section 11, the face of a person and a gaze direction within the image; a display device position estimating section 15 for estimating a position of an image display device at a side of the communication partner from the image received by the transmitting/receiving section 11; a gaze direction difference calculating section 16 for calculating a difference between the gaze direction within the image and a direction of the position of the image display device at the communication partner side; a display section 3 for displaying thereon the image received by an image receiving means; and a gaze direction difference display section 17 for displaying on the display section 3 the difference calculated by the gaze direction difference calculating section 16 together with the received image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image-attached communication apparatus, an image-attached communication display method, a program, and an image-added communication system that encode an audio signal and an image signal and transmit them through a communication network.

  In recent years, with the development of video processing technology and communication technology, a remote communication environment through images such as a video conference, a WEB conference, and a video phone has been put into practical use. In the future, especially in the business field, the importance of smooth remote communication with images will inevitably increase as work style changes such as the spread of work and working hours and the spread of telework.

  Video conferencing is a technology that connects a plurality of remote locations via a communication network and transmits images and sounds in both directions. Either a circuit switching network or a connectionless packet switching network is used as the communication network. A video conference is also a kind of groupware set for a conference.

  In order to perform smooth communication as in the face-to-face in a remote communication environment, it is desirable that the line of sight of local participants and remote participants match. However, in reality, the camera and the monitor cannot be placed on the same line due to physical restrictions, and the line of sight may be shifted. As a technique for solving this, various methods for realizing line-of-sight matching by a half mirror, a composite video, a character display, and the like have been proposed (Patent Document 1, Patent Document 2, Patent Document 3, and Patent Document 4).

On the other hand, in the remote communication system, even if information on which direction the remote participant is gazing at can be obtained even if the line of sight does not match, it will help the conversation. As a technique for obtaining this information, techniques for extracting and displaying remote line-of-sight information have been proposed (Patent Documents 5 and 6). Here, the gaze detection device provided on the remote side detects the gaze direction of the participant, and the information is displayed on the local side.
Japanese Patent Laid-Open No. 04-154289 JP 05-227528 A JP 05-284493 A JP-A-2005-218103 JP 2003-244669 A JP 2005-340974 A

  The technology to match the line of sight of local and remote participants in related technologies requires various special equipment and a lot of calculations and is not easy to use in a real conference system .

  The first problem of the related technology is that it is difficult to apply to remote communication with an arbitrary point. The reason is that a dedicated device or processing unit is required on the remote side for line-of-sight extraction. In general, remote communication is not performed only with a specific point, and it is difficult to prepare dedicated devices and processing units in advance on all remote sides.

  The second problem is that it is not easy to know whether the remote participant is looking at the local speaker. This is because the viewing direction of the participant on the remote side is displayed as it is.

  The third problem is that it is difficult to know where the entire participant is gazing if the viewing direction of many participants is displayed as it is when there are a plurality of remote participants. The reason is that there is no means for integrating the gaze directions of a plurality of participants.

  The present invention has been made in view of the above-described problems, and its purpose is to provide a line of sight of a participant on the communication partner side only by processing on the reception side without providing a dedicated device or processing unit on the communication partner side. An object of the present invention is to provide a communication device with an image that can calculate and display a direction.

  In the present invention, the communication device with an image or the communication system with an image includes communication in general with images called by various names such as a videophone, a WEB conference, and a video chat in addition to the video conference.

  Another object of the present invention is to provide an image capable of displaying the line-of-sight direction of a remote participant on the local side in such a way that it can be immediately known whether or not the remote participant is looking at the local speaker. It is to provide an attached communication device. In addition, another object of the present invention is to display where a plurality of participants on the remote side are gazing at a plurality of participants on the remote side, thereby displaying where the entire participant on the remote side is gazing. An object of the present invention is to provide a communication device with an image that is easy for a local speaker to understand.

The communication device with an image according to the first aspect of the present invention is:
Image receiving means for receiving an image from a communication partner;
Detecting means for detecting a human face and a gaze direction in the image from the image received by the image receiving means;
Estimating means for estimating the position of the image display device on the communication partner side from the image received by the image receiving means;
Difference calculating means for calculating a difference between the gaze direction in the image and the direction of the position of the image display device on the communication partner side;
Image display means for displaying an image received by the image receiving means;
A difference display means for displaying the difference calculated by the difference calculation means on the image display means together with the image received by the image receiving means;
It is characterized by providing.

  Preferably, the estimation means estimates the position of the image display device on the communication partner side by observing a gaze direction in the image over a predetermined time.

  Alternatively, when the estimation unit detects an image display device on the communication partner side in the image, the estimation unit estimates the position as the position of the image display device.

  Alternatively, when the projection unit of the image display device on the communication partner side is detected in the image, the estimation unit estimates the position of the image display device on the communication partner side from the position and direction.

  Preferably, the difference display means classifies and displays the difference calculated by the difference calculation means by classifying the difference into either one of the direction of the image display device on the communication partner side and the other.

  Alternatively, the difference display means classifies and displays the difference calculated by the difference calculation means by classifying the difference into one of the direction of the image display device on the communication partner side and the right and left.

Preferably, when detecting a plurality of human faces in the image received by the image receiving unit, the detecting unit detects a gaze direction in the image for each person,
The difference calculation means is a composite gaze direction obtained by synthesizing a gaze direction passing through a point having a maximum number of gaze directions passing through the points among the points where the gaze directions of the plurality of persons intersect with the image display device. Calculate the difference from the direction of
The difference display means displays a difference between the synthetic gaze direction and the direction of the image display device.

The image-attached communication display method according to the second aspect of the present invention includes:
An image receiving step for receiving an image from a communication partner;
A detecting step for detecting a human face and a gaze direction in the image from the image received in the image receiving step;
An estimation step for estimating the position of the image display device on the communication partner side from the image received in the image reception step;
A difference calculating step for calculating a difference between the gaze direction in the image and the direction of the position of the image display device on the communication partner side;
A difference display step for displaying the difference calculated in the difference calculation step on the image display means together with the image received in the image reception step;
It is characterized by providing.

  Preferably, the estimating step estimates the position of the image display device on the communication partner side by observing a gaze direction in the image over a predetermined time.

  Alternatively, in the estimation step, when an image display device on the communication partner side is detected in the image, the position is estimated as the position of the image display device.

  Alternatively, in the estimating step, when the projection device of the image display device on the communication partner side is detected in the image, the position of the image display device on the communication partner side is estimated from the position and direction.

  Preferably, the difference display step classifies and displays the difference calculated in the difference calculation step in one of two directions, that is, a direction of the image display device on the communication partner side and the other.

  Alternatively, in the difference display step, the difference calculated in the difference calculation step is classified and displayed in one of three directions, that is, the direction of the image display device on the communication partner side and the right and left thereof.

Preferably, when the detection step detects a plurality of human faces in the image received in the image reception step, the gaze direction in the image is detected for each person,
The difference calculating step includes, for the point having the largest number of gaze directions passing through the points among the points where the gaze directions of the plurality of persons intersect, a synthesized gaze direction obtained by combining the gaze directions passing through the points and the image display device Calculate the difference from the direction of
The difference display step displays a difference between the synthetic gaze direction and the direction of the image display device.

A program according to a third aspect of the present invention provides a computer,
Image receiving means for receiving an image from a communication partner;
Detecting means for detecting a human face and a gaze direction in the image from the image received by the image receiving means;
Estimating means for estimating the position of the image display device on the communication partner side from the image received by the image receiving means;
Difference calculating means for calculating a difference between the gaze direction in the image and the direction of the position of the image display device on the communication partner side;
Image display means for displaying an image received by the image receiving means;
A difference display means for displaying the difference calculated by the difference calculation means on the image display means together with the image received by the image receiving means;
It is characterized by functioning.

The communication system with an image according to the fourth aspect of the present invention is:
Receiving means for receiving an image from a communication partner;
Image display means for displaying an image received by the receiving means;
A transmission means for transmitting an image of a participant to a communication partner via a network;
A first image-attached communication device comprising:
Image receiving means for receiving an image from the first communication device with an image via the network;
Detecting means for detecting a human face and a gaze direction in the image from the image received by the image receiving means;
Estimating means for estimating the position of the image display device on the communication partner side from the image received by the image receiving means;
Difference calculating means for calculating a difference between the gaze direction in the image and the direction of the position of the image display device on the communication partner side;
Image display means for displaying an image received by the image receiving means;
A difference display means for displaying the difference calculated by the difference calculation means on the image display means together with the image received by the image receiving means;
A second image-attached communication device comprising:
It is comprised from these.

  According to the communication apparatus with an image of the present invention, the line-of-sight direction of the participant on the communication partner side can be determined only by processing on the reception side without providing a dedicated device or processing unit for obtaining information on the line-of-sight direction on the communication partner side. Can be calculated and displayed. The reason is that the image-attached communication device on the receiving side calculates and displays the line-of-sight direction of the participant on the communication partner side.

  In the present invention, an embodiment will be described by taking a video conference apparatus and a video conference system as examples. The present invention can be applied to a communication device with an image such as a videophone, a WEB conference, and a video chat or a communication system with an image in addition to a video conference.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration example of a video conference apparatus according to Embodiment 1 as an example of the image-attached communication apparatus of the present invention. The video conference device 1 includes a control device 2, a display unit 3, and a camera 4.

  FIG. 2 is a block diagram illustrating a configuration example of a video conference system 100 in which the video conference apparatuses 1 according to Embodiment 1 are connected via a network. The video conference apparatuses 1A and 1B installed at the base A and the base B are connected to the network N. Each part of the video conferencing apparatus 1A at the site A is indicated with a reference symbol A. Each part of the video conferencing apparatus 1B at the base B is indicated by adding B to the reference symbol.

  The video conference system 100 transmits an image captured by the camera 4A at the site A to the video conference device 1B at the site B via the network N and displays the image on the display unit 3B. On the contrary, an image photographed by the camera 4B at the site B is displayed on the display unit 3A at the site A. Also, the audio signal input from the microphone 6A at the site A is transmitted to the video conference device 1B at the site B via the network N and output from the speaker 5B. Conversely, the audio signal input from the microphone 6B at the site B is transmitted to the video conference apparatus 1A at the site A via the network N and output from the speaker 5A.

  In the following, the focused base is sometimes referred to as the local side, and the base on the communication partner side is sometimes referred to as the remote side. That is, the base for explaining the gaze direction display of the present invention is referred to as a local side, and the partner side that transmits an image displayed with the gaze direction superimposed is referred to as a remote side.

  Referring to FIG. 1, the control device 2 of the video conference device 1 includes a transmission / reception unit 11, a communication processing unit 12, a participant detection unit 13, a gaze direction detection unit 14, a display device position estimation unit 15, and a gaze direction difference calculation unit 16. , A gaze direction difference display unit 17, an image output unit 18, and an image input unit 19. The control device 2 can be configured by a computer that operates by program control, for example.

  In the present invention, voice communication is not particularly limited. The voice communication may be transmitted together with the image, or may be communicated using another line such as a telephone line. In FIG. 1, the configuration related to voice communication is omitted.

  The camera 4 captures images of participants in the video conference and transmits them to the image input unit 19. The display unit 3 includes, for example, a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), an image projector device, or the like, and displays an image by a signal sent from the image output unit 18.

  The image input unit 19 inputs an image signal from the camera 4 and transmits it to the communication processing unit 12. The image input unit 19 may also encode the image signal and compress the data. The image output unit 18 inputs an image signal from the communication processing unit 12 and displays an image on the display unit 3. If the image signal is encoded and data compressed, the image signal is decoded.

  The communication processing unit 12 receives an image signal from the image input unit 19 and transmits the image signal from the transmission / reception unit 11 via the network N to the video conference device 1 as a communication partner. In addition, the image signal received from the video conference device 1 of the communication partner is sent to the participant detection unit 13 and the image output unit 18.

  The transmission / reception unit 11 includes a network termination device or a wireless communication device connected to the network N, and a serial interface or a LAN (Local Area Network) interface connected thereto. The transmission / reception unit 11 transmits an image signal to the video conference device 1 as a communication partner via the network N, and receives the image signal from the video conference device 1 as a communication partner.

  The participant detector 13 analyzes the remote image sent from the communication processor 12, identifies the remote participant in the image, and detects the position of the participant's face. The gaze direction detection unit 14 refers to the position of the participant's face detected by the participant detection unit 13, analyzes the image, and determines the apparent gaze direction in the image from the situation such as the face direction, the gaze direction, and the posture. Is detected.

The display device position estimation unit 15 estimates the position of the remote display unit 3 from the received image. As a method of estimating the position of the display unit 3,
(1) When the apparent line-of-sight direction of a participant is the same direction for a certain ratio within a certain time, it can be considered that there is an image display device in that direction. This is because, in general, remote communication participants often watch the other party's participant (display unit 3).

Or
(2) A display device displaying a local image such as a monitor or a screen displayed in the remote image is detected.
(3) An output auxiliary device such as a projector shown in the remote image is detected, and the position of the image display device displaying a local image such as a screen is estimated from the direction.
Etc. The position direction of the remote image display device corresponds to the front direction of the remote image.

  The gaze direction difference calculation unit 16 calculates a difference between the apparent gaze direction of the remote participant detected by the gaze direction detection unit 14 and the position direction of the remote display unit 3 detected by the display device position estimation unit 15. calculate. Based on this difference, the actual gaze direction of the remote participant with respect to the remote display unit 3 outputting the local image is determined.

  The actual gaze direction of the remote participant with respect to the remote display unit 3 is input to the gaze direction difference display unit 17. The gaze direction difference display unit 17 converts the actual gaze direction into a format to be displayed on the display unit 3 and inputs it to the image output unit 18. For example, an arrow is displayed in the image or around the image. The arrow represents the actual gaze direction of the remote participant with respect to the direction of the display unit 3 on the remote side as seen from the remote participant. When the remote participant is gazing at the display unit 3, it can be displayed by a point at the tip of an arrow or a circle surrounding it.

  The image output unit 18 outputs and displays an image on which an arrow indicating a gaze direction or the like is superimposed on the display unit 3. As a method of displaying the gaze direction, the received image may be superimposed on the received image in the form of an arrow or the like, or may be separately displayed with a dedicated instrument or a laser pointer.

  The gaze direction difference display unit 17 does not display the gaze direction of the remote participant calculated by the gaze direction difference calculation unit 16 on the display unit 3 in proportion to the direction and angle, but in several ranges. You may divide and classify and display. For example, the direction is displayed in two categories, that is, the direction of the local side, which is the speaker direction, or the other direction. Or it displays in three classifications, the front direction or the right or left. In those cases, for example, when the difference (absolute value) between the gaze direction and the direction of the display unit 3 exceeds a predetermined threshold value, it is determined that it is other than the front and is displayed. In the case of the three classifications of the front and its left and right, a positive or negative sign is attached to the angle at which the difference is projected on the plane, and when the absolute value exceeds a predetermined threshold, the right or left is distinguished and displayed according to the sign of the difference To do.

  As another method for classifying and displaying the gaze direction, there is a classification method in which the direction of the display unit 3, the direction of the material, and the other. In addition, when there are a plurality of participants on the local side, the direction of each participant (this can be estimated by the position and direction of the camera on the local side) and the other are displayed separately.

  In FIG. 2, the case where the remote side is one place is taken as an example, but there may be a plurality of remote sides. When there are a plurality of remote sides, the gaze direction of the participant is analyzed and displayed for each image received from the remote site. In addition, the participant detection unit 13, the gaze direction detection unit 14, the display device position estimation unit 15, the gaze direction difference calculation unit 16, and the gaze direction that analyze and display the gaze direction of the participant and the position of the display device from the received image. The difference display unit 17 and the like may be provided on the remote side.

  FIG. 3 is a plan view illustrating a method for determining the difference between the gaze direction of the participant and the display device direction. FIG. 3A shows an angle for determining the direction of gaze on the display unit 3B. FIG. 3B shows a state in which a direction deviated from the display unit 3B is watched as an example.

  When the apparent line-of-sight direction of the remote participant is the same direction for a certain ratio within a certain period of time, for example, the display unit 3B can be regarded as having the most viewing direction in a predetermined period, and the display The position of the part 3B is estimated. In FIG. 3A, the direction from the participant to the center of the display unit 3B is a display unit direction C0. The gaze direction is referred to as right or left when viewed from the display unit 3B. In FIG. 3A, directions C1 to C3 are right, and directions D1 to D3 are left.

  The directions C1 and D1 are directions when the both ends of the screen are actually viewed at the estimated positions of the both ends of the display unit 3B. An angle formed by the display unit direction C0 and the direction C1 is represented as an angle R1. Similarly, an angle formed by the display unit direction C0 and the direction D1 is represented as an angle L1. The angles R1 and L1 represent angles at which the display unit 3B is viewed. When the display unit 3B is not included in the image sent to the local side, the true values of the angle R1 and the angle L1 are unknown, but can be estimated from the frequency distribution in the gaze direction of the participant. Alternatively, a predetermined value may be set empirically.

  The directions C2 and D2 indicate threshold directions for determining the gaze direction. An angle formed by the display unit direction C0 and the right threshold value direction C2 is represented as an angle R2. Similarly, an angle formed by the display unit direction C0 and the left threshold value direction D2 is represented as an angle L2. The angle R2 and the angle L2 may not be equal. The angle R2 and the angle L2 that are threshold values for determining the gaze direction are normally set so as to include the angle R1 and the angle L1. When the display unit 3B is large, the angles R2 and L2 may be smaller than the angles R1 and L1.

  Directions C3 and D3 indicate the observed gaze directions of the participants. FIG. 3A shows a case where the difference from the display unit direction C0 is larger than the threshold direction C2 or D2. The angle formed by the display unit direction C0 and the gaze direction C3 is represented by an angle R3, and the angle formed by the display unit direction C0 and the gaze direction D3 is represented by an angle L3. FIG. 3B shows a case where the gaze direction of the participant is shifted to the right side when viewed from the display unit 3B. At this time, the gaze direction difference display unit 17 determines that the angle R3 is larger than the threshold value angle R2, and the participant is not gazing at the display unit 3B.

  In FIG. 3, the gaze direction is limited to a plane for easy understanding, but it is also possible to determine the gaze direction in consideration of the vertical angle from an actual image. In that case, when the difference in the gaze direction is larger than either the left or right or top and bottom thresholds, it is determined that the display unit 3B is not being gaze.

  Compared with the method of line-of-sight extraction using a dedicated device or processing unit on the remote side, the method of the present invention calculates the line-of-sight direction by analyzing a remote image, but at least the center reference position is determined. It is technically possible to separate the level of gaze or whether it is shifted to the left or right. For a speaker, just knowing whether or not he / she is paying attention can be useful enough.

  Next, the overall operation of the present embodiment will be described. FIG. 4 is a flowchart showing an example of the operation of the video conference apparatus 1.

  When the transmission / reception unit 11 receives an image from the remote video conference device 1, the communication processing unit 12 sends the image to the participant detection unit 13 and the image output unit 18 (step S11). The participant detection unit 13 analyzes the image and extracts participants, and the gaze direction detection unit 14 detects the gaze direction of the participant (step S12).

  At the same time, the display device position estimation unit 15 estimates the position of the remote image display device (step S13). The gaze direction difference calculation unit 16 calculates the difference between the gaze direction of the remote participant and the direction of the image display device (step S14). Then, as described above, the gaze direction difference display unit 17 generates, for example, an arrow to be superimposed on the image and inputs it to the image output unit 18, and the image output unit 18 displays the received image and the gaze direction (difference). (Step S15).

  FIG. 5 shows an example of an image sent from the remote side. The participant detection unit 13 analyzes the main image and identifies a person in the image, and the gaze direction detection unit 14 calculates the apparent gaze direction from the face orientation and the like. In the example of FIG. 5, it can be seen that the user is looking to the right of the front. At the same time, the display device position estimation unit 15 estimates the position of the remote image display device. Here, the line-of-sight direction of a person in the image is observed for a certain period of time, and if the same direction is viewed for a certain percentage or more, it is estimated that the image display device (monitor) is in that direction.

  Subsequently, the difference between the apparent line-of-sight direction and the direction of the image display device is calculated. Here, since it can be seen that the remote participant is looking at the direction of the image display device, the actual line-of-sight direction is determined to be the front direction on the local side. Therefore, for example, the determined line-of-sight direction is displayed on the image display device by a method of superimposing an arrow symbol on the remote image as shown in FIG. By seeing this, the local speaker can easily see that the remote participant, who appears to be looking at an oblique direction, is looking at him. In addition, even when a remote participant looks away from himself / herself, it is easy to understand and smooth communication is realized.

  FIG. 7 shows an example in which the local image is shown in the remote image, and the direction of the image display device can be detected by detecting this. Here, the image sent from the local side is shown in the display in the upper center, and the two remote participants appearing in the front are looking in completely different directions. Actually, it is determined that the direction of the image display device = looking at the front. Therefore, the respective line-of-sight directions are displayed as shown in FIG. Alternatively, since both are looking in the same direction, they can be integrated and displayed as shown in FIG.

  Even if the local image itself is not shown, if the image display device, a part of the image display device, or a device including the image display device (here, a notebook computer) is shown as shown in FIG. From there, the position of the image display device can also be estimated. Further, as shown in FIG. 11, when a projector is shown, the direction of the screen can be estimated from the direction of the projector.

  As described above, in the video conference device 1 according to the first embodiment, the gaze direction of the remote participant is calculated only by the local processing without providing a dedicated device or processing unit on the remote side. Can be displayed. As a result, an environment close to face-to-face communication can be provided.

  Also, the local speaker can immediately see if the remote participant is looking at the local participant. In particular, the line-of-sight direction of the extracted remote participant is displayed as the local speaker direction and other binary values, or the local participant direction (image display device) and its left and right three values. In this case, it can be more easily grasped whether or not the local participant is being viewed.

(Embodiment 2)
In the second embodiment, in addition to the operation of the first embodiment, there are a plurality of participants at the communication partner base, and the gaze directions of the plurality of participants are combined. FIG. 12 is a block diagram illustrating an example of a configuration of the video conference apparatus according to Embodiment 2. Compared to the video conference apparatus of the first embodiment, a composite gaze direction calculation unit 20 is added. In FIG. 12, the same reference numerals are assigned to the blocks equivalent to those in the first embodiment. Blocks with the same name but with different signs have slightly different functions.

  In the video conference apparatus 1 according to the second embodiment, the participant detection unit 23 extracts a plurality of participants when the received image includes faces of the plurality of participants. The gaze direction detection unit 24 detects the gaze direction of each of the plurality of participants. The display device position estimation unit 25 estimates the position of the display device and simultaneously detects the direction of the display device for each of a plurality of participants.

  The composite gaze direction calculation unit 20 calculates a composite gaze direction obtained by synthesizing the gaze directions passing through the points where the gaze directions of the plurality of participants intersect among the points having the largest number of gaze directions passing through the points. . That is, the direction in which many of the participants are gazing is synthesized to be the combined gaze direction. The composition of the gaze direction is calculated by, for example, calculating the sum of the vectors with each direction being a vector of equal length and normalizing the unit length.

  When there are a plurality of remote participants, the gaze direction information is not displayed individually for each participant, but is displayed in an integrated manner. As a method of integration, since a method of taking an average value is not useful, a method of displaying only the maximum number, displaying by clustering, displaying a certain number or more, and the like are conceivable. In addition, it is possible to change the display according to the number of participants who have been integrated (thicken / enlarge arrows or change colors).

  In this case, it does not make much sense to take an average value or a sum. For example, when one participant is looking at the direction of the speaker, and no one is looking at the midpoint between the speaker and the material when the other participant is looking at the direction of the material. is there.

  The gaze direction difference calculation unit 26 synthesizes the direction in which the image display device is viewed from the participant who is gazing at the combined gaze direction to obtain the direction of the display device (composite display device direction). Again, vector operations can be used. Then, the difference between the composite gaze direction and the composite display device direction is calculated.

  The gaze direction difference display unit 27 is the same as that in the first embodiment, but the display device direction is changed to the composite display device direction. In addition, it may be possible to know which participant's gaze direction is displayed by, for example, superimposing and displaying some mark on the participants involved in the synthetic gaze direction.

  The method for displaying the difference in the gaze direction is the same as in the first embodiment. It is also possible to classify and display the gaze direction into two or three classifications.

  With reference to the flowchart of FIG. 13, the operation of the video conference apparatus 1 according to Embodiment 2 will be described.

  When the transmission / reception unit 11 receives an image from the remote video conference device 1, the communication processing unit 12 sends the image to the participant detection unit 23 and the image output unit 18 (step S21). The participant detection unit 23 analyzes the image and extracts a plurality of participants, and the gaze direction detection unit 24 detects the gaze direction of each of the plurality of participants (step S22).

  At the same time, the display device position estimation unit 25 estimates the position of the remote image display device (step S23). At the same time, the direction of the display device is detected for each of a plurality of participants.

  The composite gaze direction calculation unit 20 calculates the point of the points where the gaze directions of a plurality of participants intersect (within the respective threshold values of the participants) having the largest number of gaze directions passing through that point. A combined gaze direction obtained by synthesizing the gaze direction through is calculated (step S24).

  The gaze direction difference calculation unit 26 synthesizes the direction in which the image display device is viewed from the participant who is gazing at the composite gaze direction, and sets the direction as the composite display device direction. Then, the difference between the composite gaze direction and the composite display device direction is calculated (step S25).

  As described above, the gaze direction difference display unit 27 generates, for example, an arrow to be superimposed on the image and inputs it to the image output unit 18, and the image output unit 18 displays the received image and the gaze direction (difference) ( Step S26).

  FIG. 14 shows an example in the case where there are a plurality of participants on the remote side and watching from different directions. Here, it can be seen that one person is looking at the left direction and the remaining three people are looking further at the right direction than the screen estimated from the orientation of the projector. Therefore, for example, as shown in FIG. 15, by displaying a plurality of line-of-sight directions facing the same direction, it is possible to instantly grasp which group is paying attention to as a whole.

  16 and 17 are plan views showing an example of determination of the gaze direction according to the second embodiment, and shows a case where there are a plurality of participants on the remote side. Circles represent participants, and arrows indicate the gaze direction of each participant. The gaze direction is detected by the local gaze direction detection unit 24.

  From the gaze direction of each participant, the position of the display unit 3B on the remote side is estimated. For the estimated position of the display unit 3B, a gaze direction difference and a threshold value are given to each participant. When the difference is smaller than the threshold value for each participant, the gaze direction difference calculation unit 26 determines that the display unit 3B is being watched. A white circle represents a person who is gazing at the display unit 3B, and a black circle is a person who is not gazing at the display unit 3B.

  The gaze direction difference display unit 27 displays the combined gaze direction when more than half of the gaze directions of the participants intersect at a point where the gaze directions of a plurality of participants intersect (when intersecting within the threshold range of each participant). Show. In FIG. 16, more than half of the viewers look at the display unit 3B, and the combined gaze direction is displayed by an arrow indicating that the display unit 3B is viewed, for example, a downward arrow. By looking at the arrow displayed on the local display unit 3, it can be seen that attention is generally given.

  In FIG. 17, the line-of-sight directions are various, and the composite gaze direction is not displayed, or the composite gaze direction is displayed in the left direction assuming that more than half of the people are facing left as viewed from the display unit 3B. . In either case, it can be seen that more than half of them do not look at the display unit 3B, and that the whole is not paying attention.

  In addition, although communication between two bases was demonstrated in embodiment, this invention can be applied when performing a video conference simultaneously in three or more bases. When the display unit 3 divides and displays at two or more bases of communication partners, the difference between the gaze direction and the direction of the image display device is displayed for each communication partner. When the display unit 3 is provided for each communication partner, the difference between the gaze direction of the participant on the communication partner side and the direction of the image display device is displayed on each display unit 3.

  A first effect of the present invention is that the line-of-sight direction of a remote participant can be calculated and displayed only by local processing without providing a dedicated device or processing unit on the remote side. This is because the local image analysis unit calculates and displays the viewing direction of the remote participant. Generally, the information obtained directly from the remote image is only the apparent line-of-sight direction of the remote participant, but at the same time, the position of the remote image display device is estimated, and the position direction of the image display device and the apparent By calculating the difference between the gaze directions, the actual gaze direction can be determined.

  The second effect of the present invention is that the local speaker can immediately know whether or not the remote participant is watching the local speaker. The reason is that the line-of-sight direction of the extracted remote participant can be displayed as the local speaker direction and the other two values, or the local speaker direction and the left and right three values. Because.

  The third effect of the present invention is that, when there are a plurality of remote participants, it is easy for the local speaker to understand where the entire remote participant is gazing. The reason is that it has a function of appropriately integrating and displaying the gaze directions of a plurality of remote participants.

  As a result, according to the communication device with an image according to the present invention, an environment close to face-to-face communication can be provided.

It is a block diagram which shows an example of a structure of the video conference apparatus which concerns on Embodiment 1 of this invention. 1 is a block diagram illustrating an example of a system that uses a video conference apparatus according to Embodiment 1. FIG. It is a top view explaining the method of determining the difference of a participant's gaze direction and a display apparatus direction. FIG. 3A shows an angle for determining a direction of gazing at the display unit. FIG. 3B shows a state in which a direction deviated from the display unit is observed as an example. 5 is a flowchart illustrating an example of an operation for displaying a gaze direction of the video conference apparatus according to Embodiment 1; It is a figure which shows an example of the image sent from the remote side. 6 is a diagram illustrating an example of display of a gaze direction according to Embodiment 1. FIG. It is a figure explaining the method to estimate the position of an image display apparatus. It is a figure which shows the example of a display of a gaze direction. It is a figure which shows the example which synthesize | combines and displays a gaze direction. It is a figure explaining the method to estimate the position of an image display apparatus. It is a figure which shows the example in which the projector is reflected on the image. It is a block diagram which shows an example of a structure of the video conference apparatus which concerns on Embodiment 2 of this invention. 10 is a flowchart illustrating an example of an operation for displaying a gaze direction of the video conference apparatus according to Embodiment 2. 10 is a diagram for explaining an operation for displaying a gaze direction according to Embodiment 2. FIG. 10 is a diagram illustrating an example of display of a gaze direction according to Embodiment 2. FIG. 10 is a plan view illustrating an example of determination of a gaze direction according to Embodiment 2. FIG. 10 is a plan view showing an example of different gaze directions according to Embodiment 2. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Video conference apparatus 2 Control apparatus 3, 3A, 3B Display part 4, 4A, 4B Camera 11 Transmission / reception part 12 Communication processing part 13 Participant detection part 14 Gaze direction detection part 15 Display apparatus position estimation part 16 Gaze direction difference calculation part 17 Gaze direction difference display unit 18 Image output unit 19 Image input unit 20 Composite gaze direction calculation unit 23 Participant detection unit 24 Gaze direction detection unit 25 Display device position estimation unit 26 Gaze direction difference calculation unit 27 Gaze direction difference display unit

Claims (16)

Image receiving means for receiving an image from a communication partner;
Detecting means for detecting a human face and a gaze direction in the image from the image received by the image receiving means;
Estimating means for estimating the position of the image display device on the communication partner side from the image received by the image receiving means;
Difference calculating means for calculating a difference between the gaze direction in the image and the direction of the position of the image display device on the communication partner side;
Image display means for displaying an image received by the image receiving means;
A difference display means for displaying the difference calculated by the difference calculation means on the image display means together with the image received by the image receiving means;
A communication apparatus with an image, comprising:   The communication device with an image according to claim 1, wherein the estimating unit estimates the position of the image display device on the communication partner side by observing a gaze direction in the image over a predetermined time.   The communication device with an image according to claim 1, wherein when the image display device on the communication partner side is detected in the image, the estimation means estimates the position as the position of the image display device.   The said estimation means estimates the position of the said image display apparatus of the communicating party side from the position and direction, when the projection apparatus of the image display apparatus of the communicating party side is detected in the said image. Item 2. The image-attached communication device according to Item 1.   2. The difference display unit displays the difference calculated by the difference calculation unit by classifying the difference into one of two directions, that is, a direction of the image display device on the communication partner side and the other. 5. The image-attached communication device according to any one of items 4 to 4.   The difference display unit displays the difference calculated by the difference calculation unit by classifying the difference into one of the direction of the image display apparatus on the communication partner side and the right and left of the direction. 5. The image-attached communication device according to any one of 1 to 4. The detection means detects a gaze direction in the image for each person when detecting the faces of a plurality of persons in the image received by the image receiving means,
The difference calculation means is a composite gaze direction obtained by synthesizing a gaze direction passing through a point having a maximum number of gaze directions passing through the points among the points where the gaze directions of the plurality of persons intersect with the image display device. Calculate the difference from the direction of
The difference display means displays a difference between the synthetic gaze direction and the direction of the image display device.
The image-attached communication device according to any one of claims 1 to 6, An image receiving step for receiving an image from a communication partner;
A detecting step for detecting a human face and a gaze direction in the image from the image received in the image receiving step;
An estimation step for estimating the position of the image display device on the communication partner side from the image received in the image reception step;
A difference calculating step for calculating a difference between the gaze direction in the image and the direction of the position of the image display device on the communication partner side;
A difference display step for displaying the difference calculated in the difference calculation step on the image display means together with the image received in the image reception step;
A communication display method with an image, comprising:   9. The communication display method with image according to claim 8, wherein the estimating step estimates the position of the image display device on the communication partner side by observing a gaze direction in the image over a predetermined time. .   9. The communication display method with image according to claim 8, wherein when the image display device on the communication partner side is detected in the image, the estimation step estimates the position as the position of the image display device. .   The estimation step estimates the position of the image display device on the communication partner side from the position and direction when the projection device of the image display device on the communication partner side is detected in the image. Item 9. The communication display method with image according to Item 8.   The difference display step classifies and displays the difference calculated in the difference calculation step by classifying the difference into one of a direction of the image display device on the communication partner side and the other. The communication display method with an image according to any one of Items 11 to 11.   The difference display step classifies and displays the difference calculated in the difference calculation step by classifying the difference into one of the direction of the image display device on the communication partner side and the right and left thereof. The communication display method with an image according to any one of 8 to 11. The detecting step detects a gaze direction in the image for each person when detecting a plurality of human faces in the image received in the image receiving step;
The difference calculating step includes, for the point having the largest number of gaze directions passing through the points among the points where the gaze directions of the plurality of persons intersect, a synthesized gaze direction obtained by combining the gaze directions passing through the points and the image display device Calculate the difference from the direction of
The difference display step displays a difference between the synthetic gaze direction and the direction of the image display device.
The communication display method with an image according to any one of claims 8 to 13. Computer
Image receiving means for receiving an image from a communication partner;
Detecting means for detecting a human face and a gaze direction in the image from the image received by the image receiving means;
Estimating means for estimating the position of the image display device on the communication partner side from the image received by the image receiving means;
Difference calculating means for calculating a difference between the gaze direction in the image and the direction of the position of the image display device on the communication partner side;
Image display means for displaying an image received by the image receiving means;
A difference display means for displaying the difference calculated by the difference calculation means on the image display means together with the image received by the image receiving means;
A program characterized by making it function. Receiving means for receiving an image from a communication partner;
Image display means for displaying an image received by the receiving means;
A transmission means for transmitting an image of a participant to a communication partner via a network;
A first image-attached communication device comprising:
Image receiving means for receiving an image from the first communication device with an image via the network;
Detecting means for detecting a human face and a gaze direction in the image from the image received by the image receiving means;
Estimating means for estimating the position of the image display device on the communication partner side from the image received by the image receiving means;
Difference calculating means for calculating a difference between the gaze direction in the image and the direction of the position of the image display device on the communication partner side;
Image display means for displaying an image received by the image receiving means;
A difference display means for displaying the difference calculated by the difference calculation means on the image display means together with the image received by the image receiving means;
A second image-attached communication device comprising:
A communication system with an image, comprising:

Images