JPH08256316A - Communication conference system - Google Patents

Abstract

PURPOSE: To realize a communication conference system in more natural way by storing plural video image patterns of a person taking notice of a screen pattern and displaying the patterns on a small screen corresponding to the relation of position between the person taking notice of the screen pattern and a person to be noticed of. CONSTITUTION: An image rotation section 35 of each terminal equipment rotates video information obtained by a frame separate section 33 based on direction information generated by a position relation detection section 34 and stores plural kinds of video image patterns such as faces, eyes and mouths of conference participants. When a received video image pattern is inserted to video information at the position of a division screen pattern corresponding to a display section 20 and the name of a terminal equipment and the name of a screen pattern to be noticed of for noticed relation information correspond to each other, the face of the person taking notice of the pattern displayed on a small screen corresponding to each terminal equipment is directed full-face and an image in which a sight is met to that of a conference participant taking a seat in front of the terminal equipment is displayed. Thus, the person taking notice of the screen makes conversation with the person to be noticed of displayed on the small screen in face-to-face state and unnatural communication conference is eliminated.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a communication conference system for holding a conference between remote terminals via a communication path.

[0002]

2. Description of the Related Art A communication conference system has been used more and more in recent years because it can shorten the travel time of a person far away, but because of the form of the conference on the display, it is a natural conference for people to talk. Is lacking, and improvements are desired to approach a more natural form of meetings.

A conventional communication conference system is constructed by connecting a plurality of remote terminals via a communication path.

In this communication conference system, the images of the faces of the conference participants are captured by the cameras provided in the respective terminals.
While transmitting the video information to each terminal, each terminal receives the video information and displays one conference participant on each of a plurality of small screens on the display. There is also a mode in which the entire screen of the display is vertically divided into a plurality of sections and each participant is arranged, but this can be considered to be the same as a mode in which small screens are arranged adjacent to each other on the display.

By the way, the camera of the conventional communication conference system is generally installed in the central portion of the display, and the video information of the conference participants captured from the position is transmitted to each terminal.

In this case, since each terminal receiving the video information displays the sent video information as it is, each conference participant sees the screen of the display from the front. , The image is one in which each line of sight is directed downward. This is unnatural as it seems that all the participants in the conference are facing down.
Therefore, a technique has been proposed so far in which the entire display is set as one screen and only one participant is displayed in the central part of the screen to match the line of sight with the other party. In this technology, for example, a liquid crystal screen is used as a display surface and an image is projected from the back side of the screen while a part of the screen is scanned so that light from the front side can be transmitted, and the transmitted light is photographed by a camera installed on the back side. There are technologies and technologies that use half mirrors.

However, in any of these techniques, only one of the other parties can be displayed on the display, and only one person can talk with each other. Therefore, the appearance of the people around the person cannot be understood, and a plurality of people participate. The meeting will be unnatural in a different aspect from the line of sight described above.

[0008]

As described above, in the conventional communication conference system described above, when a plurality of small screens are provided on one display screen to display a plurality of conference participants, each small screen of each terminal is displayed. Since the placement of can be different for each terminal, it can be different from the screen such as who the other participants on the terminal screen are actually seeing and who is talking to whom. I didn't understand, and it was unnatural as a meeting form.

Generally, we often talk while looking at the other party's eyes, but in a conventional conference between communication terminals, it is theoretically difficult to make eye contact with the other party, and the conversation is difficult. There was a problem that it was unavoidable to feel natural.

The present invention has been made in order to solve such a problem, and an object thereof is to provide a communication conference system capable of bringing a communication conference closer to a natural conference form.

[0011]

In order to achieve the above object, the invention according to claim 1 connects a plurality of terminals through a communication path, and a conference participant displayed on a split screen on each terminal. In a teleconferencing system for conducting a conference while watching, a video capturing unit that captures the video information of the screen target person existing in front of the screen of each terminal, and the video information of the screen target user captured by the video capturing unit are displayed. On the basis of the direction of interest detecting means for detecting that the screen observer has focused on a desired small screen in the split screen, and the subject being displayed on the small screen detected by the direction of interest detecting means And image information of the screen target person captured by the image capturing means as frame information on the communication path, and transmitting means for receiving and decomposing the frame information through the communication path. Receiving means, distributing means for distributing the video information of the screen target person received and decomposed by the receiving means to corresponding small screens on the split screen, and the distributing means for distributing the video information of the screen target person. A positional relationship detecting means for detecting a positional relationship between the small screen and the small screen to which the image information of the subject is distributed, and a storage means for storing a plurality of image patterns in which the face and eyes of the screen subject are changed. And, based on the positional relationship detected by the positional relationship detecting means, correspond to the positional relationship so that the direction of the face in the image information of the screen target person is directed toward the small screen on which the target person is displayed. The image rotation means for reading out the video pattern from the storage means, and synthesizing the video pattern with the video information of the screen target person, and the video information of the synthesized screen target person and the video information of the target person are respectively separated. And and means for displaying on the small screen that is.

According to the second aspect of the invention, based on the positional relationship detected by the positional relationship detecting means, the direction of the eyes in the image information of the screen target person is set to the direction of the small screen on which the target person is displayed. It is equipped with a line-of-sight rotation means for reading out a video pattern corresponding to the positional relationship from the storage means so as to direct it and synthesizing it with the video information of the screen target person.

Further, the invention according to claim 3 is characterized in that the voice capturing means for capturing the voice of the screen observer issued to the small screen of each terminal, and the voice information captured by the voice capturing means are included in the frame. Means for adding information, means for extracting audio information from the frame information received by the receiving means from the communication path, and the image synthesized by the image rotating means according to the audio information of the extracted screen target person And means for performing timing control for switching patterns.

In the invention according to claim 4, the conference material is displayed on one small screen among the split screens, and when it is detected that the screen observer sees the small screen of the conference material, The screen rotation means reads out an image pattern from the storage means so that the face of the screen subject is directed toward the small screen of the conference material without considering the video information of the other subject, and the screen subject displays the image pattern. It is characterized by performing image rotation control to be combined with the image information of a small screen that is being displayed.

[0015]

According to the first aspect of the present invention, remote terminals are connected to each other via a communication path and displayed on the terminals when a conference is held while watching the participants displayed on the split screens of the terminals. When a screen participant focuses on a small screen of a participant who is a conference participant, the image information of the screen participant includes a video pattern from the storage device so that the person's face is directed toward the participant. Since they are read out and combined, it is possible to hold a communication conference in a more natural form.

According to the second aspect of the present invention, when the screen subject pays attention to a desired subject, the eyes in the image information of the screen subject are directed toward the small screen on which the subject is displayed. The video pattern corresponding to the positional relationship is read out from the storage means and is combined with the video information of the screen target person, so that the conference participants displayed in the split screens of the respective terminals face each other and match their line of sight. It becomes a state of talking in the open state, and it is possible to hold a communication conference in a more natural form.

According to the third aspect of the present invention, a plurality of image patterns of the storage means are switched and displayed on the screen in accordance with the voice information of the screen target person, so that the voice of the screen target person can be spoken by the person. As the movements are linked, it is possible to hold a teleconference in a more natural form.

In the invention according to claim 4, when it is detected that the person watching the screen looks at the small screen of the conference material, the meeting is performed by the screen rotating means without considering the video information of the other people. Since the image pattern is read out from the storage means so that the face of the screen subject is directed toward the small screen where the material is displayed, and is combined with the image of the screen subject, the communication conference should be conducted in a more natural form. You can

[0019]

Embodiments of the present invention will now be described in detail with reference to the drawings.

FIG. 1 is a diagram showing a schematic configuration of a communication conference system according to an embodiment of the present invention, and FIG. 2 is an external view showing a display unit of a terminal in the communication conference system of FIG.

As shown in FIG. 1, this communication conference system is constructed by connecting a plurality of terminals 1, 2, 3 to n existing at a remote location via a communication network 4.

As shown in FIG. 2, a plurality of small screens 21 to 24 and the like are displayed on the display unit 20 of each terminal of this communication conference system, and a conference participant A is displayed in the small screen 21. ing. The conference participant B is displayed in the small screen 22. The conference participant C is displayed in the small screen 23. The materials of the conference are displayed in the small screen 24. Note that each of the small screens 21 to 24 may be one screen divided into a plurality of screens. Further, a camera 25 or the like as an image capturing means for capturing an image including the faces of the conference participants existing in front of the display unit is provided above the display unit 20. Further, each terminal is provided with a microphone 26 or the like as a voice capturing means for capturing a voice uttered by a conference participant. Further, each terminal is provided with a speaker 27 or the like for outputting voices from other conference participants.

FIG. 3 is a diagram showing the internal structure of each terminal.

In the figure, reference numeral 30 is a screen-to-be-detected section, which is based on the image information taken in from the camera 25, the position on the screen the person (screen-focused person) is paying attention to, that is, the small screen 21. The position of any one of 24 to 24 is detected. Reference numeral 31 is a person-to-be-detected section, which detects the person (or material) displayed on the small screen at that position based on the position on the screen detected by the screen-to-be-detected section 30 and detects the relationship information of interest. To create. 32 is a framing unit,
After converting the video information from the camera 25, the attention-related information and the audio information from the microphone 26 into frame information, the communication network 4
Send to. A frame decomposing unit 33 receives frame information through the communication network 4 and decomposes video information, attention-related information, and audio information from the frame information. The decomposed relationship information of interest is distributed to the positional relationship detection unit 34 and output. The positional-relationship detecting unit 34 detects from the input attention-related information what the positional relationship between the screen subject and the subject on the terminal is, and creates attention-related information. This attention-related information is coordinate information such as coordinates X and Y. Here, for convenience of description, information that X is oriented in the direction of Y is expressed as X → Y. The voice information is also output to the speaker 27. The image information is the image rotation unit 35.
Output to The image rotation unit 35 uses the frame decomposition unit 3 based on the direction information created by the positional relationship detection unit 34.
The image information obtained in 3 is rotated. The image rotating unit 35 stores a plurality of types of image patterns such as faces, eyes, and mouths of the conference participants. The image rotating unit 35 stores several image patterns when the face line of sight or the mouth opening degree of the conference participants change. An image memory 36 fits (combines) the input video pattern into the video information at the position of the corresponding divided screen.

At this time, the terminal name and the focused screen name of the focused relationship information correspond to each other, that is, the small screen 21, the small screen 22, and the small screen 23 correspond to the terminal 1, the terminal 2, and the terminal 3, respectively. If so, the face of the target person displayed on the small screen corresponding to each terminal faces the front, and further, an image is displayed in which the line of sight faces the conference participant sitting in front of the terminal.

Hereinafter, with reference to FIGS. 4 and 5, a description will be given on the assumption that three persons such as conference participants A, B, and C hold a conference in this communication conference system.

FIG. 4 is a diagram showing an example of arrangement of small screens when the conference participants A, B, and C are having a conference.

As shown in the figure, conference participants A, B, C
Each terminal 1, 2, 3 corresponds to. Each of the conference participants A, B, and C can select each terminal 1,
Due to the fact that the camera 25 is arranged at a position of some 2, 3, face images of the conference participants A, B, and C are captured at the position of the camera.

FIG. 5 shows the above-mentioned arrangement example of the conference participants A,
It is a figure which shows one scene where B and C had a meeting, and operation | movement of this communication conference system is demonstrated based on this figure.

As shown in the figure, in this case, the conference participant A and the conference participant C are looking at each other's split screens,
It is assumed that the conference participant B is looking at the document screen.

In the image captured by the terminal 1 of the conference participant A, the small screen of the conference participant A is the target screen detection unit 30.
Is detected to be in the lower left direction.

Then, it can be seen from the person-to-be-detected section 31 in the next stage that there is a small screen of the conference participant C in the lower left direction. As a result, the attention detecting section 31 creates attention relationship information of the conference participants A → C and transfers it to the framing section 32. The framing unit 32 converts the input attention-related information, video information, and audio information into frame information and sends the frame information to the communication network 4, and the frame information is delivered to each terminal 1, 2, 3 or the like via the communication network 4. To be

When the frame information is received by the terminal 2,
The frame decomposition unit 33 extracts the attention relationship information of the conference participants A → C and outputs it to the positional relationship detection unit 34.

The positional relationship detecting unit 34 first checks the screen relationship between the conference participant A and the conference participant C from the split screen layout information in the noticed relation information of the terminal 2. From the center coordinates of the small screen of the conference participant A and the center coordinates of the small screen of the conference participant C, vector amounts in the X-axis direction and the Y-axis direction can be obtained. From this, the combined vector direction is obtained. The direction information of this combined vector is output to the image rotation unit 35 at the next stage. The image rotation unit 35 rotates the video information input from the frame decomposition unit 33 based on the input direction information. As this rotation technique, for example, an image synthesis technique is used.

Here, the procedure until the direction information is generated by the positional relationship detecting section 34 will be described in detail.

The direction information, that is, the information such as the face orientation and the eye orientation of the focused person (subject) is generated by the following procedure based on the focused relationship information.

The positional relationship detecting unit 34 first obtains the horizontal rotation angle of the face from the right and left edges of the face and the center positions of the inner and outer canthus in the attention relationship information (step 1).

Next, an angle formed by a line segment connecting the outer corners (both inner corners) of the eyes and a line segment connecting the midpoints of both inner corners of the eyes and the midpoints of both end points of the lips,
The tilt angle is obtained based on the horizontal rotation angle of the face obtained in step 1 (step 2). Further, the straight line connecting the midpoints of the straight lines connecting the outer corners of the eyes from the center of the ellipse of the face is set as the front direction of the face, and the depth of the outer corners of the eyes is obtained (step 3).

Then, the center of the eyeball on the screen is obtained (step 4).

Further, the size of the eyeball is obtained based on the depth of the outer corner of the eye and the position of the inner corner of the eye obtained in step 3 (step 5). At this time, it is assumed that the diameter of the eyeball is equal to the distance between the inner and outer corners of the eye. The center position of the eyeball is obtained from the fact that the horizontal cross section of the face and the cross section of the eyeball are equal in distortion (step 6). Then, the edge of the iris is found, and the center direction of the iris is found by circle approximation (step 7). As for the distance between the terminal and the conference participant, the size of the image of the participant captured at the reference distance from the camera 25 is stored in advance in each terminal side, and the size of the image of the conference participant is read out when calculating the distance. It can be calculated by

The rotation angle of the face, the tilt angle,
You can know the rotation angle and distance of your eyes. By comparing this information and split screen layout information with the original information, you can determine which screen the target screen is located on and what is displayed on that screen. (Who is a person, and if it is not a person is a material).

As a result, the face of the conference participant A on the small screen on the terminal 2 faces the direction of the small screen of the conference participant C.

When the frame information arrives at the terminal 3 on which the conference participant C is sitting, the frame disassembling unit 33 extracts the attention relation information A → C and inputs it to the positional relation detecting unit 34. Since the terminal name (terminal 3) and the target terminal name are the same here, the image of the face of the conference participant A displayed on the small screen 21 on the screen of the terminal 3 is displayed in front of the terminal 3 by the image rotation unit 35. A shape facing the direction of the line of sight of the conference participant C who sits down,
In other words, to display the image of the face in front of conference participant A,
The corresponding video pattern is taken out from a predetermined storage area of the image rotation unit 35 and image-combined with the video information of the small screen 21.

The image information obtained by the image synthesis is stored in the image memory 36.
And is fitted into the corresponding position of the small screen 21 and then displayed on the display unit 20.

When a large number of conference participants exist, a small screen exists on the screen.

In this case, the change in the focus position of the target person is hardly reflected in the rotational expression of the face.

In this case, the change in the position of interest is further reflected in the eye movement. The image rotation unit 35 also has this function.

FIG. 6 is a block diagram showing how the voice is interlocked with the screen.

The same components as those in the above embodiment are designated by the same reference numerals, and the description thereof will be omitted.

In this case, as shown in the figure, when the frame decomposition unit 33 receives the frame information, it decomposes the video information and the audio information and outputs the video information to the image rotation.
The audio information is output to the image rotation unit 6 and the audio timing adjustment unit 61, respectively. When the image rotation unit 35 takes in the video information from the frame decomposition unit 33, the image rotation unit 35 executes image synthesis in the same manner as above. When the image rotation unit 35 captures the voice information, the image rotation unit 35 moves the image of the mouth according to the way the voice is output. Therefore, the image rotation unit 35 waits until the image composition is completed and the video information can be output to each small screen. A voice output command is issued to the voice timing adjustment unit 61. The audio timing adjustment unit 61 outputs audio in accordance with the output command from the image rotation unit 35.

Now, the screen synthesizing process by the image rotating unit 35 will be described.

In order to represent the shape of the face, the image rotation unit 35 approximates the vertex coordinates of the face and the line segment connecting them.
Prepare a dimensional model. Since this 3D model is created over the entire circumference of the head, a scanner that can simultaneously measure 3D shapes and color images is used.

By rotating the scanner around the target person, a complete three-dimensional shape of the person and a plane projection image from the cylindrical direction are captured. 512 while rotating 360 degrees
It scans twice, and obtains three-dimensional coordinates and color information of 512 points for each scan. The three-dimensional information obtained in this way is of high definition, and by using a plane projection image from the entire circumference, it is possible to synthesize a face image viewed from any direction.

In this embodiment, only one camera 25 is arranged at the center of the upper part of the screen of the terminal, but the number of cameras 25 is not limited to this, and a plurality of cameras 25 are installed in each terminal. You may do it. Regarding the movement of the mouth, if a plurality of images (picture patterns) of the movement of the mouth when a person is talking are stored in advance in the image rotation unit 35, and the images are switched and displayed according to the sound, a more uncomfortable feeling is obtained. It is thought to disappear.

As described above, according to the communication conference system of the present embodiment, when the conference participants A, B, C, etc. talk while looking at the parties displayed on the split screens of the respective terminals 1, 2, 3. , The image of the face of oneself (the person watching the screen) displayed on the small screens 21 to 23 becomes an image of talking to the other party (the person being watched) on the other small screen. It becomes possible to talk in a state of facing each other, and it is possible to eliminate the unnaturalness as compared with the case where all the conference participants are lined down facing as in the past.

In addition, conventionally, it is possible to know which screen (person or material) each person is looking at from the face direction and the line-of-sight direction of the conference participants A, B and C displayed on the split screen on the terminal. It can provide functions that are not available.

In the above embodiment, a plurality of image patterns such as faces, eyes and mouths of the conference participants are stored in advance in the image rotating section 35. However, the storage capacity of each terminal, the transmission capacity such as a communication path, etc. There is plenty of room, and the compressed image processing technology such as MPEG is used to sequentially display the real-time 3D video patterns of each conference participant captured by the camera in a moving image state, making the conference by communication more natural. It can also be shaped.

[0058]

As described above, according to the invention of claim 1, the small screen of the subject displayed on the terminal is
When the screen observer pays attention, a video pattern is read from the storage means and synthesized with the image information of the screen observer so that the direction of the person's face is toward the examinee, so it is displayed on a small screen. Also, it becomes possible to talk face-to-face with the target person, and the unnaturalness caused by the communication conference is eliminated.

According to the second aspect of the invention, the video pattern corresponding to the positional relationship is read from the storage means so that the line of sight in the video information of the screen subject is directed toward the small screen on which the subject is displayed. Since it is sent out and is combined with the video information of the screen target person, each conference participant displayed on the small screen can talk in a state where their sight lines are matched.

According to the third aspect of the invention, a plurality of image patterns of the storage means are switched and displayed on the screen in accordance with the voice information of the screen target person, so that the voice of the screen target person can be displayed. The movement of the mouth becomes linked.

According to the invention described in claim 4, when it is detected that the person watching the screen looks at the small screen of the conference material, the screen rotating means does not consider the video information of the other people. An image pattern is read from the storage means so that the face of the screen subject is directed toward the small screen on which the conference material is displayed, and is combined with the image of the screen subject. In this way, the communication conference can be held in a more natural form.

[Brief description of drawings]

FIG. 1 is a diagram showing a schematic configuration of a communication conference system according to an embodiment of the present invention.

FIG. 2 is an external view showing a display unit of a terminal having this communication conference system.

FIG. 3 is a diagram showing an internal configuration of the terminal of FIG.

FIG. 4 is a diagram showing an example of a screen layout of each terminal when a conference is held by three persons such as conference participants A, B, and C.

FIG. 5: Conference participant A designates conference participant C as conference participant C
FIG. 4 is a diagram showing the orientation of the screen on each terminal when the conference participant A and the conference participant B see the material.

[Fig. 6] Fig. 6 is a diagram showing a configuration in the case where the movement of the mouth on the screen is interlocked with voice as an application example of this communication conference system.

[Explanation of symbols]

1, 2, 3 to n ... Terminal, 4 ... Communication network, 20 ... Display unit, 2
1 to 24 ... small screen, 25 ... camera, 26 ... microphone, 27
... speaker, 30 ... screen-of-interest detection unit, 31 ... target-of-subject detection unit, 32 ... framing unit, 33 ... frame decomposition unit,
34 ... Positional relationship detection part, 35 ... Image rotation part, 36 ... Image memory, 61 ... Audio timing adjustment part.

Claims (4)

[Claims] 1. A communication conferencing system in which a plurality of terminals are connected via a communication path to hold a conference while watching a conference participant displayed on a split screen on each terminal, and the conference exists in front of the screen of each terminal. Based on the image capturing means for capturing the image information of the screen observer and the image information of the screen observer captured by the image capturing device, the screen observer focused on a desired small screen in the divided screen. Direction information detecting means for detecting that, the video information of the subject displayed on the small screen detected by the attention direction detecting means, and the video information of the screen attention person captured by the image capturing means And a receiving means for receiving and decomposing the frame information through the communication path, and a screen received and decomposed by the receiving means Distribution means for distributing the video information to the corresponding small screens on the split screen; a small screen to which the video information of the screen target person is distributed by the distribution means and a small screen to which the video information of the target person is distributed. A positional relationship detecting unit that detects a positional relationship, a storage unit that stores a plurality of video patterns in which the face and eyes of the screen subject are changed, based on the positional relationship detected by the positional relationship detecting unit, The image pattern corresponding to the positional relationship is read from the storage means so that the face direction in the image information of the screen subject is directed toward the small screen on which the subject is displayed, and the image of the screen subject is read. An image rotating means for synthesizing the information and a means for displaying the synthesized image information of the target person of the screen and the image information of the target person on a small screen in which the respective images are distributed. Conference system. 2. Corresponding to the positional relationship so that the direction of eyes in the image information of the screen target person is directed to the direction of the small screen on which the target person is displayed based on the positional relationship detected by the positional relationship detection means. 2. The communication conference system according to claim 1, further comprising line-of-sight rotation means for reading a video pattern to be read from a storage means and synthesizing the video pattern with the video information of the screen observer. 3. A voice capturing means for capturing a voice of a screen observer emitted to a small screen of each terminal, and a means for adding the voice information captured by the voice capturing means to the frame information. The timing control is performed to switch the audio information from the frame information received by the receiving means from the communication path, and to switch the video pattern synthesized by the image rotating means according to the extracted audio information of the screen observer. The communication conference system according to claim 1, further comprising: 4. The conference material is displayed on one small screen of the split screens, and when it is detected that the screen observer sees the small screen of the conference material, the screen rotation means changes to another screen. An image pattern of the small screen in which the screen target person is displayed by reading an image pattern from the storage means so that the face of the screen target person is directed toward the small screen of the conference material without considering the video information of the target person. 2. The communication conference system according to claim 1, wherein the image rotation control for combining the information is performed.