JP3027983B2 - Two-way interactive system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a two-way interactive system provided with a mechanism for matching eyes of interlocutors via transmission means. Specifically, a counseling system for realizing two-way dialogue via transmission means, such as a relationship between a doctor and a patient at a considerable distance, a counselor and a client, etc. It is about an oriented system.

[0002]

2. Description of the Related Art Conventionally, counseling performed in the fields of education, medical care, psychology, and the like is generally performed in a situation where interlocutors face each other directly while building a close mutual trust between them. However, those who have counseling skills with those who need counseling due to natural disasters, remote places, etc. (Counselors)
There are situations where it is difficult to share the same place and time with the same. Therefore, with the development of communication technology, the need for a system for realizing counseling between persons located in remote places has increased.

In order to construct such a counseling system using communication technology, in order to build a close relationship and an ideal counseling environment between interlocutors as in the conventional face-to-face method, First, it is important to match the eyes of the interlocutors.

In view of the above, a conventional image pickup apparatus that matches the line of sight between interlocutors via a transmission line is disclosed in
-269128 (the first prior art). In the first prior art, a half mirror provided between the interlocutor and the image capturing apparatus is provided, so that light emitted from the interlocutor serving as a subject is transmitted to the image capturing apparatus and the image display apparatus. Are separated. Therefore, according to the device configuration of the first prior art, the imaging device is arranged at a position avoiding the display screen of the image display device viewed by the interlocutor, and the input optical axis of the imaging device and the line of sight of the interlocutor. And match.

Secondly, Japanese Patent Application Laid-Open No. Hei 4-213287 (second prior art) uses a special display device capable of switching between reflection and transmission of light, and employs a special display device located behind the display device (this display device is referred to as a display device). An imaging device having a configuration in which the imaging device is arranged on the side opposite to the interlocutor via the intermediary) is disclosed. According to this device configuration, the line of sight of the interlocutor is directed to the imaging device arranged behind the display device and hence behind the display screen of the display device seen by the interlocutor. It is possible to match eyes.

Third, in Japanese Patent Application Laid-Open No. 4-977663 (third prior art), a small mirror or a small half mirror is mounted on the upper part of a display device, and a small camera whose optical axis is directed to the mirror is used for a user. There is disclosed an imaging apparatus configured to capture images. In this device configuration, the interlocutor image reflected by the mirror is captured by a small camera arranged at a position avoiding the display screen of the display device viewed by the interlocutor. Therefore, the elevation angle of the line of sight of the talker is suppressed within 10 ° which is the allowable range of the line of sight coincidence in the vertical direction, and the line of sight of the talker is achieved.

Fourth, Japanese Patent Application Laid-Open No. Hei 4-279915 (fourth prior art) has two display screens that are deployed at different positions so that different persons can see each other from opposite directions. There is disclosed an interactive workstation in which a consultation partner can see one display screen while operating a key input, and another consultation partner can display different interactive information on two screens. According to this device configuration, by displaying necessary information on the counselor side and the partner side, effective information excluding the counselor-side dedicated information can be easily provided to the partner side without interrupting the consultation partner dialogue environment. Show, smooth counseling becomes possible.

[0008]

In the first prior art in which light is separated from a subject by a half mirror, a half mirror having a size approximately equal to the screen size of a display device is required. Inevitably, a large space for accommodating such a large half mirror is required. In particular, when the screen size exceeds 40 inches, it cannot be easily manufactured. Also, since we use a half mirror,
An image picked up by an image pickup device installed at a position avoiding the screen on which the interlocutor image is displayed becomes dark, and an ideal counseling environment cannot be realized.

Next, the second prior art using a special display device for switching between reflection and transmission of light cannot be incorporated into a commercially available TV device or the like which has already been mass-produced and introduced.
Therefore, if an attempt is made to realize a system using this principle, the development of a dedicated machine is required, and the price of the apparatus becomes high. Further, an imaging device for imaging the interlocutor is arranged to image the interlocutor via the special display device. Therefore, the captured interlocutor image becomes dark, and an ideal counseling environment cannot be realized.

[0010] Further, in the third prior art in which a reflecting mirror and a small camera are combined, the camera is arranged at a position avoiding a display screen seen by the interlocutor, and the interlocutor is imaged through the small reflecting mirror. Therefore, the person image to be photographed becomes dark. Further, in order to obtain a wide angle of view, the size of the reflection mirror must be equal to or larger than the size of the small camera, and the display of a human image on the display device is hindered by the large reflection mirror. Also, when the position of the person to be photographed changes, the imaging direction of the small camera is constant,
You cannot follow this. Therefore, it is not always possible to ensure that the eyes of the interlocutors match.

The structural features common to the above-described prior arts 1 to 3 are a display screen (a screen on which an image of the other party is displayed) for the user to see the image pickup device (including the camera) and the other party. This point is located at a position avoiding the space between the two. In other words, a reflection mirror, a half mirror,
It is necessary to install a special display device. Therefore, the image of the other party to be transmitted becomes dark, and an ideal counseling environment cannot be realized.

Further, the above-described first to third prior arts are techniques that focus only on the display of an image such as a person image on a monitor screen, and are necessary for ordinary conversation (particularly in counseling). No consideration is given to how information and graphic information are displayed on the monitor screen.

In the above-mentioned fourth prior art, the display mode of the character information and the graphic information on the monitor screen is considered, but it is assumed that two persons use the same workstation to interact with each other. And cannot perform dialogue between remote locations.

SUMMARY OF THE INVENTION The present invention has been made in view of the above, and has a counseling system in which a line-of-sight between interlocutors is transmitted through a transmission unit in order to reduce a sense of discomfort during interlocutory interlocutors. It is an object of the present invention to provide a two-way interactive system that includes a mechanism for matching characters, and that processes character / graphic information in addition to the video of the interlocutor separately to shared information or non-shared information.

[0015]

SUMMARY OF THE INVENTION According to the present invention, there is provided a first terminal installed at a first point where a first interlocutor is located, and a first terminal located at a considerable distance from the first point and having a second interlocutor. Second
Bi-directional communication of transmission information having at least image and audio data of the first interlocutor and the second interlocutor between the second terminal installed at the point and the first terminal and the second terminal The present invention relates to a two-way interactive system provided with a transmission means for enabling the communication. In this system, at least the first terminal device is: (a) a display unit for displaying an image of the second interlocutor received via the transmission means on a monitor; The imaging unit detects a part of the human image corresponding to the position where the human image is actually installed, and in a state where the detected part of the human image is fixed, performs enlargement or reduction of the entire human image,
In a second area other than the first area on the monitor other than the first area where the person image is displayed, at least one of a character and a figure simultaneously displayed on the second terminal device is displayed. A display unit including a display control unit that displays at least one of a character and a graphic that are not displayed on the second terminal device in the third area other than the second area or the third area; A position defining means for defining a position of the first interlocutor with respect to the monitor screen, and (c) an apparatus which enables transmission of an image of the first interlocutor to the second terminal device via the transmission means, An imaging unit for directly imaging the first interlocutor;
An imaging apparatus having a support mechanism for setting the imaging unit at a predetermined position between the first interlocutor and the monitor while the imaging direction of the imaging unit is directly directed to the first interlocutor; and The imaging unit determines that the distance from the first interlocutor to the monitor specified by the position defining unit and that the line of sight of the first interlocutor matches the line of sight of the monitor image of the second interlocutor And a predetermined parallax angle to be set.

Here, it is preferable that the second area and the third area are reduced or expanded in conjunction with expansion or reduction of the first area.

Further, (i) the first area is arranged above and at the center of the monitor, (ii) the second area is arranged below the first area on the monitor, and (iii) The third area may be located on the monitor beside the first area.

By arranging the second area below the first area, it is possible to make maximum use of the width direction of the entire screen as a form for securing the area of the shared information section.

The transmission information can include character / graphic information in addition to the image and voice of the interlocutor. Thus, based on the character / graphic information received via the transmission means, the character or graphic shared by the first interlocutor and the second interlocutor transmitted from the other party is displayed on the second part of the display unit. It is also possible to display in the area. That is, by displaying not only images and voices but also characters and figures to be shared, the dialogue can be further facilitated.

Further, the system includes a character / graphic input device as a man-machine interface. With this configuration, a character or graphic input from the character / graphic input device via the character / graphic input control unit is displayed in the second area or the third area of the display unit, and
It is possible to output character information or graphic information corresponding to the character or graphic displayed in the area of the second terminal device to the second terminal device. As a result, information to be shared between the first interlocutor and the second interlocutor can be simultaneously displayed on both terminal devices, and information that the second interlocutor does not want to be known to the first interlocutor can be displayed. Interlocutors can occupy. Note that such a character / graphic input device may be provided only in one of the terminal devices at the first point and the second point where the interlocutor is located.

As described above, since the system is provided with the character / graphic input device, personal memos and conversations using characters and graphics as shared information can be performed, and a more smooth conversation can be realized. Note that, as such a character / graphic input device, a keyboard device and a pen input device (touch panel) arranged near the interlocutor can be adopted. Further, by providing both such a keyboard device and a pen input device so that the keyboard device and the pen input device can be appropriately selected, a smoother conversation can be realized. Further, the character / graphic input device is provided with a pointing / input device for inputting a pointer position pointing to a displayed character or graphic.
A device may be provided. The position indication by the pointer makes it possible to specify the conversation target at each time.
By utilizing such a character / graphic input device or the like and recognizing the input voice of the interlocutor, converting it into character information and displaying it, it is effective for establishing a smooth dialogue (or counseling).

The image capturing apparatus may further include an image capturing unit that captures the image of the first interlocutor, and a monitor that is connected to the first interlocutor while the image capturing direction of the image capturing unit is directly directed to the first interlocutor. And a support mechanism for setting the image pickup unit at a predetermined position between the two. In particular, (1) the imaging unit is a distance between the first interlocutor and the monitor defined by the position defining means, and
(2) the above-described parallax angles for which it is determined that the line of sight of the first interlocutor matches the line of sight of the monitor image of the second interlocutor;
And (2) are located at positions where the part of the monitor image of the second interlocutor is blocked from the first interlocutor. Note that the second terminal device may have a similar configuration, and the first terminal device may have the second terminal device.
The terminal device can be remotely controlled.

Further, each of the terminal devices may include a sound absorbing member which is disposed around the interlocutor and has a property of blocking external sounds. In such a case, the psychological obstacle to the leakage of the contents of the dialogue to the outside is reduced, and the sound quality is improved, so that it is possible to concentrate the feelings and to perform a deeper dialogue.

As described above, the system in which the imaging unit is installed so as to block a part of the monitor image in the above-mentioned line-of-sight area between the interlocutor and the monitor is configured by the interlocutor. Even when the imaging unit is installed so as to block a part of the human image displayed on the monitor, if the size of the imaging unit is suppressed to a certain value or less, the existence of the imaging unit is bidirectional. This is because the inventor has confirmed through experiments that it has little effect on the dialogue.

In the above two-way interactive system, the parallax that matches the line of sight of the interlocutor (first interlocutor) with the line of sight of the image of the person (second interlocutor) displayed on the monitor (monitor image). The angle is 4.5 ° or less in the left-right direction of the monitor screen and 1 ° in the direction directly above the monitor screen with respect to the line of sight of the dialogue.
It is preferable that the direction of 2 ° or less and the direction directly below is 8 ° or less. Further, the parallax angle is set such that the horizontal direction and the vertical direction of the monitor screen are each 3 ° or less with respect to the line of sight of the interlocutor when viewed from the interlocutor. Is more preferable.

It is preferable that the line-of-sight matching area where the imaging unit is installed is an area higher than the line of sight of the person who is photographed. In particular, as a result of an experiment by the inventor, the imaging unit that blocks between the interlocutor and the monitor is preferably located near the top of the human image displayed on the monitor as the installation position of the imaging unit when viewed from the interlocutor. Was confirmed.

A distance from the interlocutor specified by the position defining means to the monitor, a predetermined parallax angle at which it is determined that the line of sight of the interlocutor matches the line of sight of the person image displayed on the monitor; The means for installing the imaging unit in the area defined by
Means for moving the imaging unit itself within the line-of-sight matching area between the interlocutor and the person image on the monitor. Second, moving the interlocutor facing the monitor and the person image on the monitor. Means for moving the line-of-sight matching area itself to relatively change the position of the imaging unit and the line-of-sight matching area. Third, moving the person image on the monitor, and similarly, the line of sight of the interlocutor facing the monitor. And means for moving the line-of-sight matching area itself of the human image on the monitor (this third means is effective as an algorithm for moving the video display area in which the human image is displayed when the display control unit divides the screen). ).

The first means includes a processing unit for recognizing a position of a human image displayed on a monitor on the monitor in the terminal device, and a setting position of an imaging unit in the imaging device. This can be realized by providing a drive mechanism for relatively moving the human image whose position is recognized by the processing unit. As an embodiment, the first means is performed for each terminal device, and the position of the monitor image of the second interlocutor is recognized by the processing unit of the first terminal device. The second
In some cases, the first terminal device remotely controls the second terminal device by transmitting it to the terminal device.

The second means includes a processing unit for recognizing a position on the monitor of the person image displayed on the monitor in the terminal device, and a processing unit for recognizing the position of the person image recognized by the processing unit. This can be realized by providing a mechanism for adjusting the line of sight of the interlocutor photographed with respect to the line of sight in the position defining unit. As an embodiment, the first means is performed for each terminal device, and the position of the monitor image of the second interlocutor is recognized by the processing unit of the first terminal device. The second
In some cases, the first terminal device remotely controls the second terminal device by transmitting it to the terminal device.

Further, in order to realize the third means,
The terminal device includes a processing unit that adjusts a display position of a human image displayed on a monitor. The processing unit specifies a virtual position on the monitor image where the imaging unit should originally exist as a reference point, based on the human image received and displayed on the monitor. Based on the coordinates on the monitor corresponding to the installed position and the coordinates of the obtained reference point on the monitor, the displacement of the human image displayed on the monitor is detected. If it is determined that this person image is displaced from the display position on the monitor where it should be displayed, the coordinates of the reference point on the monitor are used as the starting point, and the monitor corresponding to the installation position of the imaging unit is used. A vector having the above coordinates as the end point is calculated, and the shifted human image is translated on the monitor based on the obtained vector. As an embodiment, a case in which the process is performed on a monitor image of the dialogue person (initial setting before the start of the dialogue)
May be performed on the monitor image of the second interlocutor. In particular, at the time of a screen division operation (including an operation of enlarging / reducing a displayed human image) by a display control unit to be described later, the display size of the human image (the monitor image of the first or second interlocutor) is arbitrarily set. When changed, the area for displaying the image can be set at an arbitrary position on the monitor screen. Therefore, when the sight line is matched between the arbitrarily displayed person image and the interlocutor, this second image is displayed. The third means is particularly effective.

The above-mentioned reference point detects the respective display positions on the monitor of both eyes of the person image received and displayed on the monitor, and moves the reference point from the center of both eyes to the upper side of the monitor. The position on the monitor is separated by a predetermined distance (for example, about 10 cm). Further, the reference point is obtained by binarizing a human image received and displayed on the monitor, detecting an outline of the human image, and specifying the position as a position on the monitor corresponding to the top of the human image. Is also good.

The drive mechanism for moving the imaging unit once recognizes the position of the interlocutor in the processing unit, and sets the angle of the imaging direction of the imaging unit based on the recognized position of the interlocutor and the position of the imaging unit. It may be configured to adjust. By adjusting the angle of the imaging unit by this drive mechanism, the imaging direction of the imaging unit is automatically directed to the line of sight of the interlocutor whose position has been recognized by the processing unit. Further, the drive mechanism for moving the imaging unit detects an average match in the line-of-sight direction, and when moving and aligning, the moving speed is equal to or lower than a predetermined value, and the centering operation is not performed when the shake in the line-of-sight direction is within a predetermined range. (This determination is made by a processing unit or the like). In addition, when the imaging unit moves, the image output from the imaging unit shakes. However, the movement speed is limited, and the movement is not frequently performed, and the movement is suppressed to a necessary minimum. .

Here, in the above-described driving mechanism for moving the imaging unit, it is preferable that a part of a member located between the interlocutor and the display unit is formed of a transparent material.

Next, the position defining means for defining the position of the interlocutor includes a height-adjustable chair (chair provided with a seat height adjusting mechanism) installed at a predetermined distance from the display unit. Also,
A table provided between the interlocutor and the display unit also functions as a position defining unit. In this system, the position defining means is used to set the distance between the photographed interlocutor and the display unit by 0.5 m or more.
m or less. With such a configuration, since the movement of the interlocutor in the front-rear and left-right directions can be sufficiently restricted by the position defining means, the gaze matching between the interlocutors in the system is caused by the physical differences of the interlocutors. This is because vertical movement adjustment is most effective.

Note that the imaging unit adapted to the system may be independent of the display unit. Further, it may be attached to a part of the display unit by a support mechanism.

The two-way interactive system according to the present invention comprises:
Since it is aimed at a counseling system between interlocutors at remote locations, it has various configurations and functions in order to collect useful counseling data and realize an ideal counseling environment.

For example, a display control section for controlling display on the display section has a function of enlarging / reducing an image displayed on the display section. Since the display size greatly affects the psychological effect of the user, by setting the size of the video display area on the monitor freely with this display control unit, the degree of freedom of the dialogue can be expanded and the comfortable dialogue can be performed. Environment can be realized. Since the display position of the reduced human image (monitor image) on the monitor is adjusted by the third line-of-sight matching means described above, the line-of-sight state between the first interlocutor and the second interlocutor is maintained. Is done.

Next, the above-described table as the position defining means is arranged between the interlocutor and the display unit such that one end faces the interlocutor and the other end substantially contacts the display unit. .
Here, the table is set to a height where (1) the interlocutor is higher than the table above the chest when sitting in a chair,
(2) The height above the chest may be set to be higher than the table with the interlocutor standing. With such a configuration, a sense of continuity is generated through the table, so that anxiety caused by losing the relative distance (psychological distance) from the other party is reduced. In this case, it is desirable that the display unit and the table are in contact with each other, but there may be a gap (for example, several tens cm or less) that does not cause a psychological obstacle.

Further, in the first terminal device, in the first area,
It is preferable that a name, an identification number, or unique information, which is identification information of a person corresponding to the person image, is further displayed in addition to the person image.

When a person image and a name, an identification number, or unique information, which are identification information corresponding to the person image, are displayed at the same time, even a first-time face-to-face conversation partner can easily be addressed by name or the like. Easy to compose a friendly conversation.
Also, seeing the name and face at the same time makes it easier to remember. Note that the unique information here includes, for example, an address, a belonging group, the number of interviews, and the like.

Further, in the first terminal device, in the first area,
In addition to the person image, it is preferable that information corresponding to the accumulated conversation time or the remaining conversation time with the person of the displayed person image is further displayed.

By displaying information (character display, bar graph display, etc.) according to the accumulated conversation time or the remaining conversation time, the conversation can proceed while checking the limited conversation time.

As a result, at the time of a one-to-one two-way conversation (for example, question and answer) with a specific person, which is used together with a one-to-one one-way conversation method (lecture style), Dialogue management in a fixed time becomes easy.

Further, in the first terminal device, in addition to the sound based on the transmitted sound information, the display of the person image in the first area is started, the display of the person image is ended, or another person image is displayed. It is preferable that a sound effect is further output in conjunction with the switching of the above.

Here, it is preferable that the sound effect is selected for each person of the displayed person image.

The sound effect is output in response to the start of the display of the human image in the first area, the end of the display of the human image, or the switching to another human image, so that the change of the screen is interlocked. Then, first, since the voice stimulus acts, the preparation time for the mind can be formed more than when a screen is suddenly generated from a silent state. Generally, an output time of 2-15 seconds for the sound effect is sufficient.

At the end, if the image suddenly disappears, the loneliness may increase. Therefore, if the screen gradually ends with the sound effect, a preparation time for the mind can be formed.

It is preferable to use a sound volume change or a rhythm change, such as gradually increasing or decreasing the sound effect. Of course, a part of the song may be used.

Further, if the sound effects are properly used for each person, personality can be added to the sound effects, and the person can be identified only by listening to the sound effects.

Further, the first terminal device notifies the end of the dialogue time when the remaining time of the dialogue with the person who is talking in the first area reaches a predetermined time in addition to the voice based on the transmitted voice information. It is preferable to output a sound or display an image for announcing the end of the conversation time.

By the audio output and the image display corresponding to the countdown of the end of the dialogue, the dialogue with a time limit can be smoothly advanced. In particular, even if it is difficult to terminate the dialogue, if the visualization is performed using a fun and unique animation, such as an animal climbing a tree and eating an apple according to the countdown, together with the countdown display, it will be accepted by the interlocutors Cheap.

Further, the first terminal device further includes a storage device for storing information on a candidate person to be interacted with, and before the display of the person image in the first area or after the display is completed, It is preferable to display information corresponding to the person stored in the storage device for a predetermined time or output sound.

By displaying information corresponding to the person over a predetermined period of time or outputting a voice before and after the display of the person image, it is possible to prepare for a dialogue and create a mild end atmosphere. This can be combined with the sound effects described above, or a message or the like may be captured. Furthermore, it is also possible to function as an individual CM corresponding to a person image, and a variety of atmospheres can be formed.

Further, in conjunction with the switching of the person image displayed in the first area to the person image of another person, the second image is displayed.
It is preferable to switch the display of any one of the area and the third area in accordance with another person.

By automatically displaying shared information (second area) and non-shared information (third area) in accordance with the displayed person image in conjunction with the displayed person image, one of such information search operations can be performed. The section (at least the search at the beginning of the dialogue) becomes unnecessary.
In particular, it is effective for a one-to-one two-way conversation (for example, a question-and-answer session) with a specific person in combination with a one-to-one one-way conversation system (lecture style).

[0056]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of a two-way interactive system according to the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements will be denoted by the same reference symbols, without redundant description.

FIG. 1 is a logical configuration diagram showing the entire interactive interactive system according to the present invention. This system uses A
Interlocutors 3A (first interlocutor) and B at point (first spot)
This is a system for realizing a two-way conversation with the interlocutor 3B (second interlocutor) at the point (second point). In particular, this system aims at a counseling system that establishes a close relationship of trust and realizes in-depth communication even with remote interlocutors, so at least one of the interlocutors is a teacher, doctor, or psychologist. Such counselors are assumed. Therefore, this system has various functions and configurations so that these counselors can provide information necessary for counseling and an ideal counseling environment.

Specifically, at the point A, (a) a chair 4A on which a talker 3A (for example, a counselor) sits;
A table 2A to which an interlocutor 3A goes;
7 and a terminal device 20A provided with a mechanism for matching the line of sight between the interlocutors. On the other hand, at the point B, (a) the chair 4B on which the interlocutor 3B (for example, a client receiving counseling) sits, (b) the table 2B to which the interlocutor 3B goes, and (c) the interlocutor via the transmission means 27 And a terminal device 20B provided with a mechanism for matching the line of sight of the user. Here, the transmission unit 27 is a light source that enables large-capacity transmission in order to transmit and receive transmission information including image information and audio information between the terminal device 20A at the point A and the terminal device 20B at the point B. It includes a fiber transmission line 31 (main transmission line) and a satellite communication transmission line 32 (backup transmission line). The transmission means 27 may be wired or wireless. In addition, the transmission means,
It includes a network such as a telephone line network already laid, and also includes an electronic bulletin board system (BBS: Bulletin Board System) provided with various databases.

At point A, the chair 4A described above functions to define the position of each interlocutor 3A. This chair 4A
Is preferably fixed to maintain a constant distance from the display unit 21A of the terminal device 20A. However, even when the chair 4A is not fixed, the table 2A is installed on the interlocutor 3A and the display unit 21A, so that the chair 4A is not fixed.
It is possible to keep the distance between A and the display unit 21A constant (an experiment by the inventor has confirmed that the existence of the table 2A serves as a reference for the psychological distance of the interlocutor 3A).
The same applies to the configuration of the terminal device 20B at the point B where the other party's interlocutor 3B (for example, a person receiving counseling) is located.

At the point A, the terminal device 20A (1)
A monitor TV1 arranged on the table 2A for displaying an image of the interlocutor 3B based on the transmission information transmitted from the terminal device 20B at the point B via the transmission means 27.
A display section 21A provided with A, and (2) an image of the interlocutor 3A captured by the CCD camera 5A is captured as image information.
The imaging unit 6A for transmission to the terminal device 20B at the point, and (3) the voice of the interlocutor 3B based on the voice information transmitted from the terminal device 20B at the point B via the transmission means 27. Output unit 23A (speaker) and (4) a voice input unit 24A (microphone) for collecting the voice of the interlocutor 3A as voice information and transmitting it to the terminal device 20B at the point B. Furthermore, in order to realize more detailed communication as a counseling system, (5) the interlocutor 3A inputs characters and figures, and the input characters and figures are displayed on the display unit 21A.
And a character / graphic input unit 25 for transmitting the character / graphic information to the terminal device 20B at the point B.
A (an interface such as a keyboard, a pointing device, a touch panel, etc.) and (6) a processing unit 26A provided between each of the above-described components and the transmission unit 27 for performing signal processing and transmission control. The terminal device 20B at the point B is configured similarly to the terminal device 20A at the point A described above.

Further, the terminal device 20 at the point A and the point B
In A and 20B, the configuration of the imaging devices 600A and 600B will be described. For convenience of description, when referring to components common to the terminal devices 20A and 20B at the point A and the point B, for example, as in the imaging device 600,
The description will be made by omitting letters A and B for distinguishing each point. In addition, unless otherwise described, it is assumed that the terminal device 20A at the point A is described in principle,
The duplicate description of the terminal device 20B at the point B having the common configuration is omitted.

The image pickup apparatus 600 includes a micro CCD camera 5 (5A) as an image pickup section and a support mechanism 10 (a first mechanism) for supporting the CCD camera 5 at a predetermined position while supporting the CCD camera 5.
An embodiment is shown in FIG. 2), and an imaging unit 6 (6A) for controlling the CCD camera 5. Monitor T
V1 (1A) is placed on the table 2 (2A), and the interlocutor 3 (3A) is at a distance L from the monitor TV1.
(M) Sitting on a chair 4 (4A) provided with a height adjustment mechanism 40 (40A) placed away. This distance L (LA)
Is set to 0.5 m or more and 5 m or less. An outer diameter of 20 mm or less is set between the person 3 to be photographed and the monitor TV 1.
A cylindrical microminiature CCD camera 5 having a length of about 100 mm is provided. The CCD camera 5 is installed so that its imaging direction is directed to the person 3 to be photographed. The image of the interlocutor 3 captured by the camera 5 is transmitted from the imaging unit 6 to the other interlocutor side as image information (transmitted to the terminal device 20B at the point B via the transmission means 27). Note that the interval between the display unit 21 and the interlocutor may be slightly shifted depending on the interlocutor. Therefore, in order to cope with such a situation, it is preferable to select a CCD camera 5 having a large depth of focus.

Next, a specific installation position of the micro CCD camera 5 will be described with reference to FIG.

First, the imaging unit 6 of the imaging device 600 is installed on the display unit 21 for displaying the transmitted image of the other interlocutor. The main body of the imaging unit 6 displays an image of the interlocutor photographed by the camera 5 (taken into the imaging unit 6 via the camera cable 50) and a transmitted image of the other interlocutor 3B. In addition to the changeover switch 6a for switching, various adjustment switches 6b are provided. Switch 2
Reference numeral 10 includes a power switch for turning on / off the power of the monitor TV1. The CCD camera 5 is installed by a support mechanism 10 (robot arm) at a predetermined position in front of a monitor screen 8 on which the transmitted image 9 of the other talker 3B is displayed.

The monitor screen 8 includes: (1) an area 81 occupying the upper central portion of the monitor screen 8 and displaying a partner image of a dialogue;
(2) areas 82a and 82b occupying the left and right sides of the area 81 for displaying non-shared information that is used only by the viewer of the monitor screen 8 and is not shown to the other party of the dialogue; (3) the areas 81 and 82a; An area 83 occupying a lower portion of the display area 82b and displaying shared information shared by both a viewer of the monitor screen 8 and a conversation partner. By arranging the area 83 below the area 81 and the areas 82a and 82b, it is possible to make maximum use of the width direction of the monitor screen 8 as a form for securing the area of the shared information section. The monitor screen 8 is preferably a screen of about 40 inches.

More specifically, the CCD camera 5 is located at a position w (cm) ahead of the monitor screen 8 and
Is arranged in the vicinity of the head 9a of the interlocutor image 9 displayed in.
The central axis of the cylindrical CCD camera 5 is located at a position h (cm) above the eye position of the interlocutor image 9 indicated by a dotted line.

As described above, since the ultra-small CCD camera 5 is located near the head above the position of the eye of the interlocutor image 9 displayed on the screen 8 of the large monitor TV 1, it is specially used for two-way conversation. No trouble occurs. For example, the CCD camera 5 is provided at about h (about 10 cm) above the eye position (the position shown by the dotted line in the figure) of the interlocutor image 9 displayed on the screen 8, and is provided between the monitor TV 1 and the interlocutor 3. In the case of a system configuration in which the distance L is set to about 2.5 (m), the parallax angle is 2.3 °.
(The distance w between the monitor screen 8 and the CCD camera 5 is 10 (c).
m) does not particularly affect the fluctuation of the parallax angle). In other words, as a result of the experiment, if the eye of the other party (the eye of the image of the other interlocutor 3B displayed on the surface 8) is clearly visible at the time of gaze coincidence, the micro-miniature camera is placed around the head. Even if 5 was provided, it was confirmed that if the screen 8 of the monitor TV 1 was large, there would be little hindrance to the two-way conversation. The size of this screen 8 is 35
It has been confirmed that a good two-way conversation can be realized if the size is about 26 cm or 26 cm or more. In addition, if the mutual interlocutors are acquaintances, a result showing that the screen size is small but the psychological disability tends to be small is obtained, and the screen size may be properly used depending on the application.

The display unit 21 may employ a plasma display or a liquid crystal display instead of the monitor TV. When such a display device is used, FIG.
As shown in (1), a structure in which the CCD camera 200 is previously incorporated in the housing 210 of the display device is preferable. 3 is the monitor screen 22 supported by the housing 210.
0, a through hole 230 is provided at a predetermined position.
A structure is provided in which the tip of the D camera 200 is directly penetrated, and the tip is located in front of the monitor screen 220.

FIG. 4 shows a specific structure. FIG. 4 is a diagram showing a cross-sectional structure of the display device along the line XX in FIG. As shown in the figure, the barrel 2 of the CCD camera 200
01 is provided with a spherical projection 202 so that it can be attached to the housing 210. An imaging device 205 and an imaging optical system 203 are incorporated in the lens barrel 201. An electric signal detected by the imaging device 205 is transmitted from the electronic circuit unit 206 to an external data processing unit via the camera cable 500. Can be Also, the lens barrel 201
Has a structure in which at least one of the lenses constituting the imaging optical system 203 can be moved in the axial direction of the lens barrel 201, and the focus of the CCD camera 200 is controlled by the adjustment knob 204. Will be

In the two-way interactive system according to the present invention, a line 5e that is perpendicular to the photoelectric conversion surface 208 of the imaging device 205 as the imaging unit and passes through the photoelectric conversion surface 208 and the three interlocutors Is within the range of the parallax angle described above. This line segment 5e is
Substantially coincides with the imaging direction. Then, the camera 200 attached to the display unit rotates the lens barrel axis of the lens barrel 201 in the direction indicated by the arrow 207 in FIG. Is moved, the imaging direction of the camera 200 (line segment 5 in the figure) is changed.
e).

The CCD camera 5 having the above structure
Is a gripping member 2 directly attached to the housing 210.
40 and 250 are attached by gripping the projection 202 of the CCD camera 5. Since the projection 202 of the lens barrel 201 has a spherical shape, the angle of the CCD camera 200 can be freely changed.
The imaging direction 5c of the camera 200 can be directed to a desired direction.

Next, the parallax angle will be described with reference to FIG. In this specification, the parallax angle is, as shown in FIG. 5, the line of sight of the photographed interlocutor 3, which is directed to the image 9 displayed on the monitor screen 8, and which is directed to the CCD camera 5. It refers to the angle θ between the line of sight of the interlocutor 3. In other words, it means a shift angle of the installation position of the CCD camera 5 with respect to the line of sight of the interlocutor 3. In addition, the line-of-sight coincidence referred to here means a conference of the Institute of Electrical Engineers of Japan in 1974 (No. 1998).
As described in the above, this means below the allowable limit of unnaturalness due to the installation position of the CCD camera 5. Quantitatively, the parallax angle is 4.5 ° or less in the left-right direction (with respect to the center A of the eye of the image 9 and both temples of the image 9), and the upward direction 12
° (on the head 9b side of the image 9 with respect to the image center A), right below 8
° (the body side of the image 9 with respect to the image center A) and below are within the visual line matching range. Of course, the smaller the parallax angle, the better, and the detection limit is 3 ° or less in the left-right and up-down directions. Therefore, the area where the CCD camera 5 can be installed is the conical area shown in FIG. Note that this conical area is defined as the center point A of the image 9 displayed on the eye 30 of the interlocutor 3 and the monitor screen 8 (in this embodiment, the center of both eyes of the image 9 is defined as the center point A of the image 9). Gaze 31 of the interlocutor 3 connecting
Is defined by the line-of-sight matching area 9b on the monitor screen 8 at a predetermined distance L (m) and the position of the eye 30 of the interlocutor 3.

On the other hand, the CCD camera 5 is set within a certain area with respect to the eye position (eg, image center A) of the image 9 of the interlocutor 3B at the point B displayed on the monitor screen 8, and
There are the following three means for placing the image 9 about 10 cm above the eye position.

The first means is to move the CCD camera 5 itself with respect to the monitor screen 8. In order to realize the first means, the interlocutor 3 displayed on the monitor screen 8
An image data processing unit 60 that performs image recognition processing on the position of the image 9 of the point 9 and the point 3 on the monitor screen 8 (whichever of the interlocutors 3A and 3B may be used); A driving mechanism for moving the CCD camera 5 is provided in the imaging unit 6 so that the CCD camera 5 is set in the above-mentioned fixed area in the monitor screen 8.

The second means is to move the person 3 to be photographed with respect to the monitor screen 8. This second
Means is that the imaging data processing unit 60 operates the monitor screen 8
The position of the image 9 of the interlocutor 3 displayed on the monitor screen 8 is subjected to image recognition processing, and the interlocutor 3 photographed with respect to the position-recognized image 9 of the interlocutor 3 itself.
This can be realized by providing a person position defining means for adjusting the position of the person. As the person position defining means, for example, a chair 4 on which the interlocutor 3 sits is preferable. Adjustment of the seat height for each interlocutor is possible by adjusting the seat height adjustment mechanism 40 provided in the chair 4. This is because the installation position of the chair 4 (distance from the monitor screen 8 and the like) is fixed in advance, so that once the interlocutor 3 sits on the chair 4, it is not necessary to adjust the position of the interlocutor 3 in the left-right direction. . In other words, the only uncertain factor for determining the position of the interlocutor is the vertical displacement caused by the physique difference between the interlocutors. Means can be realized.

The third means is to copy the image 9 of the interlocutor 3 (which may be any one of the interlocutors 3A and 3B at the point A and the point B) displayed on the monitor screen 8 to the CC whose installation position is fixed in advance.
It is moved relatively to the D camera 5. This third means, like the above means, first recognizes the position of the image 9 in the monitor screen 8 by the image data processing unit 60 provided in the image pickup unit 6, and then displays the image 9 The position needs to be moved. This allows
The CCD camera 5 can be relatively present in the fixed area.

Next, referring to the flowchart of FIG.
The initial setting operation of the D camera 5 (initial setting operation for line-of-sight matching) will be described. FIG. 9 is a diagram for conceptually explaining this setting operation.

First, the interlocutor 3 causes the CCD camera 5 to photograph itself (step ST1), and displays the self image 9 photographed by the CCD camera 5 on the monitor screen 8 of the display unit 21 (step ST2). At this time, if the installation position of the CCD camera 5 matches the line of sight of the interlocutor 3 (a line connecting the eyes of the interlocutor and the center point A of the image 9) (the parallax angle is 0).
゜), the presence of the CCD camera 5 causes a dialogue obstacle.

Then, the CCD camera 5 is moved upward by Δh.
(Step ST3), and re-install the CCD camera 5 in a position that does not cause a dialogue obstruction and is within a line-of-sight matching allowable region (for example, the region shown in FIG. 6). That is, the CCD camera 5 is installed so as to overlap the hair portion 9 b of the image 9 as viewed from the interlocutor 3.

As described above, when the installation position of the CCD camera 5 is moved by Δh, the imaging direction of the CCD camera 5 cannot be directed to the actual interlocutor 3. Therefore, the installation angle of the CCD camera 5 is corrected downward by Δθ in accordance with the movement of the CCD camera 5 (step ST4). As described above, the installation position and the set angle of the CCD camera 5 are adjusted in advance so that the parallax angle determined from the relationship between the distance L between the interlocutor 3 and the monitor screen 8 with respect to the movement amount Δh is within 3 °. , The eye-gaze matching state can be maintained.

Specifically, when the interlocutor 3 is a short child, the sitting height of the child is adjusted by the sitting height adjusting mechanism 40 of the chair 4 and the angle of the CCD camera 5 is corrected downward. It is preferable to move the image 9 itself displayed on the monitor screen 8.

On the other hand, the talker 3 displayed on the monitor screen 8
An operation of moving the image 9 of the interlocutor 3A or 3B at the point A or the point B will be described with reference to the flowchart of FIG. FIG. 10 is a diagram for conceptually explaining the image moving process. This image movement processing is performed by the imaging data processing unit 60 of the imaging unit 6 or the processing unit 26 (display control unit 285).

First, a reference point is specified from the image 9 displayed on the monitor screen 8 of the received interlocutor or self (step ST5). The reference point is a point 10 cm above the center (point A) of the eye position (indicated by a dotted line in the figure) of the image 9 or an outline of the image 9 easily obtained by binarization processing. 9 (CC)
(The position where the D camera 5 should originally exist).

Subsequently, the position 810 corresponding to the actual installation position of the CCD camera 5 is set to the absolute coordinates 80 on the monitor screen 8.
It is determined whether or not the amount of deviation between the reference point position 820 obtained from 0 and the actually obtained reference point position 820 and the actual position 810 is within an allowable range (step ST6). If the amount of shading exceeds the allowable range at this stage, it is determined that the display position of the image 9 is deviated, and the vector 830 having the coordinate 820 of the reference point as a start point and the actual position 810 as an end point Is calculated (step ST7). Then, the shifted image 9 is translated based on the vector 830 obtained as shown in FIG. 10 (step ST8).

In the present invention, any one of the above-described first to third means may be used as the means for matching the line of sight between the interlocutors via the transmission means, and these means may be used in combination. good. In the two-way conversation, the distance L of each of the chairs 4 from the monitor TV1 is fixed within a certain range, and the height of each of the chairs 4 is adjusted vertically by the seat height adjusting mechanism 40 by the above-described second means. Then, the position of the face of the image 9 of the interlocutor 3 displayed on the monitor screen 8 is present in a substantially constant area. In addition, as a result of an experiment, the inventor has confirmed that a range in which fine adjustment of a face size, an eye position, and the like for each interlocutor is within several cm to several tens cm. Therefore, for this fine adjustment, it is effective to move the display position of the image 9 of the interlocutor 3 displayed on the monitor screen 8 by the second means described above. With this display movement, it is possible to easily match the eyes of the interlocutors only by adjusting the sitting height adjustment mechanism 40 of the chair 4. Further, it is not necessary to provide a complicated moving mechanism and an optical processing device on the side of the CCD camera 5. In the embodiment described above, the installation position of the CCD camera 5 is set to a position 10 (cm) above the eye position of the image 9 of the interlocutor 3 displayed on the monitor screen 8, but the image display by the third means is performed. According to the movement, the distance h can be easily changed.

The photographing device 600 of this embodiment can be retrofitted to a TV device that is usually sold on the market. Therefore, there is no need to develop a special-purpose device as in the above-described second prior art using a display device that switches between reflection and transmission, and the above-described imaging device can be realized at low cost.
Further, there is no need for special optical components such as a half mirror and a reflection mirror as in the first and third prior arts described above. Therefore, the device configuration is simplified, and
The mirror does not darken the image of the interlocutor. Further, the problem that the screen display is disturbed as a result of enlarging the mirror to obtain a wide angle of view does not occur in the photographing apparatus according to the above-described embodiment. Therefore, according to the two-way interactive system according to the present invention, two-way interactive using the image display device having a large screen can be easily realized.

Next, a logical configuration diagram of the terminal devices 20A and 20B in the two-way interactive system according to the present invention will be described with reference to FIG. In addition, the system is aimed at a system that realizes realistic counseling between interlocutors in a remote place. In addition to a mechanism that matches the gaze between interlocutors, it is necessary to realize an ideal counseling environment. It has various functions. The configuration described below is a configuration common to both the terminal devices at the point A and the point B.

First, this terminal device is connected to the processing unit 26.
Is an integrated device including an imaging data processing unit 60. Here, the imaging data processing unit 60 takes in the imaging data from the CCD camera 5 and controls a data processing unit 600a for performing predetermined processing and a driving mechanism 600c for adjusting the set position and the set angle of the CCD camera 5. And a drive control unit 600b.

In this terminal device, initially, the monitor screen 8
Is displayed by performing a predetermined area division as shown in FIG. The initial screen 8 has a layout as shown in FIG. 12, (1) an expandable / reducible area 81 occupying the upper central portion of the monitor screen 8, displaying an image of a conversation partner, and (2).
It is used only by the viewer on the monitor screen 8 and displays a memo for the viewer, which is non-shared information that is not shown to the other party in the dialogue. It occupies the left side of the area 81, and is reduced / The right side of the area 81 where the memo board area 82a to be expanded and (3) the data search information for the viewer which is used only by the viewer of the monitor screen 8 and is not shared with the other party of the conversation and is non-shared information are displayed. And (4) occupy the lower part of the area 81 and the areas 82a and 82b, and reduce / expand the area 81 according to the expansion / reduction of the area 81. And an area 83 for displaying shared information shared by both the viewer and the conversation partner on the monitor screen 8. Here, the memo board area 82a and the data search board area 82b constitute a non-shared information area.

Also, in the vicinity of the upper frame of the screen 8, (4) the screen 8
An operation display area 84 for instructing the operation of the operation is displayed.

In this terminal device, the character / graphic input device 27
The layout and display contents of each area change according to the input information from 2 or the received transmission information.

Expansion / completion of area 81 in one terminal device
In conjunction with the reduction, the area 81 also expands / reduces in the other terminal device.

[0093] Hereinafter, in both the terminal devices during the dialogue, the occurrence of expansion / reduction of the area 81 in one terminal apparatus is notified to the other terminal as transmission information, and the screen 8 (in the area 81 in the one terminal apparatus) is transmitted. 8A or 8B) and the screen 8 (8B
Alternatively, it is assumed that a partner image of the conversation is displayed in 8A).
Such a symmetric configuration is particularly suitable for general business use and the like.

In the case of couse ring use,
It may be desirable that the expansion / reduction of the area 81 can be actively performed only by the terminal on the counselor side. In such a case, the expansion / reduction information of the area 81 is transmitted as transmission information only from the terminal device on the counselor side.

As shown in FIG. 11, the character / graphic input device 272 includes a keyboard device, a pointing device, and a pen input device (touch panel and pen) arranged near the interlocutor. Further, both the keyboard device and the pen input device can be appropriately selected.

Information input from character / graphic input device 272 is shared / input via character / graphic input control unit 286.
Input to the non-sharing determination unit 279. If it is determined that the information is shared information, the input information is output to the display control unit 285, and the input information is output to the partner terminal device via the main transmission / reception system 288 or the wireless system 287 and the transmission unit 27. Transmitted. On the other hand, when it is determined that the information is non-shared information, the input information is output only to the display control unit 285.

In the terminal device of this embodiment, the user operates the area 84 with the pointing device, selects the area 81, the area 82a, the area 82b, or the area 83, and specifies the position in each area. Such input information from the pointing device is processed as non-shared information.

When the area 81 is selected by the pointing device, the area 81 can be expanded / reduced. The distinction between the expansion and the reduction or the designation of the expansion ratio or the reduction ratio is performed by a keyboard device or a pen input device. Expansion by input information from keyboard device or pen input device /
In accordance with the designation of the reduction, the expansion / reduction of the area 81 is performed based on the reference point determined as described above, while enlarging / reducing the image of the interlocutor while matching the eyes of the interlocutors. Expansion / reduction information from the keyboard device or pen input device is processed as shared information and transmitted to the other terminal of the conversation. Upon receiving the extension / reduction information, the partner terminal performs extension / reduction of the area 81 in accordance with the extension / reduction information.

As the area 81 is expanded / reduced, the areas 82a, 82b and 83 are reduced / expanded. FIG. 13 shows a layout of the screen 8 when the area 81 is extended from the layout of FIG. 12, and FIG.
9 shows a layout of the screen 8 when the area 81 is reduced from the layout of FIG.

When the area 82a is selected by the pointing device, writing of memo information to the area 82a as a memo board becomes possible. In this state, when character information or graphic information is input from the keyboard device or pen input device, it is processed as non-shared information, notified to the monitor 1 via the display control unit 285, and the memo information is displayed in the area 82a. Thus, a memo used only for personal use can be displayed on the screen 8.

When the area 82b is selected by the pointing device, data can be searched in the area 82b as a data search board. In this state, when a search instruction is input from the keyboard device or pen input device, it is processed as non-shared information and notified to the display control unit 285. The search instruction is displayed in area 82b, analyzed by display control unit 285, and displayed in display information database 266.
Is searched, and the corresponding data is displayed in the area 82b.
Thus, data used only for personal use can be displayed on the screen 8.

When the area 83 is selected by the pointing device, writing of the shared information to the area 83 as the shared information board becomes possible. In this state, when character information or graphic information is input from a keyboard device or a pen input device, it is processed as shared information, notified to the monitor 1 via the display control unit 285, and displayed in the area 83 while the shared information is displayed. The shared information is transmitted to the partner terminal device via the transmission means. Upon receiving the shared information, the partner terminal displays the shared information in the area 83. The area 83 also displays the shared information received from the terminal device on the conversation partner side. In this way, the dialogue can proceed while referring to the shared information by both parties.

It is preferable that the regions 82a, 82b and 83 have a scroll function. In FIG. 12, a scroll instruction section is provided at the right end of each area.

[0104] Smooth dialogue is possible by counseling with characters and figures.
It is most preferable that the voice of the interlocutor can be input, recognized, converted to character information and displayed (this function is controlled by the main control unit 273 under the control of the voice / sound output). Unit 282).

The sound output unit 23 has a function 284 for outputting background music (BGM) and a sound / sound output control function 283 for automatically detecting the conversation volume and adjusting the output volume of the background music. The output is controlled by the unit 282. Upon detecting that the conversation volume is substantially zero for a predetermined time, the voice / sound output control unit 282 starts outputting background music, increases the volume of background music, or increases background music based on the detection result. Adjust volume reduction.
Further, the voice / sound output control unit 282 outputs background music that calms the mind of the interlocutor 3 without superimposing it on the voice of the interlocutor 3 without hindering the hearing of the interlocutor 3. be able to. In particular, when the silence continues, the background music is started or the volume is changed.
Is released, and the interlocutor 3 can be calmed down.

The voice input section 24 is preferably a telephoto microphone. Generally, a microphone with a table or attached to the chest can be used, but a microphone with a table may impair consciousness during counseling. Also, if the microphone at the chest is too far away, the cord may be broken. Therefore, a telescopic microphone which does not require the presence of the telescopic microphone is arranged on the
It is preferable to incorporate them into a single unit. Since the line of sight of the interlocutor 3 has already been recognized, it is possible to improve the directivity by directing the sound collecting direction to the mouth.

FIG. 15 is an explanatory diagram of a display example of the monitor screen 8 in which the display of the area 81 in FIG. 12 is enhanced. FIG.
As shown in the figure, the area 81 has a human image 8 in the first area.
In addition to 1 _1, which is identification information 81 ₂ of the person in accordance with the figures displayed, the address is a name and specific information is displayed. Further, the integrated interaction time and residual interaction time is displayed as a bar graph 81 _3.

The identification information 81 ₂ is displayed by the display control unit 285 based on the data stored in the display information database 266 or the data input from the keyboard device in accordance with the person of the display person image 81 ₁ .

[0109] The bar graph 81 ₃ shows the main control unit 273.
Is displayed by the display control unit 285 on the basis of the result of the conversation time monitoring by the user.

When a person image and a name and address corresponding to the person image are displayed at the same time, even a first-time face-to-face conversation partner can easily call by name or the like, and can easily form a friendly conversation. Also, seeing the name and face at the same time makes it easier to remember. It is also possible to display the identification number, belonging group, number of interviews, and the like of the display person.

By displaying information (character display, bar graph display, etc.) according to the accumulated conversation time or the remaining conversation time, the conversation can proceed while checking the limited conversation time. As a result, in a one-to-one one-way conversation method (lecture style), a one-to-one two-way conversation (for example, a question and answer session) with a specific person is not biased toward a specific person, and is performed in a fixed time. Dialogue management becomes easy.

Further, it is preferable to output a sound effect in conjunction with the start of the display of the human image in the first area, the end of the display of the human image, or the switching to another human image. here,
It is preferable that the sound effect is selected for each person of the displayed person image.

These sound effects are transmitted to the voice / sound output control unit 2 based on an instruction to the main control unit 273 for managing dialogue.
84 is output.

The sound effect is output in response to the start of the display of the human image, the end of the display of the human image, or the switching to another human image.
First, since a voice stimulus acts, a mental preparation time can be formed as compared with a case where a screen is suddenly generated from a silent state. The output time of the sound effect generally ranges from 2 to 15
Seconds is enough. Also, at the end, if the image suddenly disappears, loneliness may increase. Therefore, if the screen gradually ends with the sound effect, a preparation time for the mind can be formed.

The sound effect may be changed according to volume or rhythm, such as gradually increasing or decreasing. Of course, a part of the song may be used.

Furthermore, if the sound effects are properly used for each person, personality can be added to the sound effects, and the person can be identified only by listening to the sound effects.

Further, when the remaining time of the conversation with the person in the conversation reaches a predetermined time, it is preferable to output a sound for notifying the end of the dialog time or to display an image for notifying the end of the dialog time.

The voice of the end notice is sent to the voice / sound output control unit 2 based on an instruction to the main control unit 273 which performs dialogue management.
84 is output. Further, in response to an instruction to the main control unit 273 for performing dialogue management, the display control unit 285 displays and outputs the image of the end notice based on the display data stored in the display information database 266.

By the audio output and image display corresponding to the countdown of the dialogue end, the dialogue end with a time limit can be smoothly progressed. In particular, even if it is difficult to terminate the dialogue, if the visualization is performed using a fun and unique animation, such as an animal climbing a tree and eating an apple according to the countdown, together with the countdown display, it will be accepted by the interlocutors Cheap.

It is preferable to display information corresponding to the person over a predetermined period of time or to output a sound before or after the display of the person image.

The above display is output by the display control unit 285 based on the display data stored in the display information database 266 in response to an instruction to the main control unit 273 which performs the dialogue management.

The above audio output is output by the audio / sound output control unit 284 based on an instruction to the main control unit 273 which performs dialogue management.

By displaying information corresponding to the person over a predetermined period of time or outputting a voice before and after the display of the person image, it is possible to prepare for a conversation and form a mild end atmosphere. This can be combined with the sound effects described above, or a message or the like may be captured. Furthermore, it is also possible to function as an individual CM corresponding to a person image, and a variety of atmospheres can be formed.

Further, in conjunction with the switching of the person image displayed in the area 81 to the image of another person, the area 8
2a, 82b and the display of one of the regions 83
It is preferable to switch according to another person.

Such switching is performed in response to an instruction from the main control unit 273 which performs dialogue management, in accordance with the display information database 26.
6 based on the display data stored in the display control unit 6.
Do.

By automatically displaying shared information (area 83) and non-shared information (areas 82a and 82b) in accordance with the displayed person image in conjunction with the displayed person image 81 ₁ , it is possible to search for such information. Department (at least search at the beginning of the dialogue)
Becomes unnecessary. In particular, it is effective for a one-to-one two-way conversation (for example, a question-and-answer session) with a specific person in combination with a one-to-one one-way conversation system (lecture style).

In FIG. 1, two independent transmission paths are prepared as the transmission means 27, but three or more transmission paths may be provided. In addition, a coaxial cable transmission line or another wireless transmission line may be used as the transmission line in addition to the optical fiber transmission line 31 and the satellite communication transmission line 32.

Further, the two-way interactive system according to the present invention is provided with a backup system for detecting a failure of a transmission line during interactive use and automatically switching the transmission line to be used. By connecting via a plurality of transmission paths, if a transmission path failure occurs during a dialogue (or during counseling), the dialogue can be continued using another transmission path. In such a case, it is preferable that at least voice transmission be ensured in another transmission path. Generally, line abnormalities include coaxial cables, optical fiber cables,
Alternatively, since the cause is considered to be the transmission device, it is preferable that the other transmission path be a wireless system.

The present invention is not limited to the above embodiment, but can be modified. For example, it can be configured to further include an image printing device (printer 270) for printing a display screen, or a storage unit (for example, a backup D / B 264, an IC card or the like 269) for storing display screen information. is there. As the storage means, an optical card, an IC card, an optical disk, or a magneto-optical disk can be used. In this case, it is possible to save information during the dialogue, particularly information on characters and figures used during the dialogue.

[0130]

As described above in detail, according to the two-way interactive system of the present invention, while observing the expression of the other party of the dialogue while matching their eyes, the personal information is not shown to the other party. In addition to displaying the information as shared information, it is possible to establish integrated communication that enables a conversation while displaying information to be shared on both sides and sharing the information within one screen for each of the interlocutors. As a result, counseling and multimedia communication can be facilitated and progressed, shortening the time required to solve problems, and making it easier to find more creative solutions, goals, and approaches. It becomes possible.

Further, from the psychological point of view, the character / graphic information stimulus which can be simultaneously recognized while looking at the image of the interlocutor whose eyes are coincident makes it easy for human emotions / sensitivity communication and logical thinking to simultaneously function. The activity balance of the right brain and left brain can be balanced, and communication of creative problem solving type can be realized.

[Brief description of the drawings]

FIG. 1 is a logical configuration diagram showing an entire interactive interactive system according to the present invention.

FIG. 2 is a view showing the image pickup apparatus of FIG. 1 and is a view for explaining a method of installing a CCD camera (image pickup unit) at a predetermined position.

FIG. 3 is a diagram illustrating another example of the terminal device in the interactive interactive system.

4 is a view showing a mounting structure of a CCD camera of the support mechanism shown in FIG. 3, and is a cross-sectional view taken along line XX of FIG. 3;

FIG. 5 is a diagram for explaining a parallax angle.

FIG. 6 is a diagram showing an area where a CCD camera is to be installed.

FIG. 7 is a flowchart illustrating an initial setting operation of the CCD camera.

FIG. 8 is a flowchart for explaining an operation for moving an image displayed on a monitor so that the lines of sight coincide with each other.

FIG. 9 is a diagram conceptually illustrating an initial installation operation of the CCD camera described in the flowchart of FIG. 7;

FIG. 10 is a diagram conceptually illustrating an operation of moving an image displayed on a monitor, which is described in the flowchart of FIG. 7;

FIG. 11 is a logical block diagram showing an overall configuration of the terminal device shown in FIG.

FIG. 12 is a diagram showing a screen configuration example of a display screen displayed by the monitor of FIG. 1;

13 is a diagram showing a screen configuration example of a display screen displayed by a monitor after an area 81 in FIG. 12 is extended.

14 is a diagram showing a screen configuration example of a display screen displayed on a monitor after the area 81 in FIG. 12 is reduced.

FIG. 15 is a diagram showing a screen configuration example of a display screen displayed by the monitor of FIG. 1;

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Monitor TV, 2 ... Table, 3 ... Interactor, 4 ... Chair, 5, 200 ... Imaging device (CCD camera), 6 ... Imaging unit, 8 ... Screen, 9 ... Interviewer image, 10, 110 ... Supporting mechanism , 20 terminal device, 21 display unit, 23 speaker, 24 microphone, 25, 272 character / graphic input device,
26 processing unit, 27 transmission means, 60 imaging data processing section, 81 interlocutor image area, 82 non-shared information area, 83 shared information area, 84 operation area, 271
Shared / non-shared determination unit, 275 dialogue processing unit, 281, 60
0c drive mechanism, 269 recording medium, 279 voice / sound data processing unit, 282 voice / sound output control unit, 28
5 display control unit, 600, 601 imaging device.

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-62-269128 (JP, A) JP-A-4-213287 (JP, A) JP-A-4-97663 (JP, A) JP-A-4-97 279915 (JP, A) JP-A-5-37927 (JP, A) JP-A-6-205409 (JP, A) JP-A-7-95549 (JP, A) JP-A-6-90444 (JP, A) JP-A-7-123131 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) H04N ^7/ 14-7/173 H04N 5/222-5/225

Claims (34)

(57) [Claims] A first terminal installed at a first point where a first interlocutor is located, and a first terminal installed at a second point where a second interlocutor is located a considerable distance from the first point. A second terminal, and transmission between the first terminal and the second terminal for enabling bidirectional communication of transmission information having image data of at least the first interlocutor and the second interlocutor; In a two-way interactive system provided with means, at least the first terminal device is a display unit for displaying an image of the second interlocutor received on the transmission means on a monitor, In the vicinity of the top of the head, a part of the human image corresponding to the position where the imaging unit is actually installed is detected, and in a state where the detected part of the human image is fixed, the entire human image is Enlarge or reduce, and the In a second region other than the first region object image is displayed, to display at least one of the second character and graphics to be displayed simultaneously on the terminal device, the first on the monitor
A display unit including a display control unit that displays at least one of a character and a graphic that is not displayed on the second terminal device in the third area other than the second area or the third area, and a monitor of the display unit A position defining unit that defines a position of the first interlocutor with respect to a screen; and an apparatus that enables transmission of the image of the first interlocutor to the second terminal device via the transmission unit, An imaging unit for directly imaging the first interlocutor; and an imaging unit positioned at a predetermined position between the first interlocutor and the monitor with the imaging direction of the imaging unit directly facing the first interlocutor. An imaging device having a support mechanism for setting the distance between the first interlocutor and the monitor specified by the position defining unit, and the line of sight of the first interlocutor. Judge that the line of sight of the monitor image of the second interlocutor matches Bidirectional interactive system, characterized in that installed in a predetermined area defined predetermined parallax angle, and by that. 2. The interactive interactive system according to claim 1, wherein the second area and the third area are reduced or enlarged according to expansion or reduction of the first area. . 3. The system according to claim 1, wherein the first area of the second terminal expands or contracts in conjunction with the expansion or contraction of the first area in the first terminal device. A two-way interactive system as described. 4. The monitor according to claim 1, wherein the first area is located above and at the center of the monitor, the second area is located below the first area on the monitor, and the third area is located on the monitor. 2. The interactive system according to claim 1, wherein the interactive system is arranged on a side of the first area on the monitor. 5. The transmission information further includes character information or graphic information, and the character or graphic is displayed in the second area of the display unit based on the transmitted character information or graphic information. The eye-gaze matching type interactive system according to claim 1, wherein: 6. The first terminal device further includes a character / graphic input device for inputting the character or the graphic, and the character or the graphic input from the character / graphic input device is provided on the display unit. The character information or the graphic information corresponding to the character or the graphic displayed in the second area or the third area is displayed in the second area or the third area, and directed to the second terminal device. The eye-gaze matching type interactive system according to claim 1, wherein the system outputs the information. 7. The character / graphic input device, a voice recognition device for inputting and recognizing a voice of the first interlocutor and converting the input into character information, a keyboard device arranged near the first interlocutor, The eye-gaze-matching interactive system according to claim 6, further comprising at least one of a pen input device and a pen input device arranged near the first interlocutor. 8. The eye-gaze matching type interactive system according to claim 6, wherein said character / graphic input device includes a pointer input unit for inputting a pointer position indicating a position of a displayed screen. 9. The apparatus according to claim 1, wherein the imaging unit is in the predetermined area defined by the distance specified by the position defining unit and the predetermined parallax angle, and the second area as viewed from the first interlocutor. 2. The interactive system according to claim 1, wherein the interactive system is installed at a position where a part of the monitor image of the interlocutor is blocked. 10. The parallax angle that matches the line of sight of the first interlocutor with the line of sight of the monitor image of the second interlocutor is the first parallax angle.
The monitor screen according to claim 1, wherein the horizontal direction of the monitor screen is 4.5 ° or less, the overhead direction is 12 ° or less, and the downward direction is 8 ° or less, as viewed from the first interactor. 1
A two-way interactive system as described. 11. The parallax angle that matches the line of sight of the first interlocutor with the line of sight of the monitor image of the second interlocutor is the first parallax angle.
The two-way interactive system according to claim 10, wherein the left-right direction and the up-down direction of the monitor screen are each 3 degrees or less with respect to the line of sight of the interlocutor as viewed from the first interlocutor. 12. The first terminal device further comprises a processing unit for recognizing a position on the monitor of the image of the first interlocutor displayed on the monitor, and an imaging device of the first terminal device. Comprises a driving mechanism for moving the position of the imaging unit or changing the imaging angle of the imaging unit relative to the first interlocutor whose position is recognized by the processing unit. The interactive system of claim 1. 13. The second terminal device, wherein at least a second point imaging unit that images the second interlocutor, and a position of the second point imaging unit is moved, or the second point imaging unit A second point driving mechanism for changing an imaging angle, wherein the processing unit of the first terminal device is configured to display, on the monitor, an image of the second interlocutor received via the transmission unit and displayed on the monitor. Transmitting the control signal for recognizing the position and moving the position of the second point imaging unit or changing the imaging angle of the second point imaging unit relative to the second interlocutor; 13. The interactive system according to claim 12, wherein the data is transmitted to the second terminal device via a means. 14. The first terminal device further comprises a processing unit for recognizing a position on the monitor of the image of the first interlocutor displayed on the monitor, wherein the position defining unit performs the processing. 2. The apparatus according to claim 1, further comprising a mechanism for adjusting a position of the first interlocutor's own eye photographed with respect to an eye position of the monitor image of the first interlocutor recognized by the unit. Two-way interactive system. 15. The second terminal device, wherein at least a second point display unit that displays an image of the first interlocutor on a monitor, and a unit that defines a position of the second interlocutor, wherein the monitor image A second point position defining means having a mechanism for adjusting the position of the second interlocutor's eye with respect to the position of the second interlocutor, wherein the processing unit of the first terminal device is received via the transmission means. Recognizing the position on the monitor of the image of the second interlocutor displayed on the monitor, and recognizing the position of the eye of the second interlocutor on the monitor image of the second point display unit. The control signal for adjusting a position is transmitted to the second terminal device via the transmission unit.
4. The interactive system according to claim 4. 16. The interactive system according to claim 1, wherein the support mechanism of the imaging device includes a transparent member for directly supporting the imaging unit. 17. The two-way interactive system according to claim 1, wherein said position defining means is a chair provided with a seat height adjusting mechanism. 18. The bidirectional apparatus according to claim 1, wherein said position defining means is a table having a predetermined height, which is provided between said first interlocutor and said display unit. Interactive system. 19. The interactive interactive system according to claim 1, wherein said position defining means is a table having a predetermined height on which said display unit is directly installed. 20. The display unit, wherein the display unit has a through hole communicating from a back surface of the display unit to a monitor screen on a front surface of the display unit,
The imaging unit is attached to the display unit by the support mechanism, with a tip of the imaging unit penetrating a through hole of the display unit and a tip of the imaging unit protruding forward of the monitor screen. The interactive system of claim 1, wherein 21. The imaging unit is attached to the display unit by the support mechanism in a state where the imaging unit is rotatable about a predetermined portion of the imaging unit. 21. The interactive system of claim 20, wherein the direction is changed. 22. The interactive system according to claim 21, wherein the first terminal device further comprises a drive mechanism for changing an attachment angle between the display unit and the imaging unit. 23. The first terminal device includes a processing unit that adjusts a display position of the image of the second interlocutor displayed on the monitor, and the processing unit receives the image via the transmission unit. Based on the image of the second interlocutor displayed on a monitor, a virtual position on the monitor image where the imaging unit should originally exist is specified as a reference point, and a position where the imaging unit is actually installed Detecting the displacement of the monitor image of the second interlocutor based on the coordinates of the reference point obtained on the monitor and the coordinates of the obtained reference point on the monitor, and calculating the coordinates of the reference point on the monitor. Calculating a vector with the coordinates on the monitor corresponding to the installation position of the imaging unit as an end point as a start point, and translating the monitor image of the second interlocutor based on the obtained vector. Features Bidirectional interactive system of claim 1, wherein that. 24. The processing unit detects each display position on the monitor of both eyes of the image of the second interlocutor received via the transmission means and displayed on the monitor, 24. The interactive system according to claim 23, wherein a position on the monitor, which is separated from the center of both eyes by a predetermined distance above the monitor, is specified as the reference point. 25. The processing unit specifies, as a reference point, a position on the monitor corresponding to the top of a monitor image of the second interlocutor received via the transmission means and displayed on the monitor. 24. The interactive system of claim 23, wherein: 26. The first terminal device further displays, in the first area, a name, an identification number, or unique information, which is identification information of a person corresponding to the person image, in addition to the person image. The interactive interactive system according to claim 1, further comprising a display control unit. 27. The first terminal device further displays, in the first area, information according to an accumulated conversation time or a remaining conversation time with the person of the displayed person image in addition to the person image. The interactive interactive system according to claim 1, further comprising a display control unit that performs the operation. 28. The first terminal device, in addition to the voice based on the transmitted voice information, starts displaying the human image in the first area, ends displaying the human image, or performs other processing. 2. The interactive interactive system according to claim 1, further comprising a voice output control unit that further outputs a sound effect in conjunction with the switching to the person image. 29. The interactive interactive system according to claim 28, wherein the sound effect is selected uniquely for a person of a displayed person image. 30. When the remaining time of the conversation with the person who is interacting in the first area reaches a predetermined time in addition to the voice based on the transmitted voice information, the first terminal device ends the conversation time. The two-way interactive system according to claim 1, further comprising a voice output control unit that outputs a voice for notifying the user of the notification. 31. The first terminal device displays, in addition to the person image, an image for notifying the end of the conversation time when the remaining time of the conversation with the person in conversation in the first area reaches a predetermined time. The interactive interactive system according to claim 1, further comprising a display control unit that performs the operation. 32. The first terminal device further comprises a storage device for storing information on a candidate person to be interacted with, and before displaying the person image in the first area or ending the display. 2. The interactive interactive system according to claim 1, further comprising a display control unit for displaying information corresponding to the person stored in the storage device for a predetermined time. 33. The first terminal device further includes a storage device for storing information on a candidate person to be interacted with, and before displaying the person image in the first area or ending the display. 2. The apparatus according to claim 1, further comprising a voice output control unit configured to voice output information corresponding to the person stored in the storage device for a predetermined time.
A two-way interactive system as described. 34. Display of either one of the second area and the third area in conjunction with switching from a person image displayed in the first area to a person image of another person. 2. The interactive interactive system according to claim 1, further comprising a display control unit that switches according to the other person.