JPH09130768A - Two-way interactive system provided with device matching lines of sight of interactive parties via transmission means, terminal equipment and image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide the 2-way interactive system provided with a device realizing matching of sight of lines between interactive parties via a transmission means inexpensively with a simple configuration without the need for a special structure and device to ensure lines of sight of the interactive parties. SOLUTION: A terminal equipment 20 consisting of a display section 21 provided at least with a monitor 1 to display an image of an opposite party, a position regulation means 4 regulating a position of an interactive party 3 itself with respect to a monitor screen, and an image pickup device 600 having an image pickup section 5 picking up the image of the interactive party itself and a support mechanism to set the image pickup section at a prescribed position between the interactive party 3 and the monitor 1. The image pickup section 5 is installed at a position especially within a sight of line matching area defined by the distance between the interactive party 3 whose position is specified by the position regulation means 4 and the monitor 1 and a parallax angle used for discriminating the matching of sight of lines between the interactive party 3 and the monitor image of the opposite party to interrupt part of the monitor image of the opposite party to the interactive party 3.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a two-way interactive system having a mechanism for matching the lines of sight of interlocutors via transmission means, as well as a terminal device and an image pickup device constituting the system. . Specifically, it is a counseling system that realizes a two-way dialogue through a transmission means such as a relationship between a doctor and a patient who are far away from each other and a counselor and a client. The present invention relates to a system aimed at and devices constituting the system.

[0002]

2. Description of the Related Art Counseling conventionally performed in fields such as education, medical care, and psychology is generally performed in a situation where the interlocutors face each other directly while establishing a close relationship of trust between them. However, those who have counseling skills with those who need counseling due to natural disasters or remote areas (counselors)
There are situations where it is difficult to share the same place and time with the same. Therefore, with the development of communication technology, the need for a system that enables counseling between people in remote areas has increased.

In order to construct such a counseling system using communication technology, in order to build a close relationship and an ideal counseling environment between interlocutors as in the conventional face-to-face method, First, it is important to match the eyes of the interlocutors.

In view of the above, a conventional image pickup apparatus that matches the line of sight between interlocutors via a transmission line is disclosed in
-269128 (the first prior art). In the first prior art, a half mirror provided between the interlocutor and the image capturing apparatus is provided, so that light emitted from the interlocutor serving as a subject is transmitted to the image capturing apparatus and the image display apparatus. Are separated. Therefore, according to the device configuration of the first prior art, the imaging device is arranged at a position avoiding the display screen of the image display device viewed by the interlocutor, and the input optical axis of the imaging device and the line of sight of the interlocutor. And match.

Secondly, Japanese Patent Application Laid-Open No. Hei 4-213287 (second prior art) uses a special display device capable of switching between reflection and transmission of light, and a rear portion of the display device (this display device is referred to as a second display device). An imaging device having a configuration in which an imaging device is arranged on the side opposite to an interlocutor) is disclosed. According to this device configuration, the line-of-sight of the interlocutor is directed toward the image-capturing device located behind the display device and thus behind the display screen of the display device seen by the interlocutor. It is possible to match the line of sight.

Thirdly, Japanese Patent Application Laid-Open No. 4-977663 (third prior art) discloses that a small mirror or a small half mirror is mounted on the upper part of a display device, and a small camera whose optical axis is directed to the mirror is used as an interlocutor. There is disclosed an imaging apparatus configured to capture images. In this device configuration, the interlocutor image reflected by the mirror is captured by a small camera arranged at a position avoiding the display screen of the display device seen by the interlocutor. Therefore, the elevation angle of the line-of-sight of the talker is suppressed within 10 ° which is the allowable range of line-of-sight match in the vertical direction, and the line-of-sight of the talkers can be matched.

[0007]

In the first prior art in which the light is separated from the subject by the half mirror, a half mirror having a size comparable to the screen size of the display device is required. Inevitably a large space is needed to accommodate such a large half mirror. In particular, when the screen size exceeds 40 inches, it cannot be easily manufactured. Also, because I use a half mirror,
The image captured by the imaging device installed at a position away from the screen where the interlocutor image is displayed becomes dark, and an ideal counseling environment cannot be realized.

Next, the second conventional technique using a special display device for switching light reflection / transmission cannot be incorporated into a commercially available TV device or the like which has already been mass-produced.
Therefore, in order to realize a system using this principle, it is necessary to develop a dedicated machine, and the device price will be high. Further, the image pickup device for picking up an image of the interlocutor is arranged to take an image of the interlocutor through the special display device. Therefore, the image of the imaged interlocutor becomes dark, and an ideal counseling environment cannot be realized.

Further, in the third prior art in which the reflecting mirror and the small camera are combined, the camera is arranged at a position avoiding the display screen seen by the interlocutor, and the interlocutor is imaged through the small reflecting mirror. As a result, the image of the person being photographed becomes dark. Further, in order to obtain a wide angle of view, the size of the reflection mirror needs to be equal to or larger than the size of the small camera, and the display of a human image on the display device is hindered by the large reflection mirror. Also, when the position of the person to be photographed changes, the imaging direction of the small camera is constant, so
It is not possible to follow this. Therefore, the line-of-sight agreement between the interlocutors cannot always be ensured.

The structural feature common to the above-mentioned prior art is that a display screen (a screen on which an image of the other party's interlocutor is displayed) viewed by the interlocutor is present between the imaging device (including the camera) and the interlocutor. The point is that it is placed in a position avoiding the space. In other words, in order to match the line of sight between the interlocutors, it is necessary to install a reflection mirror, a half mirror, and a special display device between the interlocutor and the imaging device. Therefore, the transmitted image of the interlocutor on the other side becomes dark and an ideal counseling environment cannot be realized.

The present invention has been made in order to solve the above problems, and as a counseling system, a transmission means is used to reduce discomfort during a dialogue between interlocutors at distant places. It is an object of the present invention to provide a two-way interactive system having a mechanism for matching the lines of sight between interlocutors and a device constituting the system. In particular, the two-way interactive system according to the present invention integrally processes voices (including background music (BGM)), character / graphic information, and the like in addition to the interlocutor's video, and creates an organic relationship between the interlocutors. Creation (realization of close communication) and the provision of an ideal counseling environment.

[0012]

According to the present invention, there is provided a first terminal device installed at a first point where a first interlocutor is located, and a point which is a considerable distance from the first point and which is located between the second interlocutor. At least two-way communication of the image / sound data of the first interlocutor and the second interlocutor is performed between the second terminal device installed at the second point and the first terminal device and the second terminal device. It relates to a two-way interactive system with transmission means for enabling. In this system, at least the first terminal device defines the display unit for displaying the image of the second interlocutor on the monitor, which is received via the transmission unit, and at least the distance between the first interlocutor and the display unit. A position defining means for defining the position of the first interlocutor, and an imaging device for transmitting the photographed image of the first interlocutor to the second terminal device via the transmitting means.

[0013] Here, the above-mentioned image pickup device, at least, an image pickup section for picking up an image of the first interlocutor, and the first interlocutor with the image pickup direction of the image pickup section directly directed to the first interlocutor. A support mechanism for setting the image pickup unit at a predetermined position between the monitor and the monitor. In particular, (1) the image capturing unit includes a distance between the first interlocutor defined by the position defining unit and the monitor, and
(2) The parallax angle described above in which it is determined that the line of sight of the first interlocutor and the line of sight of the monitor image of the second interlocutor are the same (1)
And within the line-of-sight matching area defined by the elements (2) and (2), the first interlocutor is installed at a position where a part of the monitor image of the second interlocutor is blocked. The second terminal device may also have the same configuration, and the first terminal device may be the second terminal device.
The terminal device can be remotely controlled. further,
Each terminal device is provided with a sound absorbing member that is arranged around the interlocutor and has a property of blocking external sound. Therefore, the psychological obstacle to the leakage of the dialogue contents to the outside is reduced, and the sound quality is improved, so that the user can concentrate his / her feelings and a deeper dialogue is possible.

As described above, the reason why the system is constructed by installing the image pickup unit so as to block a part of the monitor image in the above-mentioned line-of-sight matching area between the interlocutor and the monitor is Even if the image pickup unit is installed so as to block a part of the human image displayed on the monitor, if the size of the image pickup unit is suppressed below a certain level, the presence of this image pickup unit is bidirectional. This is because the inventor confirmed by experiments that it had almost no effect on dialogue.

In the above interactive interactive system, the parallax for matching the line of sight of the interlocutor (first interlocutor) and the line of sight of the person (second interlocutor) displayed on the monitor (monitor image). Regarding the angle, with respect to the line of sight of the interlocutor, when viewed from the interlocutor, the horizontal direction of the monitor screen is 4.5 ° or less, and the direction directly above is 1 °.
It is preferable that the direction of 2 ° or less and the direction directly below is 8 ° or less. Further, the parallax angle is set such that the horizontal direction and the vertical direction of the monitor screen are each 3 ° or less with respect to the line of sight of the interlocutor when viewed from the interlocutor. Is more preferable. In particular, as shown in FIG. 8, the angle formed by a line perpendicular to the photoelectric conversion surface 208 of the imaging device 205 and passing through the photoelectric conversion surface 208 and the first interlocutor is within the range of the parallax angle described above. It is.

Further, it is preferable that the line-of-sight matching region in which the image pickup unit is installed is a region higher than the line-of-sight of the interlocutor to be photographed. In particular, as a result of the experiment by the inventor, the imaging unit that blocks between the interlocutor and the monitor is preferably located near the top of the human image displayed on the monitor as viewed from the interlocutor as the installation position of the imaging unit. Was confirmed.

The distance from the interlocutor specified by the position defining means to the monitor, and a predetermined parallax angle at which it is determined that the line of sight of the interlocutor and the line of sight of the person image displayed on the monitor match. The first means for installing the image pickup unit in the area defined by
Means for moving the imaging unit itself within the line-of-sight matching area between the interlocutor and the person image on the monitor; and second, by moving the interlocutor himself facing the monitor, Means for moving the line-of-sight matching area itself to relatively change the positions of the image pickup unit and the line-of-sight matching area, and third, moving the image of the person on the monitor, similarly, the line of sight of the interlocutor facing the monitor And means for moving the line-of-sight matching area of the human image on the monitor itself (this third means is effective as an algorithm for moving the video display area in which the human image is displayed when the screen is divided by the display control unit described later). ).

The first means provides a processing unit for recognizing a position of a human image displayed on a monitor on the monitor in the terminal device, and at the same time, installs a position of an image pickup unit in the image pickup device. This can be realized by providing a drive mechanism that moves the human image whose position is recognized by the processing unit relatively. As an embodiment, the first means is performed for each terminal device, and the position of the monitor image of the second interlocutor is recognized by the processing unit of the first terminal device, and the processing unit issues a predetermined control signal. The second
In some cases, the first terminal device may remotely control the second terminal device by transmitting to the terminal device.

The second means is provided with a processing unit for recognizing the position on the monitor of the person image displayed on the monitor in the terminal device, and for detecting the position of the person image on the monitor unit. This can be realized by providing the position defining means with a mechanism for adjusting the line of sight of the interlocutor who is photographed with respect to the line of sight. As an embodiment, the first means is performed for each terminal device, and the position of the monitor image of the second interlocutor is recognized by the processing unit of the first terminal device, and the processing unit issues a predetermined control signal. The second
In some cases, the first terminal device may remotely control the second terminal device by transmitting to the terminal device.

Further, in order to realize the above-mentioned third means,
The terminal device includes a processing unit that adjusts the display position of the person image displayed on the monitor. The processing unit specifies a virtual position on the monitor image where the imaging unit should originally exist as a reference point based on the human image received and displayed on the monitor, and then the imaging unit actually Based on the coordinates on the monitor corresponding to the installed position and the coordinates of the obtained reference point on the monitor, the displacement of the human image displayed on the monitor is detected. If it is determined that this person image is deviated from the display position on the monitor where it should be displayed, the coordinates of the reference point on the monitor are used as the starting point, and the monitor corresponding to the installation position of the imaging unit is used. A vector having the above coordinates as the end point is calculated, and the shifted human image is translated on the monitor based on the obtained vector. As an embodiment, when performing on the monitor image of the interlocutor himself (initial setting before the start of the dialogue)
May be performed on the monitor image of the second interlocutor. In particular, the display size of the person image (the monitor image of the first dialogue person or the second dialogue person) is arbitrarily set during the screen division operation (including the enlargement / reduction operation of the displayed person image) by the display control unit described later. When changed, the area for displaying the image can be set at an arbitrary position on the monitor screen. Therefore, when the line-of-sight match is made between the person image whose display size is arbitrarily changed and the interlocutors, The method of 3 is particularly effective.

The reference points detect the respective display positions on the monitor of both eyes of the person image received and displayed on the monitor, and the positions above the monitor from the center of both eyes. The position is set on the monitor at a predetermined distance (for example, about 10 cm). Further, this reference point is obtained by binarizing the received human image displayed on the monitor, detecting the contour of the human image, and specifying it as the position on the monitor corresponding to the crown of the human image. Good.

The drive mechanism for moving the image pickup unit once recognizes the position of the interlocutor in the processing unit, and based on the recognized position of the interlocutor and the position of the image pickup unit, determines the angle of the image pickup direction of the image pickup unit. It may be configured to adjust. By adjusting the angle of the imaging unit by the driving mechanism, the imaging direction of the imaging unit is automatically directed to the line of sight of the interlocutor whose position has been recognized by the processing unit. Further, the drive mechanism for moving the imaging unit detects an average match in the line-of-sight direction, and when moving and aligning, the moving speed is equal to or less than a predetermined value, and the centering operation is not performed when the fluctuation in the line-of-sight direction is within a predetermined range. (This determination is made by a processing unit or the like). In addition, when the imaging unit moves, the image output from the imaging unit shakes. However, the movement speed is limited, and the movement is not performed frequently and the movement is suppressed to the minimum necessary. .

Here, in the above-mentioned drive mechanism for moving the image pickup section, it is preferable that a part of a member located between the interlocutor and the display section is formed of a transparent material.

Next, the position defining means for defining the position of the interlocutor includes a height-adjustable chair (a chair having a seat height adjusting mechanism) installed at a predetermined distance from the display section. Also,
A table provided between the interlocutor and the display unit also functions as a position defining unit. In this system, this position defining means is used, and the interactive person to be photographed and the display unit are 0.5 m or more 5
Separated by a distance of m or less. With such a configuration, since the position of the interlocutor can sufficiently limit the movement of the interlocutor in the front, back, left, and right directions, the line-of-sight agreement between the interlocutors in the system is caused by a difference in the physical constitution of the interlocutor. This is because the vertical movement adjustment is most effective.

The image pickup section adapted to the system may be independent of the display section as shown in FIG. 2, for example. Further, as shown in FIG. 7, it may be attached to a part of the display unit by a support mechanism.

The two-way interactive system according to the present invention is
Since it aims to provide a counseling system for interlocutors in remote areas, it has various configurations and functions in order to collect useful counseling data and realize an ideal counseling environment.

For example, the display control unit for controlling the display on the display unit has a function of enlarging / reducing the image displayed on the display unit. The display size has a great influence on the psychological effects of the interlocutor. Therefore, by freely setting the size of the video display area on the monitor with this display control unit, the degree of freedom of conversation increases and a comfortable conversation is possible. The environment can be realized. Since the display position on the monitor of the reduced person image (monitor image) is adjusted by the above-described third line-of-sight matching means, the line-of-sight matching state between the first interlocutor and the second interlocutor is maintained. To be done.

The transmission information may be character / graphic information in addition to the image and voice of the interlocutor. With this, it becomes possible for the display unit to display a character or a graphic together with a person image based on the character / graphic information received via the transmission means. That is, by displaying not only images and sounds but also characters and figures, the dialogue can be further facilitated.

As a method of displaying characters or figures, the characters may be superimposed on the displayed person image. Further, the monitor screen may separate the area for displaying characters or figures from the area for displaying a person image. As a mode of this screen separation, (1) the monitor screen is divided in the vertical direction and the person image is displayed on the upper side, or (2) the monitor screen is divided in the horizontal direction and the one side or the central part is displayed. A mode such as displaying a person image is preferable. The inventor has confirmed that the line-of-sight agreement between the interlocutor and the human image does not deteriorate even in the modes (1) and (2). As an application example of this aspect, the display unit may have a plurality of display units, such as a first monitor that displays a person image and a second monitor that displays characters, figures, and the like.

Further, in this system, a character / graphics input device is provided as a man-machine interface. With this configuration, the display unit can display the character or graphic input from the character / graphic input device through the character / graphic input control unit, and the character corresponding to the character or graphic thus captured in the device. It is also possible to output the information or the graphic information to the other terminal device. Note that such a character / graphic input device may be provided only at a terminal device at either the first location or the second location where the interlocutor is present.

As described above, since the system is provided with the character / graphics input device, it is possible to have a dialogue with characters and figures, and a smoother dialogue is possible. As such a character / graphics input device, a keyboard device, a pen input device (touch panel) arranged in the hands of an interlocutor
Can be adopted. Further, by providing both such a keyboard device and a pen input device so that the keyboard device and the pen input device can be appropriately selected, a smoother conversation can be realized. Further, the character / graphics input device may include a pointing device for inputting a pointer position indicating a displayed character or graphic. The position indication by the pointer makes it possible to specify the conversation target at each time. By using such a character / graphic input device or the like and recognizing the input voice of the interlocutor, converting it into character information and displaying it, it is effective for establishing a smooth dialogue (or counseling).

The two-way interactive system has a speaker as a voice output unit, which is arranged on the side of the interlocutor and near the ear. This is because whispering at the ears improves the intimacy. A headphone is typical as the audio output unit.

Next, the system has a telephoto microphone as a voice input section. It is also possible to adopt a microphone with a table or a microphone attached to the chest, but the microphone with a table may disturb the consciousness of dialogue. Also, if the chest microphone is too far away, the cord may be damaged. Therefore, it is desirable to dispose a telephoto microphone on the top of the display or to incorporate it into the display so that it is not necessary to be aware of its presence. Especially, since the line of sight has already been recognized,
By directing the sound collection direction toward the mouth, it is possible to improve the directivity.

The voice / acoustic data processing unit for fetching a voice signal from the voice input unit has a voice input amplification / adjustment function for controlling the collected sound volume. Here, the voice / sound data processing unit automatically adjusts the output volume of the collected voice to a predetermined level. The voice / sound data processing unit is configured to detect an average level of the collected sound volume in a predetermined time and to automatically adjust the sound volume when a difference between the collected sound volume and the average level is equal to or more than a predetermined value. May be.

At this time, the display section displays the output volume of the voice / sound data processing section (volume of the interlocutor himself output to the other party). The act of speaking to a person on the screen while the microphone is not directly visible makes it difficult to judge how much you should adjust your volume if you are not familiar with it. Planned. Further, by displaying the output volume of the interlocutor on the monitor screen so that the interlocutor can recognize the output volume, it is possible to feed back to the interlocutor himself whether or not the output volume is appropriate for transmission.

On the other hand, the voice / sound output control unit for controlling the voice output unit outputs background music (BGM) and
It has a function of detecting the conversation volume and adjusting the output volume of the background music. Here, the voice / sound output control unit detects that the conversation volume is substantially zero for a predetermined time, and based on the detection result, at least starts the background music, adjusts the volume of the background music, and adjusts the background music. Adjust the volume of music. Further, the voice / sound output control unit detects the volume of the voice of the interlocutor and adjusts the volume of the background music so as not to hinder the hearing of the interlocutor's voice, thereby impressing the interlocutor's heart. The calming background music may be superimposed on the voice of the other party's interlocutor and output from the voice output unit. In particular, when background music is started or the volume is changed when silence continues, the interlocutor's tension can be released and the interlocutor can be calmed down.

Further, in the bidirectional interactive system,
Each terminal device includes a printer that prints the display screen of the monitor, and a storage unit that stores information on the display screen. The storage means is a card-shaped recording medium including at least an IC (integrated circuit), and for example, an optical card, an IC card, an optical disk, or a magneto-optical disk can be used, and information during the dialog, particularly during the dialog, can be used. It is possible to save the information of used characters and figures.

In order to maintain the security of the system, it is preferable that each photographing device is provided with a person authentication section for confirming the person of the interlocutor. This is because the security of the system can be sufficiently maintained by starting the interactive operation after the person confirmation. Here, the person authentication section may be configured to read the authentication data from a dedicated reading device from the storage medium in which the authentication data is written, and perform person verification based on the read result. The storage medium is
A card-shaped recording medium including at least an IC, and, for example, an optical card, an IC card, or a magnetic card can be used. Furthermore, this person authentication unit (1) performs person confirmation based on the analysis result of the interlocutor's face, (2) performs person confirmation based on the voice analysis result of the interlocutor's voice, or (3) The person confirmation may be performed based on the character analysis result of the signature character, and further, (4) the person confirmation may be performed based on the sentence analysis result of the message text of the interlocutor. By such person authentication, the interlocutor can be identified. It is preferable to properly use the authentication method according to the confidential content. Note that the sentence analysis of the message text is, for example, for the text "X is Y."
It refers to the collation and analysis of the characters entered in X and Y with the person's database and the designated person's cryptographic database.

Further, by using an external recording medium in the form of a card such as an IC card, it is possible to reserve a dialogue, authenticate a designated other party's interlocutor, reserve a dialogue time, and preset a dialogue time. Authentication or control of dialogue conditions may be performed. Of course, it can be used as a means of electronic payment of charges. Also, if the recording medium is used for summarizing conversation contents and recording homework items, it is convenient for the next counseling and communication. If data such as a dialogue start time and a dialogue partner (other party's dialogue person) is input in advance in a recording medium such as an IC card, the other party's terminal device is searched for on the network such as an electronic bulletin board system based on the data. Based on the search result, wherever the first interlocutor and the second interlocutor are, a line connection is established between the terminal devices at the specific first point and second point, and the first and second interlocutors are connected. It is possible to build a dialogue environment between the interlocutors. Further, the name, face (image data), and conversation items (the subject of the conversation, etc.) of the other party's interlocutor are input to the recording medium (IC card, etc.), and these information are stored for a predetermined period (before the start of the dialogue). By displaying the information on the monitor of the interlocutor for only 30 seconds (for example, about 30 seconds), mental preparation for the interlocutor's own dialogue is made, and it becomes easy to enter the subject of the dialogue. At this time, if BGM, voice, and the like are output in conjunction with each other, it is possible to provide a more comfortable dialogue environment for the interlocutor. It is effective to perform the above-mentioned monitor display during and / or after the dialogue as required. In particular, by performing the above-mentioned monitor display after the end of the dialogue, the interlocutor can smoothly proceed with the summarization of the dialogue and the like (the time of the monitor display after the end of the dialogue is different from the end of the dialogue for the interlocutor). It will be a mental preparation period until work).

Next, the above-mentioned table as the position defining means is arranged between the interlocutor and the display unit so that one end faces the interlocutor and the other end is substantially in contact with the display unit. .
Here, the table is set to a height where (1) the interlocutor is higher than the table above the chest when sitting in a chair,
(2) The height above the chest may be set to be higher than the table with the interlocutor standing. With such a configuration, a sense of continuity is generated through the table, and anxiety caused by not knowing the relative distance (psychological distance) to the other party is reduced. In this case, it is desirable that the display unit and the table are in contact with each other, but there may be a gap (for example, several tens cm or less) that does not cause a psychological obstacle.

Next, in the bidirectional interactive system, the terminal device includes an interactive processing unit for statistically analyzing the interactive time of the interlocutor. Here, the dialogue processing unit calculates the total dialogue time, the average dialogue time, and the number of dialogues, and displays the total dialogue time, the average dialogue time, the number of dialogues, and the frequency distribution of the dialogue time on the display unit. It may be done. Further, the voice / acoustic data processing unit in the terminal device statistically analyzes the voice spectrum of the interlocutor or the voice spectrum corresponding to the transmitted voice information. Here, the statistical analysis processing means that the voice spectrum of the interlocutor or the voice spectrum according to the transmitted voice information is analyzed over time, and the average spectrum and the voice spectrum that has changed by a predetermined value or more are detected. .

Next, each terminal device of the system is provided with a dialogue processing section for detecting a given term issued by the interlocutor or a given term in the voice corresponding to the transmitted voice information. In addition, the terminal device may include a position change analysis unit that analyzes a movement trajectory, a movement change amount, a movement speed, or a movement acceleration of a head, an upper body, or a hand of a person or a transmitted person image ( This analysis is performed by a processing unit or the like).

In the case of performing counseling or the like, it is possible to quantify the dialogue situation by conducting dialogue situation analysis such as dialogue time analysis, voice spectrum analysis, predetermined term analysis, or body movement analysis.

The transmission means in the two-way interactive system has a plurality of independent transmission lines between the first point and the second point that are separated from each other, and at least one of the plurality of transmission lines is secured. Is a voice transmission path for transmitting voice information. Here, the transmission means includes an already installed network such as a telephone network, and further has an electronic bulletin board system (BBS: Bulletin Board SBS) equipped with various databases.
ystem) is also included. Further, the system is equipped with a backup system that detects a failure in the transmission line being used interactively and switches the transmission line to be used. Therefore, when a failure occurs in a transmission path during a conversation, the conversation can be continued using another transmission path. In such a case, it is preferable to secure at least audio transmission on the other transmission path. In addition, since it is generally considered that the line abnormality is caused by the abnormality of the coaxial cable, the optical fiber cable, or the transmission device, it is preferable to set the other transmission lines as a wireless system.

[0045]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of a bidirectional interactive system according to the present invention will be described below with reference to FIGS.

FIG. 1 is a logical configuration diagram showing the entire bidirectional interactive system according to the present invention. This system uses A
Interlocutors 3A (first interlocutor) and B at the point (first point)
This is a system that realizes a two-way dialogue with a dialogue person 3B (second dialogue person) at a point (second point). In particular, this system aims at a counseling system that establishes intimate relationships of trust and enables close communication even between interlocutors in remote areas, so at least one interlocutor can be a teacher, doctor, or psychologist. I am assuming a counselor. Therefore, this system has various functions and configurations so that these counselors can provide information necessary for counseling and an ideal counseling environment.

Specifically, at the point A, (a) a chair 4A on which the interlocutor 3A (for example, counselor) sits, and (b)
A table 2A to which an interlocutor 3A goes;
7 and a terminal device 20A provided with a mechanism for matching the line of sight between the interlocutors. On the other hand, at the point B, (a) the chair 4B on which the interlocutor 3B (for example, a client who receives counseling) sits, (b) the table 2B to which the interlocutor 3B goes, and (c) the interlocutor via the transmission means 27 And a terminal device 20B provided with a mechanism for matching the line of sight of the user. Here, the transmission means 27 enables transmission and reception of image information and audio information between the terminal device 20A at the point A and the terminal device 20B at the point B, and thus the optical fiber transmission line 31 that enables large-capacity transmission. (Main transmission line)
And a satellite communication transmission line 32 (a backup transmission line) and the like. The transmission means 27 may be wired or wireless. In addition, the transmission means includes an already installed network such as a telephone network, and further has an electronic bulletin board system (BBS: Bulletin Board) equipped with various databases.
System).

At point A, the chair 4A described above functions to define the position of each interlocutor 3A. This chair 4A
Is preferably fixed to maintain a constant distance from the display unit 21A of the terminal device 20A. However, even when the chair 4A is not fixed, the table 2A is installed on the interlocutor 3A and the display unit 21A, so that the chair 4A is not fixed.
It is possible to keep the distance between A and the display unit 21A constant (an experiment by the inventor has confirmed that the existence of the table 2A serves as a reference for the psychological distance of the interlocutor 3A).
The configuration of the terminal device 20B at the point B where the other party's interlocutor 3B (for example, a person who receives counseling) is the same.

At the point A, the terminal device 20A (1)
A monitor TV 1 arranged on the table 2A for displaying the image of the interlocutor 3B based on the image information transmitted from the terminal device 20B at the point B via the transmission means 27.
A display section 21A provided with A, and (2) an image of the interlocutor 3A captured by the CCD camera 5A is captured as image information.
The image pickup unit 6A for transmission to the terminal device 20B at the point, and (3) The voice of the interlocutor 3B is output based on the voice information transmitted from the terminal device 20B at the point B via the transmission means 27. And a voice input unit 24A (microphone) for collecting the voice of the interlocutor 3A as voice information and transmitting it to the terminal device 20B at the point B. Furthermore, in order to realize more detailed communication as a counseling system, (5) the interlocutor 3A inputs characters and figures, and the input characters and figures are displayed on the display unit 21A.
And a character / graphic input unit 25 for transmitting the character / graphic information to the terminal device 20B at the point B.
A (an interface such as a keyboard, a pointing device, a touch panel, etc.) and (6) a processing unit 26 that is provided between each of the above components and the transmission unit 27 and performs signal processing and transmission control. The terminal device 20B at the point B is configured similarly to the terminal device 20A at the point A described above.

Further, the terminal devices 20 at the points A and B
In A and 20B, the configuration of the imaging devices 600A and 600B will be described. For convenience of description, when referring to components common to the terminal devices 20A and 20B at points A and B, for example, like the imaging device 600,
The description will be made by omitting letters A and B for distinguishing each point. In addition, unless otherwise described, it is assumed that the terminal device 20A at the point A is described in principle,
The duplicate description of the terminal device 20B at the point B having the common configuration is omitted.

The image pickup device 600 includes a microminiature CCD camera 5 (5A) which is an image pickup unit, and a support mechanism 10 (first unit) for installing the CCD camera 5 at a predetermined position in a supported state.
2), and an image pickup unit 6 (6A) for controlling the CCD camera 5. Monitor T
V1 (1A) is placed on the table 2 (2A), and the interlocutor 3 (3A) is at a distance L from the monitor TV1.
(M) Sitting on a chair 4 (4A) provided with a height adjustment mechanism 40 (40A) placed away. This distance L (LA)
Is set to 0.5 m or more and 5 m or less. Between the interlocutor 3 and the monitor TV 1 to be photographed, the outer diameter is 20 mm or less,
A cylindrical microminiature CCD camera 5 having a length of about 100 mm is provided. The CCD camera 5 is installed so that its imaging direction is directed to the person 3 to be photographed. The image of the interlocutor 3 captured by the camera 5 is transmitted from the imaging unit 6 to the other interlocutor side as image information (transmitted to the terminal device 20B at the point B via the transmission means 27). Note that the interval between the display unit 21 and the interlocutor may be slightly shifted depending on the interlocutor. Therefore, in order to cope with such a situation, it is preferable to select a CCD camera 5 having a large depth of focus.

Next, the specific installation position of the microminiature CCD camera 5 will be described with reference to FIG.

First, the image pickup unit 6 of the image pickup apparatus 600 is installed on the display section 21 for displaying the transmitted image of the other interlocutor. The main body of the imaging unit 6 displays an image of the interlocutor photographed by the camera 5 (taken into the imaging unit 6 via the camera cable 50) and a transmitted image of the other interlocutor 3B. In addition to the changeover switch 6a for switching, various adjustment switches 6b are provided. Switch 2
Reference numeral 10 includes a power switch for turning on / off the power of the monitor TV 1. The CCD camera 5 is installed by a support mechanism 10 (robot arm) at a predetermined position in front of a monitor screen 8 on which the transmitted image 9 of the other interlocutor 3B is displayed. In addition, this monitor screen 8 is 4 inches or more
The screen is about 0 inches.

Specifically, the CCD camera 5 is at a position forward w (cm) from the monitor screen 8, and the monitor screen 8
Is arranged in the vicinity of the head 9a of the interlocutor image 9 displayed in.
The central axis of the cylindrical CCD camera 5 is located at a position h (cm) above the eye position of the interlocutor image 9 shown by the dotted line.

As described above, since the microminiature CCD camera 5 is located near the head above the eye position of the interlocutor image 9 displayed on the screen 8 of the large monitor TV 1, it is particularly suitable for two-way dialogue. No trouble will occur. For example, the CCD camera 5 is provided at a position h = about 10 (cm) above the eye position (the position indicated by the dotted line in the figure) of the interlocutor image 9 displayed on the screen 8, and between the monitor TV 1 and the interlocutor 3. In the case of the system configuration in which the distance L is set to about 2.5 (m), the parallax angle is 2.3 °.
(The distance w between the monitor screen 8 and the CCD camera 5 is 10 (c).
m) does not particularly affect the fluctuation of the parallax angle). In other words, as a result of the experiment, the inventor, if the other party's eyes (the eyes of the image of the other interlocutor 3B whose monitor is displayed on the surface 8) are clearly visible at the time of line-of-sight matching, the inventor finds a microminiature camera around the head. It was confirmed that even if the screen 5 is provided, if the screen 8 of the monitor TV 1 is large, there is almost no hindrance to the interactive dialogue. The size of this screen 8 is 35
It has been confirmed that a good two-way conversation can be realized if the size is about 26 cm or 26 cm or more. It should be noted that if mutual interlocutors are acquaintances, there is a result that there is a tendency that there are few psychological obstacles even if the screen size is small, and the screen size may be properly used according to the application.

Next, the structure of the robot arm which is the camera support mechanism 10 of the image pickup apparatus 600 of FIG. 1 will be described with reference to FIG.

This robot arm has a first rod 101
a, a second rod 102a, and a holder 103a that holds the camera 5. The main body of the image pickup unit 6 and the first rod 101a are connected to the first joint portion 10
0 depending on the first rod 101a and the second rod 102a
Is connected to the second joint portion 101, and the second rod 102a and the holder portion 103a are connected to the third joint portion 10.
2 are rotatably connected. In particular, the third joint portion 102 includes a protrusion 103 directly rotatably connected to the holder portion 103a to enable special rotation. The first joint part 100 is rotatable in the direction shown by the arrow 100b, and the first rod 1
The movement in the direction indicated by arrow 100c of 01a is enabled. The second joint portion 101 supports the second rod 102a and enables movement of the second rod 102a in the direction indicated by the arrow 101b. Third joint part 1
02 enables the protrusion 103 to move in the direction indicated by the arrow 102b. In addition, the protrusion 103 and the holder 103
a can change the angle in the direction indicated by the arrow 103b. The above-described configuration of the robot arm (the first embodiment of the support mechanism) allows the camera 5 to be installed at a desired position and also allows the angle of the camera 5 to be adjusted. Therefore, the setting position of the CCD camera 5 and the imaging direction 5a can be freely set by the robot arm.

Next, as an application example of the image pickup apparatus 600, the structure of an image pickup apparatus 601 having another supporting mechanism (second embodiment) will be described with reference to FIGS.

The camera support mechanism of the image pickup device is the interlocutor 3B.
Since the monitor for displaying the image is installed in front of the surface 8, it is preferable that the monitor is made of a very inexpensive light transmissive material such as transparent thread, tape, transparent plate such as glass and acrylic. This is because it does not hinder the display of the monitor screen 8.

The image pickup device 601 shown in FIG.
A support member 110 made of a light-transmissive material for fixing the camera 5 to one end and supporting it and installing it at a predetermined position in front of the monitor screen 8, and the transparent support member 110.
A pedestal portion 61 that holds the other end of the transparent support member 110 and supports the entire transparent support member 110. Particularly, the pedestal portion 61 is the imaging device 601.
The circuit board 6c that is housed inside the imaging unit 6 and realizes functions such as a data processing unit is also housed inside the imaging unit 6. In the figure, reference numeral 62 denotes a drive mechanism (FIGS. 5 and 6) from the circuit board 60, which sends an electric signal for driving and controlling the pedestal portion 61 itself to move the installation position of the CCD camera 5.
Are a plurality of wires (ribbons) to be communicated to.

The horizontal and vertical movements of the CCD camera 5 are realized by the mechanism shown in FIG. That is,
The pedestal portion 61 that supports the transparent support member 110 is installed on the first moving table 63a, and the position in the horizontal direction is defined by the wire 64. Accordingly, when the first motor 65 pulls the wire 64 in either direction, the pedestal 61 moves in the direction indicated by the arrow 63c. Further, the first moving table 63a is also the first
It is installed on the second moving table 63b together with the motor 65, and the position in the vertical direction is defined by the wire 67. Therefore, the second motor 66 is connected to the wire 67.
By pulling in either direction, the entire first moving table 63a including the first motor 65 is moved to the arrow 63
Move in the direction indicated by d.

On the other hand, the angle adjustment of the CCD camera 5 is performed as shown in FIG.
It is realized by the mechanism shown in. That is, the base 6
Reference numeral 1 is composed of a pair of plate members 68a and 68b and a gap adjusting section 69 which defines a gap between these plate members. At this time, the transparent support member 110 is pushed up by the spring member 70 in the direction indicated by the arrow 70a. In such a configuration, in accordance with a control signal (electrical signal) output from the circuit board 60 via the wiring 62 connected by the connector 62a, the space adjusting unit 69 moves the plate member 68a in the direction indicated by the arrow 69a, By adjusting the spacing of 68b,
The transparent support member 110 is angularly adjusted in the direction indicated by arrow 110a. In other words, by adjusting the angle of the transparent support member 110 in the direction indicated by the arrow 110a,
The angle of the CCD camera 5 in the imaging direction 5b can be changed.

As the display unit 21, a plasma display or a liquid crystal display can be adopted instead of the monitor TV. When such a display device is used, FIG.
As shown in (1), a structure in which the CCD camera 200 is incorporated in the housing 210 of the display device in advance is preferable. The display device of FIG. 7 has a monitor screen 22 supported by a housing 210.
A through hole 230 is provided at a predetermined position of 0.
A structure is provided in which the tip of the D camera 200 is directly penetrated, and the tip is located in front of the monitor screen 220.

A specific structure is shown in FIG. FIG. 8 is a diagram showing a cross-sectional structure of the display device taken along line XX of FIG. As shown in the figure, the lens barrel 2 of the CCD camera 200
01 is provided with a spherical projection 202 so that it can be attached to the housing 210. An imaging device 205 and an imaging optical system 203 are incorporated in the lens barrel 201, and an electric signal detected by the imaging device 205 is transmitted from the electronic circuit unit 206 to an external data processing unit via the camera cable 500. Can be Also, the lens barrel 201
Has a structure in which at least one of the lenses constituting the imaging optical system 203 can be moved in the axial direction of the lens barrel 201. The focus of the CCD camera 200 is controlled by an adjustment knob 204. Will be

In the bidirectional interactive system according to the present invention, the line 5e perpendicular to the photoelectric conversion surface 208 of the image pickup device 205 of the camera 200, which is the image pickup unit, and the line 5e passing through the photoelectric conversion surface 208 and the three interlocutors. The angle formed by and is within the range of the parallax angle described above. This line segment 5e is the camera 200
Substantially coincides with the imaging direction. Then, the camera 200 attached to the display unit centers on a predetermined portion of the lens barrel 201 in a direction indicated by an arrow 207 in the figure in a direction of the lens barrel axis (one line 5e in the figure). Is moved, the imaging direction of the camera 200 (line segment 5 in the figure) is changed.
e).

CCD camera 5 having the above structure
Is a gripping member 2 directly attached to the housing 210.
40 and 250 are attached by gripping the projection 202 of the CCD camera 5. In addition, since the projection 202 of the lens barrel 201 is spherical, the angle of the CCD camera 200 can be freely changed.
The imaging direction 5c of the camera 200 can be directed to a desired direction. Note that FIG. 9 is a view showing an assembly process for realizing the structure shown in FIG. In this figure,
The gripping members 240, 250 are bolts 252, 253, 25.
4 fixed. Further, in the figure, reference numeral 255 is a through hole for inserting a bolt, which is provided in the grip member 250.

Further, FIG. 10 is a view showing a terminal device in which the CCD camera 5 and the display section are integrally constructed. FIG.
In this terminal device, the camera drive mechanism 2 for driving the CCD camera 200 in the directions (up, down, left and right) indicated by the arrow 260a.
60 are provided. The camera driving mechanism 260 allows the imaging direction 5d of the CCD camera 200 to be changed to the direction shown by the dotted line in the figure. The image captured by the CCD camera 200 is captured by the captured data processing unit 60 and sent to the other terminal device 20B (point B) via the transmission unit 27. Further, in this embodiment, the control of the camera driving mechanism 260 is performed by the imaging data processing unit 60.
Done in Further, in this embodiment, the imaging data processing unit 60 is configured in the processing unit 26, but FIG.
Alternatively, they may be independently configured as shown in FIG.

Next, the parallax angle will be described with reference to FIG. In this specification, the parallax angle is, as shown in FIG. 11, the line of sight of the photographed interlocutor 3 directed to the image 9 displayed on the monitor screen 8 and the line of sight to the CCD camera 5. It refers to the angle θ between the line of sight of the interlocutor 3. In other words, it means a shift angle of the installation position of the CCD camera 5 with respect to the line of sight of the interlocutor 3. In addition, the line-of-sight coincidence referred to here is the 1972 Union of Electrical Engineers of Japan (No. 19).
98), this means that the unnaturalness due to the installation position of the CCD camera 5 is below the allowable limit. Quantitatively, the parallax angle is 4.5 ° or less in the horizontal direction (image 9
Of the image 9 with respect to the center A of the image 9), 12 ° directly above (the head 9b side of the image 9 with respect to the image center A), 8 ° directly below (with respect to the image center A). Nineth body side) and below are within the line-of-sight matching range. Of course, the smaller the parallax angle, the better, and the detection limit is 3 ° or less in the horizontal and vertical directions. Therefore, the area where the CCD camera 5 can be installed is
It is the conical region shown in FIG. Note that this conical area is the image 30 displayed on the eyes 30 of the interlocutor 3 and the monitor screen 8.
Center point A (in this embodiment, the center of both eyes of image 9 is
Defined as the center point A of the interlocutor 3) and the line-of-sight matching area 9b on the monitor screen 8 at a predetermined distance L (m) from the line of sight 31 of the interlocutor 3, and the interlocutor 3 of the interlocutor 3. Eye 30
The position is defined by.

On the other hand, the CCD camera 5 is within a certain area with respect to the eye position (for example, the image center A) of the image 9 of the interlocutor 3B at the point B displayed on the monitor screen 8, for example,
There are the following three means for placing the image 9 about 10 cm above the eye position.

The first means is to move the CCD camera 5 itself with respect to the monitor screen 8. In order to realize the first means, the interlocutor 3 displayed on the monitor screen 8
An imaging data processing unit 60 (for example, as shown in FIG. 10) that performs image recognition processing on the position of the image 9 of the interlocutors 3A and 3B at the points A and B on the monitor screen 8. , A drive mechanism for moving the CCD camera 5 so that the CCD camera 5 is installed in the above-mentioned certain area within the monitor screen 8 with respect to the position-recognized image 9.
The robot arm shown in FIG. 3 and the mechanism shown in FIGS. 5, 6 and 8 are provided in the imaging unit 6.

The second means is to move the interlocutor 3 himself to be photographed with respect to the monitor screen 8. This second
Means is the image data processing unit 60, and the monitor screen 8
The position of the image 9 of the interlocutor 3 himself displayed on the screen is subjected to image recognition processing, and the interlocutor 3 imaged with respect to the image 9 of the interlocutor 3 himself who has been position-recognized
This can be realized by providing person position defining means for adjusting the position of the person. As the person position defining means, for example, a chair 4 on which the interlocutor 3 sits is preferable. Further, the seat height can be adjusted for each interlocutor by adjusting the seat height adjusting mechanism 40 provided in the chair 4. This is because the installation position of the chair 4 (such as the distance from the monitor screen 8) is fixed in advance, so that once the interlocutor 3 sits on the chair 4, the lateral position adjustment with respect to the interlocutor 3 becomes unnecessary. . In other words, the only uncertain factor for determining the position of the interlocutor is the vertical displacement caused by the physique difference between the interlocutors. Means can be realized.

A third means is a CC in which an image 9 of the interlocutor 3 (any of the interlocutors 3A and 3B at the points A and B) displayed on the monitor screen 8 is fixed in advance.
It is moved relative to the D camera 5. Similar to the above-mentioned means, the third means first recognizes the position of the image 9 on the monitor screen 8 by the image pickup data processing section 60 provided in the image pickup unit 6 and displays this image 9. It is necessary to move the position. This allows
The CCD camera 5 can relatively exist in the above-mentioned fixed area.

Next, using the flowchart of FIG. 13, C
The initial setting operation of the CD camera 5 (initial setting operation for line-of-sight matching) will be described. Note that FIG. 15 is a diagram for conceptually explaining this setting operation.

First, the interlocutor 3 makes himself / herself photograph the CCD camera 5 (step ST1), and displays the image 9 of himself / herself photographed by the CCD camera 5 on the monitor screen 8 of the display unit 21 (step ST2). At this time, if the installation position of the CCD camera 5 coincides with the line of sight of the interlocutor 3 (the line connecting the eye of the interlocutor and the center point A of the image 9) (the parallax angle is 0).
)), The presence of the CCD camera 5 causes a dialogue obstacle.

Therefore, the CCD camera 5 is moved upward by Δh.
Only (step ST3), and the CCD camera 5 is installed again in a position where the dialogue is not obstructed and within the line-of-sight matching allowable region (for example, the region shown in FIG. 12). That is, the CCD camera 5 is installed so as to overlap the hair portion 9b of the image 9 when viewed from the interlocutor 3.

As described above, when the installation position of the CCD camera 5 is moved by Δh, the imaging direction of the CCD camera 5 cannot be oriented toward the actual interlocutor 3. Therefore, the installation angle of the CCD camera 5 is corrected downward by Δθ in accordance with the movement of the CCD camera 5 (step ST4). As described above, the installation position and the set angle of the CCD camera 5 are adjusted in advance so that the parallax angle determined from the relationship between the distance L between the interlocutor 3 and the monitor screen 8 with respect to the movement amount Δh is within 3 °. , The eye-gaze matching state can be maintained.

Specifically, when the interlocutor 3 is a child with a short height, the sitting height of the child is adjusted by the sitting height adjusting mechanism 40 of the chair 4, and the angle of the CCD camera 5 is corrected downward. It is preferable to move the image 9 itself displayed on the monitor screen 8.

On the other hand, the interlocutor 3 displayed on the monitor screen 8
The operation of moving the image 9 of the interlocutors 3A and 3B at the points A and B will be described with reference to the flowchart of FIG. Note that FIG. 16 is a diagram for conceptually explaining this image movement processing. This image movement processing is performed by the imaging data processing unit 60 of the imaging unit 6 or the processing unit 26 (display control unit 285).

First, the reference point is specified from the image 9 of the received interlocutor or self displayed on the monitor screen 8 (step ST5). This reference point is a point 10 cm above the center (point A) of the eye position (indicated by a dotted line in the figure) of the image 9 or the contour of the image 9 easily obtained by binarization processing. 9 (CC)
(The position where the D camera 5 should originally exist).

Subsequently, a position 81 corresponding to the actual installation position of the CCD camera 5 is obtained from the absolute coordinates 80 of the monitor screen 8, and the amount of deviation between the already obtained reference position 82 and the actual position 81. Is within the allowable range (step ST6). If the amount of shading exceeds the allowable range at this stage, it is determined that the display position of the image 9 is deviated, and the coordinates 82 of the reference point are set as a starting point, and the actual position 81 is determined.
The vector 83 with the end point is calculated (step ST
7). Then, the vector 8 obtained as shown in FIG.
Based on 3, the shifted image 9 is translated (step ST8).

For the purpose of the present invention, any of the above-mentioned first to third means may be used as means for matching the line of sight between the interlocutors via the transmitting means, or by combining these means. You may use. In the two-way dialogue, the distance L of each chair 4 of both the interlocutors from the monitor TV 1 is fixed within a certain range, and the height of each chair 4 is vertically adjusted by the sitting height adjusting mechanism 40 by the above-mentioned second means. Then, the position of the face of the image 9 of the interlocutor 3 displayed on the monitor screen 8 exists within a substantially constant area. As a result of an experiment, the inventor has confirmed that the range in which fine adjustment of the size of the face, the position of the eyes, and the like for each interlocutor is within several cm to several tens of cm. Therefore, for this fine adjustment, it is effective to move the display position of the image 9 of the interlocutor 3 displayed on the monitor screen 8 by the second means described above. By this display movement, the seat height adjusting mechanism 40 of the chair 4 is
It is possible to easily achieve line-of-sight agreement between the interlocutors simply by adjusting. Further, it is not necessary to provide a complicated moving mechanism and an optical processing device on the side of the CCD camera 5. In the above-mentioned embodiment, the installation position of the CCD camera 5 is displayed on the monitor screen 8.
10 from the eye position of the image 9 of the interlocutor 3 displayed in
Although the position is set to (cm), the distance h can be easily changed by moving the image display by the third means.

The photographing device 600 of this embodiment can be retrofitted to a TV device which is usually on the market. Therefore, it is not necessary to develop a dedicated machine unlike the above-described second conventional technique that uses a display device that switches between reflection and transmission, and the above-described imaging device can be realized at low cost. Further, there is no need for special optical components such as a half mirror and a reflection mirror as in the first and third prior arts described above. Therefore, the device configuration is simplified and the mirror does not darken the image of the interlocutor.
Also, as a result of enlarging the mirror to get a wide angle of view,
The problem that the screen display is disturbed does not occur in the image capturing apparatus according to the above-described embodiments. Therefore, according to the interactive dialog system according to the present invention, interactive dialog using the image display device having a large screen can be easily realized.

17 and 18 are diagrams showing the structure of an image pickup device 600 which can be retrofitted to a commercially available TV device. As shown in FIGS. 1 and 2, this image pickup device 600 supports a CCD camera 5 for picking up an interlocutor, and supports the CCD camera 5 and sets the CCD camera 5 at a desired position. Support mechanism 10 (robot arm) for
Processing unit 26 including the imaging data processing unit 60 described above
Is configured as a circuit board.

In this image pickup device 600, the CCD camera 5
The electrical signal from the terminal 51 via the camera cable 50
Is sent to a processing unit 26 electrically connected to the processing unit 26. The voice of the interlocutor 3 is captured by the microphone 24,
It is sent as an electric signal to the processing unit 26 electrically connected to the terminal 241 via the cable 240. on the other hand,
The image data and audio data of the interlocutor sent from the transmission means 27 are taken into the processing unit 26 which is electrically connected to the terminal 611 via the cable 610. Then, the image data and the audio data of the other interlocutor (for example, the interlocutor 3B at the point B) once taken into the processing unit 26 are sent to a commercially available TV device via the cable 650. The terminals 601a (audio terminal) and 601b (image terminal) of the cable 650 are electrically connected to the processing unit 26, while the terminal 602a (audio terminal),
602b (image terminal) is connected to a predetermined connector of the TV device.

FIG. 18 is a diagram showing a specific structure of the connector provided on the back surface of the image pickup device 600. As can be seen from this figure, the terminal 51 of the CCD camera 5 is inserted and fixed to the image input connector 52, and the terminal 241 of the speaker 24 is inserted and fixed to the audio input connector 242, and the image and audio data are sent to the TV device. Terminals 601a and 601b of the cable 650 for insertion are fixedly inserted into the audio output connector 603a and the image output connector 603b, respectively, and the terminal 611 of the cable 610 for transmitting / receiving data to / from the transmission means 27 is data transmission / reception. The connector 612 is inserted and fixed. The image input connector 620 and the audio input connector 630 are spare connectors for transmitting and receiving data to and from the transmission means 27.

Next, a logical configuration diagram of the terminal devices 20A and 20B in the bidirectional interactive system according to the present invention will be described with reference to FIG. In addition, the system is aimed at a system that realizes realistic counseling between interlocutors in remote areas, and in addition to a mechanism that matches the line of sight between interlocutors, an ideal counseling environment is realized. It has various functions. The configuration described below is a configuration common to both the terminal devices at the point A and the point B.

First, this terminal device includes a processing unit 26.
It is an integrated device including the imaging data processing unit 60. Here, the imaging data processing unit 60 takes in the imaging data from the CCD camera 5 and controls a data processing unit 600a for performing predetermined processing and a driving mechanism 600c for adjusting the set position and the set angle of the CCD camera 5. And a drive control unit 600b.

The main control unit 273 of the processing unit 26 has at least a person authentication unit 274 (at least IC (integrated circu
It is a card-shaped recording medium 289 provided with it), which also uses information read by the reading device 271 from an IC card or the like), for analyzing and recording the content of the dialogue exchanged between the interlocutors. The dialogue processing unit 275, the backup system 276 for performing crisis management of the entire system (for example, the above transmission / reception system is detected, and the main transmission / reception system 288 such as an optical fiber is set to the wireless system 28).
7), an external memory management unit 277 for outputting display information, voice analysis information, etc. handled by the system to a card-shaped recording medium 269 having at least an IC such as the printer 270 or an IC card, reception, keyword An integrated information processing system 278 is provided for comprehensively managing information, consultation item information guide, counselor information, fee information, and the like with an integrated information D / B (data base) 267. The main controller 273 also recognizes the position of the person image (including the monitor images of the first and second interlocutors) displayed on the monitor.

Further, the processing unit 26 adjusts the installation position of the voice / acoustic data processing unit 279 (for example, spectrum analysis of voice information) for processing voice information taken from the microphone 24. Drive control unit 280 for controlling the drive mechanism 281 of the above, and a voice / sound output unit 282 for causing the speaker 23 to output predetermined voice or sound information (this voice / sound output unit 282
Utilizes dedicated voice / sound information D / B 265 to realize a volume control function 283 and a BGM output function 284 for providing an ideal counseling environment. ), A display control unit 285 for displaying image data, character data, etc. on the monitor 1 (having processing functions such as image enlargement / reduction function), and a character / graphics input device 272 as a man-machine interface. A character / graphics input control unit 286 for performing the operation is provided. A plurality of monitors 1 may be provided, and as shown in the figure, the first and second monitors 1a, 1
The display control unit 285 may perform control to display different information for each of b (for example, display image information on the first monitor 1a and display character / graphic information on the second monitor 1b).

Further, the processing unit 26 is individually dedicated D / B (for example, image information D / B 261, audio / sound information D / B 262, character information D / B 263, backup D / B 264, BGM information D / B 265). , Integrated information D
/ B267, display information D / B266, etc.). The character / graphic input device 272 includes, for example,
There are keyboards, pointing devices, touch panels and special pens.

The operation of the bidirectional interactive system as the counseling system will be described in detail below with reference to the configuration of FIG. Note that FIG. 20 shows the voice output unit 23.
FIG. 3 is a diagram showing the configuration of FIG. By using the speaker arranged on the side surface of the interlocutor 3 and near the ear, an environment of whispering in the ear is realized, and the intimacy is further improved. Headphones are typically used as the audio output unit 23, but the ear independent speaker 12 fixed to the chair 4 by the flexible joint 13 shown in this figure is suitable because it can realize whispering without contact. The speaker may be attached to the surrounding wall (for example, the sound absorbing members 22A and 22B shown in FIG. 1).

The voice output unit 23 has a function 284 for outputting background music (BGM) and a volume / volume control function 283 for automatically detecting the conversation volume and adjusting the output volume of the background music. The output is controlled by the unit 282. When the voice / sound output control unit 282 detects that the conversation volume is substantially zero for a predetermined period of time, based on the detection result, the output of the background music is started, the volume of the background music is increased, or the background music is increased. Adjust the volume reduction.
Further, the voice / sound output control unit 282 outputs background music that calms the mind of the interlocutor 3 while superimposing it on the voice of the interlocutor 3 without hindering the hearing of the interlocutor 3. be able to. In particular, when the silence continues, the background music is started or the volume is changed.
Is released, and the interlocutor 3 can be calmed down.

The voice input section 24 is preferably a telephoto microphone. In general, a microphone with a table or attached to the chest can be adopted, but a microphone with a table may impair consciousness during counseling. Also, if the chest microphone is mistakenly left, the code may be broken. Therefore, a telephoto microphone that does not need to be aware of the presence is arranged on the upper part of the display unit 21, or
It is preferable to incorporate it in. Since the line-of-sight of the interlocutor 3 has already been recognized, it is possible to improve the directivity by directing the sound collecting direction to the mouth.

The voice input section 24 has a voice input amplification / adjustment function for controlling the collected sound volume, automatically converts the output volume of the collected sound into a predetermined volume, and collects the collected sound at a predetermined time. The voice / acoustic data processing unit 279 having a volume adjusting function for detecting the average level and automatically converting the volume when the difference between the collected volume and the average level is equal to or more than a predetermined value is controlled. Further, the display unit 21 displays the volume to be transmitted from the voice / acoustic data processing unit 279 to the other interlocutor side via the transmission means 27 under the control of the main control unit 273 of the processing unit 26. . The act of speaking to a person (image of an interlocutor) on the monitor screen 8 in a state in which the microphone 24 cannot be directly seen makes it difficult to determine how much one's volume should be set unless the user gets used to it. Therefore, by adjusting the speaking volume, a smooth dialogue is facilitated. Also, by displaying the output volume (volume of the audio information sent to the other party) on the screen, it is possible to feed back to the interlocutor whether the output volume is appropriate for transmission.

On the other hand, the character / graphic information input from the character / graphic input device 272 is stored in the character / graphic input control unit 28.
6, the character information or graphic information corresponding to the character or graphic input from the character / graphic input device 272 is transmitted and output to the terminal device 20B at the point B. Therefore, the transmission information transmitted on the transmission path further includes character information or graphic information (text data) in addition to human image information (image data) and audio information (audio data). The display unit 21 displays a character or graphic together with the image 9 of the interlocutor 3, who is the other party, based on the transmitted character information or graphic information.
In this way, counseling can be facilitated by displaying the characters or figures together with the person image.

Here, the characters or figures displayed on the monitor TV1 of the display unit 21 may be displayed by being superimposed on the image of the interlocutor 3, but as shown in FIGS. The display area and the display area of the image 9 of the interlocutor are preferably separated. In particular, when a person image is displayed on a monitor with a different display size, the person image is moved on the monitor by the algorithm shown in FIG. 16 (this algorithm is based on the monitor image of the first interlocutor or the second image). This can be realized by using any of the monitor images of the interlocutors), and the gazes are matched between the first interlocutor and the second interlocutor. When the display area 8 (monitor screen) of the monitor TV1 of the display unit 21 is vertically divided, the image 9 of the interlocutor is preferably displayed on the upper side (see FIG. 21). Alternatively, the monitor TV of the display unit 21
It is also possible to display the image of the interlocutor on one side or in the center of the display area 8 of 1 (see FIGS. 22 and 23). In particular, when the screen is divided into left and right, it is preferable to display a character or a figure on the right side toward the interlocutor who looks at the monitor screen and an image of the interlocutor on the other side on the left side. Generally, humans perform artistic / subjective thinking in the right brain and computational / logical thinking in the left brain. However, visually, the operations of the right and left brains are reversed, and information from the right eye is mainly processed by the left brain, and information from the left eye is mainly processed by the right brain. Therefore, when displaying two types of characters / graphics (logical) and the image of the interlocutor (subjective) simultaneously, it is preferable to display the characters / graphics on the right side and the human figure on the left side.

Furthermore, as shown in FIG. 24, a monitor TV
An image of the interlocutor may be displayed in the upper center portion of the display area 8 of 1. In this case, the display control unit 285 performs image processing to reduce the interlocutor image 9.

The display control unit 285 is capable of enlarging or reducing the interlocutor image 9 (which may be either the first interlocutor or the second interlocutor) displayed on the monitor TV1. At this time, in the vicinity of the crown of the interlocutor image 9, a part of the image corresponding to the position where the camera 5 is actually installed is detected, and the detected image part is fixed, and the dialogue is performed. Enlarge or reduce the entire image of the person. At this time, a predetermined area on the monitor TV 1
And at least one of a character and a graphic is displayed in the area on the monitor excluding the area where is displayed.

According to the above separation method, the line-of-sight match does not deteriorate. A display device for displaying an image of the interlocutor;
It is of course possible to separate the display device for displaying characters or graphics (for example, as shown in FIG. 19, the display control unit 285 causes the first monitor TV 1a and the second monitor TV 1a).
It is also possible to control the display of 1b independently). Further, the character / graphics input device 272 may be arranged at either the terminal device 20A or 20B at the point A or the point B.

A smooth dialogue is possible by performing counseling with characters and figures. As such a character / figure input device 272, the dialogue can be selected by the interlocutor.
It is most preferable that the voice of the interlocutor can be input, recognized, converted to character information and displayed (this function is controlled by the main control unit 273 under the control of the voice / sound output). Unit 282).

The character / graphic input device 272 is shown in FIG.
As shown in FIG. 9, a keyboard device, a pointing device,
A device, and further, a pen input device (touch panel and pen) arranged at the hands of the interlocutor can be adopted. Further, by providing both the keyboard device and the pen input device so that they can be appropriately selected, the counseling process can be more smoothly progressed.

The system is intended to enable ideal counseling between interlocutors at remote locations. Therefore, in order to preserve such a special environment,
As a security function, the system includes a person authenticating unit (person authenticating unit 274 in the processing unit 26) for confirming the person of the interlocutor, and the counseling operation is started after the person is confirmed.

In this case, the person authentication section 274 may be characterized in that the authentication data is read from the storage medium 289 in which the authentication data is written, and the person is confirmed based on the read result. The storage medium 289 is a recording medium including an IC or the like, and for example, an optical card, an IC card, or a magnetic card can be used (when such a recording medium 289 is used, a dedicated reading device is used in advance. 271
Prepared).

Further, the person authenticating section 274 may perform person confirmation based on the image analysis result of the interlocutor's face, or may perform person confirmation based on the voice analysis result of the interlocutor's voice. The person may be confirmed based on the character analysis result of the sign character of the interlocutor, and the person confirmation may be performed based on the text analysis result of the message text of the interlocutor. By such person authentication, the interlocutor can be identified. It is preferable to properly use the authentication method according to the confidential content.
The above-mentioned message sentence analysis refers to, for example, comparing a sentence "X is Y." Refers to analysis.

Next, the table 2 is set to a height such that the upper part of the chest is higher than the table when the interlocutor 3 is sitting on the chair 4. However, the height of the table 2 may be set so that the height above the chest is higher than the table when the interlocutor 3 is standing. This is because the effect of active dialogue and shortening of dialogue can be expected by conducting dialogue while standing. Further, since the sense of continuity is generated between the interlocutor 3 and the monitor screen 8 by using the table 2, the anxiety caused by not knowing the relative distance (psychological distance) to the other party is reduced. In this case, the display unit 2
1 and the table 2 are preferably in contact with each other (configuration of FIG. 1), but a gap that does not cause a psychological obstacle (for example, several tens cm)
There may be the following).

In the case of counseling or the like, it is preferable that the processing unit 26 perform a dialogue situation analysis such as a dialogue time analysis, a voice spectrum analysis, a predetermined term analysis, or a body movement analysis. As a result, it becomes possible to quantitatively analyze the psychological changes, etc. of the interlocutor who is undergoing counseling (also serves as counselor data).

At least a part of the periphery of the interlocutor is provided with a sound absorbing member 22 having a property of blocking external sound (see FIG. 1). As a result, the psychological obstacle to the leakage of the conversation contents (counseling contents) to the outside is reduced, and the sound quality is improved.

In FIG. 1, two transmission lines independent of each other are prepared as the transmission means 27, but the number may be three or more. In addition, a coaxial cable transmission line or another wireless transmission line may be used as the transmission line in addition to the optical fiber transmission line 31 and the satellite communication transmission line 32.

Further, the two-way interactive system according to the present invention is provided with a backup system for detecting a failure of the transmission path during the interactive use and automatically switching the used transmission path. By connecting via a plurality of transmission paths, if a failure occurs in a transmission path during a conversation (or during counseling), the conversation can be continued using another transmission path. In such a case, it is preferable to secure at least audio transmission on the other transmission path. Generally, a line abnormality is caused by a coaxial cable, an optical fiber cable,
Alternatively, it is considered that the above is the cause of the transmission device. Therefore, it is preferable that the other transmission lines are wireless systems.

Next, the result of the experiment of the bidirectional image communication dialogue using the bidirectional dialogue type system having the line-of-sight matching mechanism according to the above-mentioned embodiment will be explained. (For detailed system configuration, see FIG. reference).

On the experimental monitor TV 1, the screen diagonal line 10
A 3.4 cm 40 inch large projection monitor was used. As the CCD camera 5 for the experiment, a micro camera having a camera head diameter φ of 12 mm was used.
Each monitor TV1 was installed in each adjacent room, and the distance between the monitors was about 20 m. The distance L between the monitor TV1 and the interlocutor 3 is about 2 m, the upper half of the interlocutor 3 is photographed within a width of 88 cm and a height of 66 cm, and the image is transmitted to the monitor TV1 of the other party. . Also, CCD
The camera 5 was fixed at a position h = 10 cm above the line of sight of the interlocutors 3, and the height of the chair 4 was adjusted according to the sitting height of each interlocutor 3.

In the two-way dialogue performed with such a system configuration, in the basic posture in which the interlocutor 3 sits on the chair 4 in the normal state, the parallax angle falls within the range of about 2.86 °. However, in reality, the head and upper body of the interlocutor 3 move when talking. This movement was measured about 6 interlocutors, and it was about ± 15 cm at maximum in the horizontal direction and about ± 10 cm at maximum in the vertical direction. Therefore,
When the parallax angle including the movement of the interlocutor 3 is calculated with the above system configuration, the maximum is 4.3 in the horizontal direction of the screen of the monitor TV 1.
Up to 5.7 ° in the vertical direction. For this reason, according to the present embodiment, the parallax angle is within the range of 4.5 ° or less in the left-right direction, 12 ° just above, and 8 ° or less just below, which is within the line-of-sight matching allowable range of the Association of Electrical Engineers of Japan. Fits. In addition,
The average value of the movement of the interlocutor 3 is about ± 7 cm in the left-right direction and about ± 4 cm in the up-down direction, which are less than half of the maximum values. Therefore, it is considered that the parallax angle obtained by this system has no problem in actual dialogue. In addition, in the experimental results, the interlocutor did not feel any particular discomfort with respect to the line-of-sight matching, and the two-way conversation was smoothly performed.

According to this experiment, by matching the lines of sight between the interlocutors via the transmission means 27, it is possible to realize emotional exchanges among the interlocutors with respect to the theme related to distress. It was confirmed that counseling is possible with the system. In addition, by using characters or figures together, "character / figure recognition dialogue" is possible, in which the dialogue proceeds while gazing at the letters or figures, making it possible to deepen the conversation in a shorter time.
It was confirmed that the conversation could be organized.

In order to further improve this system, the distance L between the monitor TV 1 and the interlocutor 3 is separated to about 2.5 m, and the interlocutor image 9 displayed on the monitor screen by the display control unit 285 is displayed. It is preferable to finely adjust the position so that the line-of-sight position of the interlocutor 3 is always within the parallax angle of 3 ° with respect to the CCD camera 5. With such a configuration, even if the interlocutors 3 slightly move, it is possible to maintain proper line-of-sight matching, and it is possible to realize line-of-sight matching between the interlocutors more realistically.

Further, the above experiment was conducted using the large monitor TV 1 and keeping a certain distance between the photographed interlocutor 3 and the monitor TV 1. When a small monitor, for example, a 10-inch monitor TV is used as the monitor TV1, the interlocutor image displayed on the monitor screen 8 is naturally small.
Therefore, the parallax angle can also be reduced by combining the small monitor TV with an ultra-small CCD image pickup device and setting the size of the image pickup device in the small monitor screen to a certain ratio or less. In this case, the distance L between the interlocutor 3 and the monitor TV1 can be easily reduced to 1 m or less, which is advantageous in saving space. The inventor used a 14-inch monitor TV and h =
The counseling experiment was tried under the conditions of 5 (cm) and L = 1.2 (m). As a result, 83% of the 18 clients rated this method of counseling positively.

According to the above-described embodiment, the monitor TV 1 normally shipped can be retrofitted, no special optical components such as a half mirror and a reflection mirror are required, and only the CCD camera 5 can be used. In addition, if an image processing function is added, it is possible to realize an imaging device having a line-of-sight matching mechanism having a line-of-sight matching tracking function (FIGS. 17 and 18).
reference). In addition, in the above-described embodiment, the configuration in which the image pickup device is retrofitted to the existing TV device has been described, but the image pickup unit (camera) may be integrated in a part of the display unit. That is, a device configuration in which an image pickup unit such as a CCD camera is embedded in a part of a display device such as a liquid crystal television or a plasma display, or a device in which the image pickup unit is previously formed as a part of a device in a partial area of the display device. It may be configured (see FIGS. 7 to 10). In this case, the development of a dedicated device is required, but the device configuration will be extremely effective for thinning the monitor in the future.

The present invention is not limited to the above embodiment, but can be modified. For example, an image printing device (printer 270) that prints a display screen, or a storage unit that stores display screen information (for example, display information D / B).
266, backup D / B 264, etc., IC card, etc. 269) can be further provided. As the storage means, an optical card, an IC card, an optical disk, or a magneto-optical disk can be used. In this case, it is possible to store information during the dialogue, particularly information on characters and figures used during the dialogue.

[0118]

As described above, in the system according to the present invention, when the image pickup unit (camera) is installed within a certain area in front of the image of the interlocutor on the display unit (the interlocutor is superimposed on the image of the other party). If the size of the camera is kept below a certain level, the presence of the imaged interlocutor and the image pickup unit placed between the display units has almost no effect on the interactive dialogue even when the user views the camera. It was realized based on the experimental result of not giving.

Therefore, by retrofitting the components to the existing display device, a photographing device, a terminal device, and a system composed of these devices, which are provided with a mechanism for easily realizing the line of sight between the interlocutors, are realized. can do. Therefore, it is not necessary to develop a dedicated machine, and the device can be realized at low cost. In addition, since the device can be configured simply by fixing the image pickup unit to a fixed position on the screen, it is not necessary to provide a half mirror having the same size as the screen size as in the conventional case, and thus the device configuration space is not required to be wide, and moreover, It is easy to increase the size of the display screen. Moreover, since the interlocutor's image is taken directly without using a mirror, the obtained image will not be dark.

Furthermore, by moving the image pickup section by the drive mechanism, adjusting the line-of-sight direction of the interlocutor himself by position adjustment (sitting height adjustment, etc.) by the interlocutor himself, and by image movement processing by the display control section. By moving the display position of the image of the interlocutor displayed on the screen of the display unit, the imaging unit (camera) can be positioned within a certain area of the image of the other interlocutor displayed on the screen as seen from the interlocutor. Is installed. Also, by combining these means, it is possible to easily achieve the parallax angle within 3 ° required for ideal line-of-sight matching. In particular, by performing image processing in the display processing unit and moving the image of the interlocutor on the other side displayed on the screen to a certain position with respect to the image capturing unit, it is possible to mechanically achieve more accurate line-of-sight matching and line-of-sight recognition. It can be realized without using a special drive system. Further, the line-of-sight coincidence degree can be changed and set according to the individual level.

Further, the drive mechanism of the image pickup unit adjusts the angle of the image pickup unit, and by adjusting the angle of the image pickup direction of the image pickup unit, the image pickup direction is recognized by the image pickup data processing unit, for example. Automatically oriented to match the person's line of sight. Therefore, even if the interlocutors move during the two-way conversation, it is possible to maintain the optimum line-of-sight matching state among the interlocutors.

Also, by using characters or figures together, "character / figure recognition dialogue (visual communication)" in which the dialogue proceeds while gazing at the letters or figures becomes possible, and the conversation can be deepened in a shorter time. You can organize the conversation. As described above, the use of this system enables a smooth dialogue through the monitor TV accompanied by emotional exchange. Furthermore, this system can be widely applied to multimedia communication, multimedia counseling, multimedia helping, audio-visual counseling, etc.

[Brief description of the drawings]

FIG. 1 is a logical configuration diagram showing an entire bidirectional interactive system according to the present invention.

FIG. 2 is a view showing the image pickup apparatus of FIG. 1 and is a view for explaining a method of installing a CCD camera (image pickup unit) at a predetermined position.

3 is a view showing a first embodiment of a support mechanism in the image pickup apparatus of FIG. 1, and is a view showing a structure for installing a CCD camera at a predetermined position.

4 is a view showing a second embodiment of the support mechanism in the image pickup apparatus of FIG. 1, and is a view showing a structure for installing a CCD camera at a predetermined position.

5 is a diagram showing a structure of a camera moving mechanism of the support mechanism (second embodiment) shown in FIG. In particular, this mechanism is C
Move the CD camera horizontally and vertically.

FIG. 6 is a view showing the structure of an angle adjusting mechanism of the support mechanism (second embodiment) shown in FIG. In particular, this mechanism is CC
Change the imaging direction of the D camera.

FIG. 7 is a diagram showing a second embodiment of the terminal device in the two-way interactive system, which is a support mechanism (third embodiment) for setting the CCD camera at a predetermined position. The CCD lamella has a structure attached to the display unit.

FIG. 8 is a C of the support mechanism (third embodiment) shown in FIG.
FIG. 8 is a view showing a mounting structure of the CD camera, which is X in FIG. 7.
It is sectional drawing which followed the -X line.

9 is a perspective view showing an assembling process of the support mechanism (third embodiment) shown in FIG. 7. FIG.

10 is a diagram showing a structure for adjusting a position and an image pickup direction of a CCD camera as an application example of the support mechanism (third embodiment) shown in FIG.

FIG. 11 is a diagram illustrating a parallax angle.

FIG. 12 is a diagram showing an area where a CCD camera is to be installed.

FIG. 13 is a flowchart for explaining an initial installation operation of the CCD camera.

FIG. 14 is a flowchart for explaining an operation for moving the image displayed on the monitor so that the lines of sight of the interlocutors match.

FIG. 15 is a flowchart of the CC described in the flowchart of FIG.
FIG. 5 is a diagram for conceptually explaining the initial installation operation of the D camera.

FIG. 16 is a diagram conceptually illustrating an operation of moving an image displayed on a monitor, which is described in the flowchart of FIG. 13;

17 is a diagram illustrating an application example (imaging device) of the imaging unit alone in FIG. 1;

FIG. 18 is a diagram showing the structure of the back surface of the imaging unit shown in FIG. 17.

FIG. 19 is a logical block diagram showing an overall configuration of the terminal device shown in FIG. 1.

FIG. 20 is a perspective view showing a configuration of an audio output unit in FIG. 1;

21 is a diagram showing a screen configuration example of a display screen (particularly, in the case of screen split display) displayed by the monitor of FIG. 1 (No. 1). FIG.

FIG. 22 is a diagram illustrating an example of a screen configuration of a display screen (particularly, in the case of screen division display) displayed by the monitor in FIG. 1 (part 2).

FIG. 23 is a diagram showing a screen configuration example of a display screen (particularly, in the case of screen split display) displayed by the monitor of FIG. 1 (No. 3).

FIG. 24 is a diagram showing a screen configuration example of a display screen (particularly, in the case of screen split display) displayed by the monitor of FIG. 1 (Part 4).

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Monitor TV, 2 ... Table, 3 ... Interlocutor, 4 ... Chair, 5, 200 ... Imaging device (CCD camera), 6 ... Imaging unit, 9 ... Interlocutor image, 10, 110 ... Support mechanism, 2
0 ... Terminal device, 21 ... Display unit, 23 ... Speaker, 24 ...
Microphone, 25, 272 ... Character / figure input device, 26 ... Processing unit, 27 ... Transmission means, 60 ... Imaging data processing unit,
271 ... Reading device, 274 ... Person authentication part, 275 Dialog processing part, 281, 600c ... Driving mechanism, 269, 289 ...
Recording medium, 279 ... voice / sound data processing unit, 282 ...
Voice / sound output control unit 285 ... Display control unit 600,
601 ... Imaging device.

Claims (46)

[Claims] 1. A first terminal device installed at a first point where a first interlocutor is, and a second terminal where a second interlocutor is located, which is a distance away from the first point. A second terminal device and a transmission means for enabling bidirectional communication of image data of at least the first interlocutor and the second interlocutor between the first terminal device and the second terminal device. In the two-way interactive system, at least the first terminal device displays on the monitor an image of the second interlocutor received via the transmission unit, and a monitor screen of the display unit, A position defining unit that defines the position of the first interlocutor, and a device that enables the image of the first interlocutor to be transmitted to the second terminal device via the transmission unit. The imaging unit that directly images the person and the imaging direction of the imaging unit An imaging device having a support mechanism for setting the imaging unit at a predetermined position between the first interlocutor and the monitor in a state of facing the interlocutor; The distance from the first interlocutor to the monitor specified by the means, and a predetermined parallax angle at which it is determined that the line of sight of the first interlocutor and the line of sight of the monitor image of the second interlocutor match. A two-way interactive system characterized by being installed within a defined area. 2. The image pickup section is within the predetermined area defined by the distance specified by the position defining means and the predetermined parallax angle, and is the second area seen from the first interlocutor. The interactive interactive system according to claim 1, wherein the interactive interactive system is installed at a position where a part of the monitor image of the interlocutor is blocked. 3. The parallax angle for matching the line of sight of the first interlocutor with the line of sight of the monitor image of the second interlocutor is a monitor as seen from the first interlocutor with respect to the line of sight of the first interlocutor. The bidirectional interactive system according to claim 1, wherein the horizontal direction of the screen is 4.5 ° or less, the upward direction is 12 ° or less, and the downward direction is 8 ° or less. 4. The parallax angle for matching the line of sight of the first interlocutor and the line of sight of the monitor image of the second interlocutor is a monitor as seen from the first interlocutor with respect to the line of sight of the first interlocutor. 4. The bidirectional interactive system according to claim 3, wherein the horizontal direction and the vertical direction of the screen are each 3 degrees or less. 5. The first terminal device includes a processing unit for recognizing a position on the monitor of an image of the first interlocutor himself displayed on the monitor, the imaging device of the first terminal device. Includes a drive mechanism that moves the position of the imaging unit or changes the imaging angle of the imaging unit relative to the first interlocutor whose position is recognized by the processing unit. The interactive system of claim 1. 6. The second terminal device moves at least a position of a second spot image pickup unit that picks up an image of the second interlocutor, or a position of the second spot image pickup unit, or a second spot image pickup unit of the second spot image pickup unit. A second point drive mechanism for changing an imaging angle is provided, and the processing unit of the first terminal device is provided on the monitor of the image of the second interlocutor received through the transmission means and displayed on the monitor. The control signal for recognizing the position and moving the position of the second point image pickup unit or changing the image pickup angle of the second point image pickup unit relative to the second interlocutor is transmitted. 6. The two-way interactive system according to claim 5, wherein the two-way interactive system transmits to the second terminal device via means. 7. The first terminal device comprises a processing unit for recognizing the position of the image of the first interlocutor himself displayed on the monitor on the monitor, and the position defining means includes the processing unit. 2. A mechanism for adjusting the position of the eyes of the first interlocutor himself imaged with respect to the position of the eyes of the monitor image of the first interlocutor himself recognized by the unit. Two-way interactive system. 8. The second terminal device, at least a second point display unit for displaying an image of the first interlocutor on a monitor,
A second point position defining means for defining the position of the second interlocutor, the second point position defining means having a mechanism for adjusting the eye position of the second interlocutor with respect to the eye position of the monitor image; The processing unit of the first terminal device recognizes the position on the monitor of the image of the second interlocutor received through the transmission means and displayed on the monitor, and monitors the second point display unit. 8. A control signal for adjusting the eye position of the second interlocutor with respect to the eye position of the image is transmitted to the second terminal device via the transmitting means.
The described interactive system. 9. The two-way interactive system according to claim 1, wherein the support mechanism of the imaging device includes a transparent member for directly supporting the imaging unit. 10. The two-way interactive system according to claim 1, wherein the position defining means is a chair having a seat height adjusting mechanism. 11. The bidirectional system according to claim 1, wherein the position defining means is a table having a predetermined height, which is installed between the first interlocutor and the display unit. Interactive system. 12. The two-way interactive system according to claim 1, wherein the position defining means is a table having a predetermined height on which the display section is directly installed. 13. The first terminal device calculates a total dialogue time, an average dialogue time, and a dialogue count during a two-way dialogue between the first dialogue person and the second dialogue person, and displays on the display unit. 2. The bidirectional interactive system according to claim 1, further comprising an interactive processing unit that sequentially displays the total interactive time, the average interactive time, the number of times of interaction, and the frequency distribution of the interaction time. 14. The first terminal device includes a sound absorbing member having a property of blocking external sound, and the sound absorbing member is arranged at least at a part of the periphery of the first interlocutor. The two-way interactive system according to claim 1, wherein: 15. The first terminal device has been transmitted via a voice / acoustic data processing unit that captures the voice of the first interlocutor collected by a voice input unit as an electrical signal, and the transmitting unit. The two-way interactive system according to claim 1, further comprising a voice / sound output control unit that causes the voice output unit to reproduce the voice data of the second interlocutor. 16. The audio output unit includes a pair of speakers, and the pair of speakers are arranged on a side surface of the first interlocutor and near an ear. The described interactive system. 17. The two-way interactive system according to claim 15, wherein the voice input unit is a telephoto microphone. 18. The audio / acoustic data processing unit detects an average level of the collected sound volume for a predetermined time, and automatically adjusts the sound volume when a difference between the collected sound volume and the average level is a predetermined value or more. 16. The interactive system of claim 15, wherein the interactive system comprises: 19. The voice / sound output control unit detects that the conversation volume is substantially zero for a predetermined time, and based on the detection result, at least starts the background music and adjusts the increase in the volume of the background music. 16. The interactive system according to claim 15, wherein the volume of the background music is reduced. 20. The voice / acoustic data processing unit temporally analyzes a voice spectrum of the first interlocutor or a voice spectrum corresponding to the transmitted voice information of the second interlocutor to obtain an average spectrum and The interactive interactive system according to claim 15, wherein statistical analysis processing is performed to detect a voice spectrum that has changed by a predetermined value or more. 21. The display unit has a through hole that connects the back surface of the display unit with the monitor screen on the front surface of the display unit,
The image pickup unit is attached to the display unit by the support mechanism in a state in which a tip of the image pickup unit penetrates a through hole of the display unit and a tip of the image pickup unit projects forward of the monitor screen. A two-way interactive system according to claim 1. 22. The image pickup unit is attached to the display unit by the support mechanism in a state of being rotatable about a predetermined portion of the image pickup unit, whereby the image pickup direction of the image pickup unit is desired. 22. The interactive system of claim 21, wherein the interactive system is changed in direction. 23. The two-way interactive system according to claim 22, wherein the first terminal device further includes a drive mechanism that changes a mounting angle between the display unit and the imaging unit. 24. The first terminal device comprises a processing unit for adjusting the display position of the image of the second interlocutor displayed on the monitor, the processing unit being received via the transmission means. Based on the image of the second interlocutor displayed on the monitor, the imaging unit identifies a virtual position on the monitor image that should originally exist as a reference point, and the position where the imaging unit is actually installed. Based on the coordinates on the monitor corresponding to the above, and the coordinates on the monitor of the obtained reference point, the deviation of the monitor image of the second interlocutor is detected, and the coordinates of the reference point on the monitor are determined. Calculating a vector having the start point as the end point and the coordinates on the monitor corresponding to the installation position of the imaging unit as the end point, and translating the monitor image of the second interlocutor in parallel based on the obtained vector. Featuring Bidirectional interactive system of claim 1, wherein that. 25. The processing unit detects each display position on the monitor of both eyes of the image of the second interlocutor received via the transmission means and displayed on the monitor, 25. The two-way interactive system according to claim 24, wherein a position on the monitor, which is separated from the center of both eyes by a predetermined distance above the monitor, is specified as the reference point. 26. The processing unit specifies, as a reference point, a position on the monitor corresponding to a top of a monitor image of the second interlocutor received via the transmission means and displayed on the monitor. 25. The two-way interactive system of claim 24, wherein: 27. The first terminal device comprises a display control unit for enlarging or reducing a human image displayed on the monitor, the display control unit being in the vicinity of the crown of the human image, It is possible to detect a part of the human image corresponding to the position where the image pickup unit is actually installed, and to enlarge or reduce the entire human image with the detected part of the human image fixed. The two-way interactive system of claim 1 characterized. 28. The display control unit displays at least one of a character and a figure in a predetermined area on the monitor, excluding an area in which the person image is displayed, on the monitor. The interactive system of claim 27, wherein the interactive system comprises: 29. The two-way interactive type according to claim 1, wherein the first terminal device includes a person authenticating unit for performing person authentication of a user of the first terminal device before starting a dialogue. system. 30. The first terminal device is provided with a reading device for reading the predetermined items from a card-shaped external storage medium having at least an IC, in which the predetermined items are recorded in advance. 29. A two-way interactive system according to 29. 31. At least one of reservation information and information for authenticating a person of a user of the second terminal device who is a dialogue partner is recorded as the predetermined item in the external storage medium. 31.
The described interactive system. 32. The external recording medium records, as the predetermined items, at least a reserved dialogue start time, a reserved dialogue time, and a dialogue condition including dialogue contents. A two-way interactive system according to item 30. 33. In the external storage medium, dialogue reservation data including at least information regarding a reserved dialogue start time and dialogue partner is recorded as the predetermined item, and the first terminal device is configured to execute the dialogue reservation. It is characterized in that, after being line-connected to the second terminal device searched for via the communication network which is the transmitting means based on the data, bidirectional communication is started with the second terminal device. The interactive system of claim 30. 34. At least one of a character, a figure and an image is recorded as the predetermined item on the external recording medium, and the first terminal device stores information recorded on the external recording medium. The part is displayed on the monitor of the first terminal device at least at any stage before the start of the dialogue with the second interlocutor, during the dialogue, and after the end of the dialogue. 33. A two-way interactive system according to 33. 35. The first terminal device starts a dialogue with the second interlocutor, the control information for causing the second terminal device to display and output a part of the information recorded in the external recording medium. 35. The two-way interactive system according to claim 34, wherein the interactive system is transmitted to the second terminal device via the transmission means in at least one of a stage before, during a dialogue, and after a dialogue. 36. The first terminal device comprises an acoustic medium, which is recorded / stored in advance inside or outside the first terminal device and includes at least one of background music and voice.
35. The two-way interactive system according to claim 34, wherein the output is performed in conjunction with the display operation according to claim 32 in the first terminal device. 37. The first terminal device causes the second terminal device to output an acoustic medium containing at least one of background music and voice, which is recorded / stored in advance inside or outside the first terminal device. The control signal for
35. The two-way interactive system according to claim 34, wherein the two-way interactive system transmits data to the second terminal device via the transmission means in conjunction with the transmission operation described in claim 33 in the terminal device. 38. A housing and a monitor provided on the front surface of the housing for displaying an image of the first interlocutor transmitted through the transmission means, and the back surface of the housing and the front surface of the monitor are communicated with each other. A display unit having a through hole; an image capturing unit for capturing an image of a second interlocutor located in front of a monitor screen of the display unit; a tip of the image capturing unit penetrating the through hole of the display unit; A terminal device comprising a support mechanism for attaching the imaging unit to the display unit in a state where a tip of the unit protrudes forward of the monitor screen. 39. The image pickup unit is attached to the display unit by the support mechanism in a state of being rotatable about a predetermined portion of the image pickup unit, whereby the image pickup direction of the image pickup unit is desired. 39. The terminal device according to claim 38, wherein the terminal device is changed in direction. 40. A drive mechanism for changing a mounting angle between the display section and the image pickup section is provided.
9. The terminal device according to item 9. 41. A processing unit is provided for adjusting a display position on the monitor of the image of the first interlocutor displayed on the monitor, the processing unit being received via the transmission means on the monitor. Based on the displayed image of the first interlocutor, the virtual position on the monitor image where the imaging unit should originally exist is specified as a reference point, and the monitor corresponding to the position where the imaging unit is attached. Based on the coordinates above and the coordinates of the obtained reference point on the monitor, the deviation of the monitor image of the first interlocutor is detected, and the coordinates of the reference point are used as the starting point, and 39. The terminal device according to claim 38, wherein a vector having a coordinate corresponding to the installation position as an end point is calculated, and the monitor image of the first interlocutor is translated based on the obtained vector. . 42. The processing unit detects each display position on the monitor of both eyes of the image of the first interlocutor received via the transmission means and displayed on the monitor, 42. The terminal device according to claim 41, wherein a position on the monitor, which is separated from the center of both eyes by a predetermined distance above the monitor, is specified as the reference point. 43. The processing unit specifies, as a reference point, a position on the monitor corresponding to the top of the monitor image of the first interlocutor received via the transmission means and displayed on the monitor. 42. The terminal device according to claim 41, wherein: 44. The first terminal device includes a display control unit for enlarging or reducing a human image displayed on the monitor, the display control unit being in the vicinity of the crown of the human image, It is possible to detect a part of the human image corresponding to the position where the image pickup unit is actually installed, and to enlarge or reduce the entire human image with the detected part of the human image fixed. The two-way interactive system of claim 38, wherein the interactive system is characterized by: 45. The display control unit displays at least one of characters and figures in a predetermined area on the monitor, excluding an area in which the person image is displayed, on the monitor. The interactive system of claim 44, wherein the interactive system comprises: 46. An image pickup apparatus adapted to the two-way interactive system according to claim 1, wherein the image pickup unit picks up an image of the first interlocutor, and the image pickup direction of the image pickup unit is directly directed to the first interlocutor. In a state of having a transparent support member to which the image pickup unit is attached, a support mechanism for setting the image pickup unit at a predetermined position, and image data of the first interlocutor imaged by the image pickup unit. As a transmission to a predetermined transmission means,
And a processing unit for receiving the image data of the second interlocutor transmitted from the transmission means.