JP2003309828A - Method for correcting display position of monitor image, terminal, mount tool, and two-way interactive system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for correcting a display position of a monitor image or the like for sequentially detecting a position deviation in monitor image caused by a motion of an imaged object and correcting the display position of the monitor image in real time. <P>SOLUTION: Image coordinates (O<SB>2</SB>) of a particular position of an interactive person's image of an opposite side are calculated on the basis of image coordinates of one or more marker images (M1 to M3) suggesting the image coordinates (O<SB>2</SB>) of a particular part in order to realize coincidence of lines of sight between interactive people from an image including the interactive person's image of the opposite side and the interactive person's image of the opposite side displayed on a monitor screen (8) is moved so that the image coordinates (O<SB>2</SB>) of the particular part are placed on target coordinates (O<SB>1</SB>) of the part on the monitor screen (8) overlapped with an imaging section. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a monitor image display position correction method and the like applicable to a two-way interactive system for realizing remote communication via a terminal device connected by a predetermined transmission means. is there.

[0002]

2. Description of the Related Art In recent years, a two-way interactive system has been constructed to realize remote dialog for transmitting and receiving text data and video data between remote sites via a transmission means for learning English conversation and counseling. There is. In such a remote dialogue, not only the exchange of text data and the image of the other party's interlocutor, but also a more intimate situation in which the interlocutors indirectly see each other through the image of the other party's interlocutor is displayed. As a system for realizing a dialogue environment, for example, Patent No. 3
No. 074677 proposes a two-way interactive system that enables remote communication while keeping the line of sight.

[0003]

DISCLOSURE OF THE INVENTION As a result of studying a conventional two-way interactive system, the inventors have found the following problems. That is, in a conventional interactive interactive system for realizing a remote dialogue, in order to indirectly establish a line-of-sight matching state between the interlocutors, a specific part of the image of the other interlocutor displayed on the display unit such as a monitor ( For example, an image pickup unit (included in the image pickup device) such as a CCD camera for picking up an image of the interlocutor located in front of the display unit is arranged in a state of overlapping with the head above the eyes). Since the installation position of the image pickup unit is substantially fixed except for the fine adjustment of the image pickup direction and the like, it is necessary to adjust the position of the interlocutor with respect to the image pickup unit. However, in order to achieve a line-of-sight matching state in which the display area of the image of the interlocutor on the other side and the imaging unit are both fixed, it is complicated to adjust the position of the interlocutor himself / herself to be imaged, and also during dialogue If the person moves, it becomes impossible to capture images of the interlocutors in the line-of-sight matching state. As a result, there is a problem that it is difficult to maintain the eye-gaze matching state due to the movement of the interlocutor during the dialogue.

The present invention has been made in order to solve the above-mentioned problems, and sequentially detects the position shift of the monitor image due to the movement of the object being imaged and determines the display position of the monitor image. Method for correcting display position of monitor image for real-time correction, terminal device for realizing method for correcting display position of monitor image, mounting jig applied to the terminal device, bidirectional interactive system including the terminal device, display An object of the present invention is to provide a computer program that executes a position correction method and a recording medium having the computer program recorded therein.

[0005]

According to a method for correcting a display position of a monitor image according to the present invention, an interlocutor image imaged by an image pickup unit such as a CCD camera arranged in front of the monitor screen is transmitted through a predetermined transmission unit. Between the interlocutors by displaying the other interlocutor image at a predetermined position on the monitor screen so that a specific part of the other interlocutor image received by each other is overlapped with the imaging unit. In a two-way interactive system that realizes the line-of-sight matching state, a structure that enables real-time position correction of the interlocutor image is provided.

That is, in the monitor image display position correcting method according to the present invention, one or more marker images are extracted from the received image including the received interlocutor image, and the monitor screen of the extracted marker image is displayed. Based on the coordinates above, the coordinates of the specific part of the image of the other party's interlocutor on the monitor screen are calculated, and the image of the other party's interlocutor is moved to a display position where the line-of-sight match is possible between the interlocutors. The marker image is
We suggest a specific part to realize the line-of-sight matching between the interlocutors. Also, the movement of the other party dialogue person image is performed so that the other party dialogue person image displayed on the monitor screen is positioned at the target coordinates of the portion on the monitor screen where the specific portion thereof overlaps with the imaging unit.

In the monitor image display position correcting method according to the present invention, there are three or more marker images in the received image, and the center of gravity of the polygon formed by these three or more marker images is used as the image of the other party's interlocutor. It is calculated as a specific part of.
Further, at least two marker images may be present in the received image. In this case, after a new virtual marker image is determined from these marker images, the marker images and the polygons generated by the virtual marker images are formed. The center of gravity is calculated as the specific part of the image of the other party's interlocutor. Further, the specific part of the image of the interlocutor on the other side may be calculated based on the coordinates of the image of the marker image included in the received image, which is imaged around a predetermined line segment or point. It should be noted that if the coordinates of the marker image included in the received image on the monitor screen and the coordinates of the specific portion of the other-side interlocutor image on the monitor screen match, the specific portion can be easily detected. It is possible.

At least one terminal device of the two-way interactive system, which realizes the above-described method for correcting the display position of the monitor image (the terminal device according to the present invention).
Is a display unit, an image pickup unit such as a CCD camera arranged in front of the display unit, and an image of the other party dialogue person imaged by the other party image pickup unit and transmitted through a predetermined transmission means. And a control unit for displaying on the display unit so as to overlap with the imaging unit.

In particular, the control unit extracts one or more marker images from the received image containing the received image of the other party's interlocutor in order to realize the above-described method for correcting the display position of the monitor image according to the present invention. Then, based on the coordinates on the monitor screen of the extracted marker image, the coordinates on the monitor screen of the specific part of the image of the other party's interlocutor are calculated, and the line of sight of the other party's interlocutor image can be matched between the interlocutors. To a display position (target coordinates of a part on the monitor screen that overlaps with the imaging unit).

The terminal device is arranged in front of the display unit and within the image pickup area of the image pickup unit, and is provided with a marking for providing a marker image indicating a specific portion of the interlocutor image picked up by the image pickup unit. Means are preferably provided. The marking means may include more than two markers,
In this case, the control unit controls these 3 included in the received image.
The center of gravity of the polygon generated by the above marker image is calculated as the specific part of the image of the interlocutor on the other side. When the marking means includes at least two markers, the control unit determines a new virtual marker image from these two marker images included in the received image, and then generates the virtual marker image using these marker images and the virtual marker image. The center of gravity of the polygon is calculated as the specific part of the image of the other party's interlocutor.

The control unit may calculate the specific portion of the other-side interlocutor image based on the coordinates of the image of the marker image included in the received image, which is mapped around a predetermined line segment or point. Also, the marking means may include a marker that provides a marker image that matches the coordinates on the monitor screen of the specific portion of the image of the interlocutor on the other side. In any case, it can be information that easily specifies the coordinates on the monitor screen of the specific part of the image of the interlocutor on the other side.

Further, the terminal device according to the present invention preferably comprises a mounting jig for providing the marking means. That is, the mounting jig is arranged in front of the display unit and within the image capturing area of the image capturing unit, and marking means for providing a marker image indicating a specific portion of the interlocutor image captured by the image capturing unit. , A supporting unit that holds the marking means so that the relative positions of the interlocutor image and the marker image included in the received image are kept constant. The support section may be headphones, earphones, glasses, etc. worn by the interlocutor himself located in front of the display section.

The position shift detecting method as described above may be a program executed by a computer or the like. In this case, the program may be distributed via a network regardless of whether it is wired or wireless. , CD, DVD, flash memory or the like.

[0014]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of a method of correcting a display position of a monitor image according to the present invention will be described below in detail with reference to FIGS. In the description of the drawings, the same parts will be denoted by the same reference symbols and redundant description will be omitted.

A terminal device (terminal device according to the present invention) for realizing the method for correcting the display position of a monitor image according to the present invention,
It is applicable to a two-way interactive system described in, for example, Japanese Patent No. 3074677, which realizes a remote dialog between terminal devices connected via a predetermined transmission means.

First, the bidirectional interactive system to which the display position correcting method of the monitor image and the terminal device for realizing the same can be applied will be described here. In addition, this interactive interactive system is a system that realizes remote interaction in the state of line-of-sight matching, and is a interactive interactive system that can transmit and receive text data and image data in addition to transmitting and receiving the image of the interlocutor. is there.

FIG. 1 is a logical configuration diagram showing an entire interactive interactive system that enables line-of-sight matching between interlocutors at remote locations. This system realizes a two-way dialogue between an interlocutor 3A (first interlocutor) at point A (first point) and an interlocutor 3B (second interlocutor) at point B (second point). It is a system that does. In particular, this system is a counseling system that establishes intimate relationships of trust even with interlocutors in remote areas and enables close communication, and a system that aims at home learning. , Is supposed to be a counselor such as a psychologist. Therefore, this system has various functions and configurations so that these counselors can provide information necessary for counseling and an ideal counseling environment.

Specifically, at the point A, (a) a chair 4A (having a height adjusting mechanism 40A) on which an interlocutor 3A (for example, a counselor) sits, and (b) a table 2A to which the interlocutor 3A faces. (C) The terminal device 20A having a mechanism for matching the lines of sight of the interlocutors via the transmission means 27 is installed, and the interlocutor 3A is isolated by the partition 22A. On the other hand, at point B, (a) Interlocutor 3
A chair 4B (having a height adjustment mechanism 40B) on which B (for example, a client receiving counseling) sits, and (b)
The table 2B to which the interlocutor 3B faces, and (c) the transmission means 2
The terminal device 20B provided with a mechanism for matching the line of sight between the interlocutors via 7 is installed, and the interlocutor 3B is isolated by the partition 22B. Here, the transmission means 27 enables transmission and reception of image information and audio information between the terminal device 20A at the point A and the terminal device 20B at the point B, so as to enable transmission and reception of the optical fiber transmission line 31 (main transmission line) and the satellite. The communication transmission path 32 (transmission path for backup) and the like are included. The transmission means 27 may be wired or wireless. In addition, the transmission means includes a network such as a telephone line network that has already been installed, and a bulletin board system (BBS: Bulletin Board Syste
m) is also included.

At point A, the chair 4A described above functions to define the position of the interlocutor 3A. This chair 4A is preferably fixed in order to keep a constant distance from the display unit 21A of the terminal device 20A. However, even if the chair 4A is not fixed, the chair 4A can be installed by installing the table 2A on the interlocutor 3A and the display unit 21A.
It becomes possible to maintain a constant distance from the display unit 21A (the fact that the existence of the table 2A serves as a reference for the psychological distance of the interlocutor 3A has been confirmed). The configuration of the terminal device 20B at the point B where the other party's interlocutor 3B (for example, a person who receives counseling) is the same.

At the point A, the terminal device 20A is (1)
A monitor TV arranged on the table 2A for displaying the image of the interlocutor 3B based on the image information transmitted from the terminal device 20B at the point B via the transmission means 27.
A display unit 21A provided with 1A, (2) an image pickup unit 6A for taking in the image of the interlocutor 3A picked up by the CCD camera 5A as image information, and transmitting it to the terminal device 20B at the point B, (3 ) A voice output unit 23A for outputting the voice of the interlocutor 3B based on the voice information transmitted from the terminal device 20B at the point B via the transmission means 27.
(Speaker), and (4) a voice input unit 24A (microphone) for collecting the voice of the interlocutor 3A as voice information and transmitting it to the terminal device 20B at the point B. In addition, this system, as a counseling system, realizes more detailed communication.
Input a character or a graphic, display the input character or graphic on the display unit 21A, and transmit the character or graphic information to the terminal device 20B at the point B as a character or graphic information (keyboard, pointing, An image processing device 26A (including a control unit) which is provided between each component and the transmission means 27 and which performs signal processing, transmission control, video data editing work, etc. With. The terminal device 20B at the point B is also configured in the same manner as the terminal device 20A at the point A, and specifically, the display unit 21B and the CCD.
Imaging device 600 including camera 5B and imaging unit 6B
B, a voice output unit 23B, a voice input unit 24B, a character / graphics input unit 25B, and a video processing device 26B.

Further, the terminal devices 20 at the points A and B
In A and 20B, the configurations of the imaging devices 600A and 600B (imaging means) will be described. For convenience of description, when referring to components common to the terminal devices 20A and 20B at points A and B, for example, the imaging device 60
The description will be made by omitting the letters A and B that distinguish each point like 0. In addition, unless specifically described, the terminal device 20A at the point A is described in principle, and the terminal device 20 at the point B having a common configuration is described.
Overlapping description of B is omitted.

The image pickup device 600 includes a microminiature CCD camera 5 (5A) which is an image pickup section, and a support mechanism 10 for mounting the CCD camera 5 at a predetermined position in a supported state (see FIG. 2).
Reference unit) and an image pickup unit 6 (6A) for controlling the CCD camera 5. The monitor TV 1 (1A) is placed on the table 2 (2A), and the interlocutor 3
(3A) is a chair 4 (4) provided with a height adjusting mechanism 40 (40A) placed at a distance L (m) from the monitor TV 1.
Sitting in A). This distance L (LA) is set to 0.5 m or more and 5 m or less. An outer diameter of 20 mm or less and a length of about 100 mm between the interlocutor 3 and the monitor TV 1 to be photographed.
A cylindrical microminiature CCD camera 5 is provided. Further, the CCD camera 5 is installed so that its imaging direction is directed to the interlocutor 3 who is imaged. Then, the image of the interlocutor 3 imaged by the camera 5 is transmitted as image information from the imaging unit 6 to the other interlocutor side (transmitted to the terminal device 20B at the point B via the transmission means 27).
Depending on the interlocutor, the interval between the display unit 21 and the interlocutor may be slightly shifted. Therefore, in order to deal with such a situation, it is preferable that the CCD camera 5 has a deep focal depth.

Next, a specific installation position of the microminiature CCD camera 5 will be described with reference to FIG.

First, the image pickup unit 6 of the image pickup apparatus 600 is installed on the display section 21 for displaying the transmitted image of the other interlocutor. An image of the interlocutor himself / herself taken by the camera 5 is captured in the body of the imaging unit 6 (captured by the imaging unit 6 via the camera cable 50)
In addition to the changeover switch 6a, various adjustment switches 6b are provided for switching between the display of 1) and the display of the transmitted image of the interlocutor 3B. The switch 210 includes a power switch for turning on / off the power of the monitor TV 1. The CCD camera 5 is installed by a support mechanism 10 (robot arm) at a predetermined position in front of the monitor screen 8 on which the transmitted image 9 of the interlocutor 3B is displayed. The monitor screen 8 can be sufficiently used if it has a size of about 4 inches.

Specifically, the CCD camera 5 is at a position forward w (cm) from the monitor screen 8, and the monitor screen 8
It is arranged near the head 9a of the interlocutor image 9 displayed in FIG.
The central axis of the cylindrical CCD camera 5 is located at a position h (cm) above the eye position of the interlocutor image 9 shown by the dotted line.

As described above, since the microminiature CCD camera 5 is located near the head above the eye position of the interlocutor image 9 displayed on the screen 8 of the large monitor TV 1, it is special for bidirectional conversation. No trouble will occur. For example, the CCD camera 5 is provided above the eye position (the position shown by the dotted line in the figure) of the interlocutor image 9 displayed on the screen 8 at h = about 10 (cm), and between the monitor TV 1 and the interlocutor 3. In the case of the system configuration in which the distance L is set to about 2.5 (m), the parallax angle is 2.3 °.
It is possible to sufficiently fall below the detection limit parallax angle of 3 ° (the interval w between the monitor screen 8 and the CCD camera 5 is 10 (c
Even if there is about m), there is no particular effect on the variation of the parallax angle). That is, if the eyes of the other party (the eyes of the image of the other interlocutor 3B displayed on the monitor screen 8) can be clearly seen at the time of line-of-sight matching, even if the microminiature camera 5 is provided near the head, the monitor It has been confirmed that if the screen 8 of the TV 1 is large, there is almost no hindrance to the interactive dialogue. As a result of the experiment, the size of this screen 8 is 35 cm in width and 26 cm in height.
It has also been confirmed that good two-way dialogue can be realized if the size is at least a certain level. It should be noted that if mutual interlocutors are acquaintances, there is a result that there is a tendency that there are few psychological obstacles even if the size of the screen is small, and it is advisable to properly use the screen size according to the application.

Further, the parallax angle will be described with reference to FIG. In this specification, the parallax angle means, as shown in FIG. 3, the line of sight of the interlocutor 3 to be photographed, which is directed to the interlocutor image 9 displayed on the monitor screen 8, and the direction of the CCD camera 5. The angle θ formed by the line of sight of the interlocutor 3 is defined. In other words, it means the shift angle of the installation position of the CCD camera 5 with respect to the line of sight of the interlocutor 3. In addition, the line-of-sight matching here means the 4th Japan Electrical Engineers Union Meeting (N)
o. 1998), it means that the unnaturalness due to the installation position of the CCD camera 5 is below the allowable limit. Quantitatively, the parallax angle is 4.5 ° or less in the left and right direction (both temple sides of the image 9 with respect to the eye center A of the interlocutor image 9), and 12 ° directly above (with respect to the image center A, The head 9b side of the image 9) and 8 ° below (the body side of the image 9 with respect to the image center A) or less are within the line-of-sight matching range. Of course, the smaller the parallax angle, the better, and the detection limit is 3 ° or less in the horizontal and vertical directions. Therefore, the area in which the CCD camera 5 can be installed is the conical area shown in FIG. In addition,
This conical area is the center point A of the interlocutor image 9 displayed on the monitor screen 8 and the eye 30 of the interlocutor 3 (in this embodiment, the center of both eyes of the interlocutor image 9 is the center point of the interlocutor image 9). Defined as A), the line of sight 31 of the interlocutor 3 connecting
It is defined by the line-of-sight matching region 9b on the monitor screen 8 and the position of the eye 30 of the interlocutor 3 when the monitor is separated by a predetermined distance L (m).

When the size of the interlocutor image 9 displayed on the monitor screen 8 is small, the CCD camera 5 and the head of the interlocutor image 9 do not necessarily have to overlap each other. Rather, the head of the interlocutor 9 is hidden by the CCD camera 5 and becomes difficult to see. As described above, the condition for line-of-sight matching is a parallax angle of 3 °.
This means that the CCD camera 5 and the eyes of the interlocutor image 9 are present in the area (3), specifically about 3 cm (h in FIG. 2) at a position 50 cm away from the screen. Therefore, within this range, the line-of-sight matching state is maintained even if the CCD camera 5 and the head of the interlocutor image 9 are displaced. This dialogue image 9
The displacement of the CCD camera 5 can be realized by, for example, adjusting the image display position on the monitor screen 8 or adjusting the support portion of the CCD camera 5 to change the position of the CCD camera 5 itself.

FIG. 5 is a diagram showing a schematic configuration of the terminal device according to the present invention. This terminal device includes a display unit 21 such as a monitor (corresponding to the display units 21A and 21B in the interactive interactive system shown in FIG. 1) and a video processing unit 26 (FIG. 1).
(Corresponding to the video processing devices 26A and 26B in the two-way interactive system shown in FIG. 2), an external recording device 700, a pointing device 900 such as a mouse, and a display unit 21.
An image pickup device 600 including an image pickup unit such as a CCD camera 5 and an image pickup unit 6 for picking up an interlocutor positioned in front of the headphone 920, and headphones 920 (including a speaker for outputting the voice of the other interlocutor). The video processing unit 26 receives the image information from the image pickup unit 6 included in the image pickup unit and transmits / receives the image information of the other party's interlocutor via the transmission unit 27 (in the figure, I /
O), a control unit 263 for carrying out the positional deviation detection method according to the present invention, a database for storing character data, video data, etc. prepared in advance by the interlocutor, in addition to the transmitted / received image information. 262 (D / B in the figure), a drawing unit 264 for displaying a predetermined pattern on the display unit 21 in accordance with an instruction from the control unit 263, and capturing of position information from a pointing device 900 such as a mouse or the external recording device 700. A data input / output unit 265 (I / O in the figure) that enables data transfer, and a sound source controller 910 for supplying output voice data to the speaker of the headphone 920 are provided. The external recording device 700 is
For example, a magnetic tape, a CD, an optical disk, a hard disk, a DVD, a flash memory, etc. are included.

Specifically, in a two-way interactive system for realizing a remote dialogue between the interlocutors A and B, in each terminal device of the interlocutors A and B, the interlocutors A and B are placed in front of the interlocutors. As shown in FIG. 6A, the display unit 21
Etc. are installed. At this time, on the monitor screen 8 of the display unit 21 installed in front of the interlocutor A, the image 300B of the interlocutor B as the image of the interlocutor B is captured CC
It is displayed in a state where it partially overlaps with the ED camera 5. On the other hand, on the monitor screen 8 of the display unit 21 installed in front of the interlocutor B, the image 300A of the interlocutor A as the image of the interlocutor A is displayed.
Is displayed in a state where it partially overlaps with the CCD camera 5 which images the interlocutor B.

Normally, the line-of-sight matching of each interlocutor is realized in the above-mentioned installation state, but in such a bidirectional interactive system, the CCD camera 5 is used for the monitor screen 8 of the display unit 21.
Since the relative position to the interlocutor B is fixed, for example, when the interlocutor B being imaged moves, the interlocutor B is displayed on the monitor screen 8 viewed by the interlocutor A as shown in FIG. 6B. The position of the image 300B is generated. Even in such a case, in the display position correction method of the monitor image, the control unit 2
An image of the interlocutor 3B in which the position 63 is detected by detecting a specific portion (a portion overlapping the CCD camera 5 so that the line-of-sight matching state is maintained) in the image of the interlocutor B in the image sampled at a predetermined timing. 300B is moved so as to maintain the line-of-sight matching state with the interlocutor A.

Next, a specific display position correcting operation (the monitor image display position correcting method according to the present invention) in the control section 263 will be described with reference to the flowchart of FIG. Note that the following display position correction operation will be described as an image display operation in the two-way interactive system shown in FIG.

First, when the I / O 261 captures the image 500a (see FIG. 9A) including the image of the other party's interlocutor transmitted via the transmission means 27, the control section 263 performs sampling at a predetermined timing. A marker image is extracted from the received image thus obtained (step ST1), and the coordinate O ₂ on the monitor screen 8 of the head reference point of the image of the other party's interlocutor is calculated as an image specifying portion for line-of-sight matching (step ST).
2).

Here, the terminal device is equipped with a jig for attaching marking means for extracting a specific portion to be displayed so as to overlap with the CCD camera 5 in order to realize the line-of-sight matching state of the interlocutor image. A headphone 920 or the like equipped with a speaker for audio output is practical as this attachment jig. Specifically, as shown in FIG.
The mounting jig has marking means including markers M1 to M3 arranged near the head of the interlocutor, and also includes a support portion such as headphones 920 that holds the marking means. The control unit 263 has three markers M1 to M3 located near the image head 350 in the received image.
Image is extracted, the center of gravity of the triangle generated from these marker images is calculated, and this center of gravity is set as the coordinate O ₂ on the monitor screen of the specific portion of the interlocutor image.

Normally, when the lines of sight coincide with each other, the specific part of the image of the interlocutor on the other side included in the monitor image 800, which is a part of the received image 500a, is aligned with the coordinate O _{1 of the} part overlapping the CCD camera 5. (See FIG. 9A). However, when the interlocutor himself moves, the specific part deviates from the target coordinate O ₁ on the monitor screen 8 and moves as shown in FIG. 9B.

Therefore, in the monitor image display position correcting method according to the present invention, the coordinates O _{2 of} the head reference point calculated in step ST2 on the monitor screen 8 and the CCD camera 5 on the monitor screen 8 overlap. An error ΔX in the X-axis direction and an error ΔY in the Y-axis direction on the monitor screen 8 with respect to the target coordinate O ₁ are calculated (step ST
3).

Then, the control section 263 determines whether or not the calculated X-coordinate shift component ΔX and Y-coordinate shift component ΔY are less than the allowable errors ε ₁ and ε ₂ , respectively (step ST4). If at least one of the deviation components ΔX and ΔY exceeds the permissible errors ε ₁ and ε ₂ , the control unit 26
3 determines that the position shift has occurred, and the displayed dialogue image of the other party is indicated by an arrow S in FIG. 9 (b) so that the specific portion of the image is matched with the target coordinate O ₁ on the monitor screen 8. It is moved in the indicated direction (step ST5). On the contrary, when both the deviation components ΔX and ΔY are less than the allowable errors ε ₁ and ε ₂ , the control unit 263 determines that the positional deviation has not occurred, and newly adds the above-mentioned step ST1 at the next sampling timing. By repeating steps ST4 to ST4, the display position of the monitor image is corrected in real time.

In the above embodiment, three markers M are used as means for calculating the specific part of the image of the interlocutor on the other side.
Although a mounting jig (headphones 920) having 1 to M3 is shown, the determination of the specific portion is not limited to such an embodiment. For example, as shown in FIG. 10A, even if two markers M1 and M2 are attached to the headphones 920 as marking means, it is possible to calculate the specific part of the image of the other party's interlocutor. That is, as shown in FIG. 10B, the image coordinates of the virtual marker P are determined by using two auxiliary lines that are orthogonal to each other. Then, the center of gravity of the triangle specified by the image coordinates of the markers M1 and M2 and the image coordinates of the newly obtained virtual marker P is calculated, and the obtained center of gravity is set as the specific portion O ₂ of the image of the interlocutor on the other side. Good.

Furthermore, various modifications can be made to the calculation means of the specific part of the image of the interlocutor on the other side. For example, as shown in FIG. 11, a marker M4 that directly suggests a specific part of the image of the interlocutor's image may be used as the marking means. Further, as shown in FIG. 12, a line-symmetrical image of the marker M5 attached to the character of a specific shape with respect to the line L center (image shown by a dotted line in the figure) is used, and the other side dialogue is performed. It is also possible to calculate the coordinate O ₂ on the monitor screen 8 of the specific portion of the person image.

[0040]

As described above, according to the present invention, a specific portion of the image of the other party's interlocutor for realizing the line-of-sight matching state between the interlocutors is suggested in the image containing the image of the other party's interlocutor. Since the marker image is introduced, the displacement of the monitor image on the monitor screen can be easily detected, and even if the imaged interlocutor himself moves frequently, There is an effect that the indirect line-of-sight matching state can be substantially maintained.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a two-way interactive system that realizes line-of-sight matching between remote places via transmission means.

FIG. 2 is a diagram showing the image pickup device shown in FIG. 1, and a diagram for explaining a method of installing a CCD camera for line-of-sight matching.

FIG. 3 is a diagram illustrating a parallax angle.

FIG. 4 is a diagram showing an area where a CCD camera is to be installed.

FIG. 5 is a block diagram showing a configuration of a terminal device (terminal device according to the present invention) that realizes the monitor image display position correction method according to the present invention.

FIG. 6 is a diagram for explaining a line-of-sight mismatch state in a two-way dialogue.

FIG. 7 is a flow chart for explaining a display position correction method for a monitor image according to the present invention.

FIG. 8 is a diagram conceptually illustrating an example of a specific portion extracting operation in the monitor image display position correcting method according to the present invention.

FIG. 9 is a diagram for conceptually explaining the position shift amount calculation operation in the monitor image display position correction method according to the present invention.

FIG. 10 is a diagram conceptually explaining another example of the specific portion extracting operation in the monitor image display position correcting method according to the present invention.

FIG. 11 is a diagram showing an application example of marking means.

FIG. 12 is a diagram showing another application example of the marking means.

[Explanation of symbols]

8 ... Monitor screen, 21 (21A, 21B) ... Display section, 3
(3A, 3B) ... Interlocutors A, B, 5 (5A, 5B) ... C
CD camera, 6 (6A, 6B) ... Imaging unit, 600
(600A, 600B) ... Imaging device, 26 (26A, 2
6B) ... video processing device, 262 ... D / B, 263 ... control unit, 800 ... image, 900 ... mouse (pointing /
Device), 910 ... Headphones, 920 ... Sound source controller, M1 to M5 ... Markers.

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Hisao Maki             Sumitomoden 1 Taya-cho, Sakae-ku, Yokohama-shi, Kanagawa             Ki Industry Co., Ltd. Yokohama Works F term (reference) 5C064 AA01 AC02 AC06 AC12 AC13                       AC16 AC22

Claims (16)

[Claims] 1. An interlocutor image captured by an image capturing unit arranged in front of a monitor screen is transmitted to each other via a predetermined transmission means, and a specific portion of the received interlocutor image is the image capturing unit. A method for correcting the display position of a monitor image in a two-way interactive system that indirectly realizes a line-of-sight matching state between the interlocutors by displaying the other-side interlocutor image at a predetermined position on the monitor screen so that And extracting one or more marker images suggesting the specific part for realizing the line-of-sight matching state between the interlocutors from the received image including the received interlocutor image, and the extracted Based on the coordinates of the marker image on the monitor screen, the coordinates of the specific part of the partner dialogue image on the monitor screen are calculated, and the partner dialogue image displayed on the monitor screen is calculated. A method for correcting the display position of a monitor image, in which the specific portion of the monitor image is moved so as to be located at the target coordinates of the portion on the monitor screen overlapping the image capturing unit. 2. The marker image exists in three or more in the received image, and the center of gravity of a polygon generated by these three or more marker images is calculated as a specific part of the image of the other party's interlocutor. The method for correcting the display position of a monitor image according to claim 1. 3. At least two marker images are present in the received image, and after a new virtual marker image is determined from these marker images, the center of gravity of a polygon formed by these marker images and the virtual marker image is determined. The method for correcting the display position of a monitor image according to claim 1, wherein the method is calculated as a specific part of the image of the other party's interlocutor. 4. The marker image included in the received image,
2. The method for correcting the display position of a monitor image according to claim 1, wherein the display position is calculated as a specific part of the image of the interlocutor on the other side based on the coordinates of the image that is mapped around a predetermined line segment or point. 5. The coordinates of the marker image included in the received image on the monitor screen and the coordinates of the specific portion of the image of the interlocutor on the monitor screen coincide with each other. A method for correcting the display position of the monitor image described. 6. A display part, an image pickup part arranged in front of the display part, and an image of the other party's interlocutor imaged by the other party's image pickup part and transmitted through a predetermined transmission means, A terminal device comprising: a control unit for displaying on the display unit so as to overlap with the imaging unit, wherein the control unit is a line-of-sight matching state between interlocutors from a received image including the received image of the other interlocutor. One or more marker images suggesting the specific portion for realizing the above are extracted, and based on the coordinates of the extracted marker image on the display unit, the specific portion of the other-side interlocutor image is extracted. A terminal device that calculates coordinates on the display unit and moves the other-side interlocutor image displayed on the display unit so that the specific region is located at the target coordinates of the region on the display unit that overlaps with the imaging unit. . 7. Marking means arranged in front of the display unit and within an imaging region of the imaging unit, for providing the marker image indicating a specific portion of the interlocutor image imaged by the imaging unit. The terminal device according to claim 6, wherein the terminal device is provided. 8. The marking means includes three or more markers, and the control unit defines a polygonal center of gravity generated by the three or more marker images included in the received image as the other-side interlocutor image. The terminal device according to claim 7, which is calculated as the specific part of 9. The marking means includes at least two markers, and the control unit newly determines a virtual marker image from these two marker images included in the received image, and then, the marker image and the virtual marker. The terminal device according to claim 7, wherein the center of gravity of the polygon generated by the image is calculated as a specific portion of the image of the other party's interlocutor. 10. The control unit calculates the specific portion of the partner interlocutor image based on the coordinates of the image of the marker image included in the received image, which is mapped around a predetermined line segment or point. The terminal device according to claim 6, wherein 11. The terminal device according to claim 6, wherein the marking unit includes a marker that provides a marker image that matches a coordinate on the monitor screen of a specific portion of the image of the other party's interlocutor. 12. A mounting jig applied to the terminal device according to claim 6, wherein the mounting jig is arranged in front of the display unit and within an imaging area of the imaging device, Marking means for providing the marker image that suggests a specific part of the interlocutor image captured by the device, and to keep the relative position of the interlocutor image and the marker image included in the received image constant, A mounting jig having a supporting portion for holding the marking means. 13. The mounting jig according to claim 12, wherein the support portion includes headphones. 14. A two-way interactive system including the terminal device according to any one of claims 6 to 11. 15. A computer program for executing the method according to claim 1 by a computer. 16. A recording medium on which is recorded a computer program for executing the method according to claim 1 by a computer.