JP2010239583A - Communication terminal unit, method of controlling communication of communication terminal unit, and communication control program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a communication terminal unit for performing conversation in natural situations while looking away moderately, and to provide a method of controlling the communication of a communication terminal unit, and a communication control program. <P>SOLUTION: It is monitored whether eyes of a user of a terminal device unnaturally coincide with an opposite user displayed on a screen during a teleconference (S5). When an unnatural coincident state of one's eyes is detected (S5:YES), a signal (first signal) which indicates that the unnatural coincident state of one's eyes has been detected is sent to a terminal unit of the opposite user (S9). In the terminal unit of the opposite user, the position of a video camera for photographing the opposite user is changed when the first signal is received, thus avoiding the unnatural coincident state of one's eyes among members participating in a conference, thus enabling the members participating in the conference to talk smoothly. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a communication terminal device capable of bidirectionally transmitting and receiving an image and sound to and from a counterpart communication terminal device, a communication control method for the communication terminal device, and a communication control program.

  2. Description of the Related Art Conventionally, there has been known a video conference system in which a plurality of communication terminal devices are connected via a network and images and sound are bidirectionally transmitted and received to realize a conference between persons in remote locations. In order to realize an advanced video conference system with a sense of presence, it is indispensable to match the line of sight of the conference participants. Therefore, various methods for realizing line-of-sight matching by using a liquid crystal display or a projector have been proposed.

  For example, a display and imaging apparatus is known in which a projector is disposed obliquely behind the back surface of the screen and a video camera is disposed vertically on the back surface of the screen (see, for example, Patent Document 1). The projector displays a user's photographed image taken at another site on the screen. The video of the video camera is output from a projector of a display imaging device arranged at another site. With this arrangement, when the conference attendee stares at the screen, the conference attendee's line of sight is directed to the video camera through the screen. Therefore, it is possible to match the line of sight between the attendees.

JP-A-6-133111

  However, when talking in daily life, it is rare to have a conversation while keeping the line of sight with the other party. Depending on the situation, the speaker not only adjusts his / her line of sight to the other party, but also naturally talks with the other party. On the other hand, when a video conference is performed using the display imaging device described in Patent Document 1, if the conference attendee continues to stare at the screen, it is difficult to naturally remove the gaze from the other party. Continue to match. In this case, there was a problem that it would be uncomfortable and unnatural.

  The present invention has been made to solve the above-described problems, and provides a communication terminal device, a communication control method for the communication terminal device, and a communication control program capable of having a conversation in a natural situation while appropriately removing the line of sight The purpose is to do.

  In order to achieve the above object, a communication terminal device according to a first aspect of the present invention is a communication terminal device that communicates with another terminal connected via a network via an image and a sound. User photographing means for photographing, image transmitting means for transmitting the image of the user photographed by the user photographing means to another terminal, display means for displaying the image transmitted from the other terminal on the screen, and the user The line-of-sight position on the screen displayed by the display unit displayed by the display unit ahead of the line-of-sight direction detected by the line-of-sight direction detected by the line-of-sight direction detection unit and the line-of-sight direction detected by the user The line-of-sight position specifying means, the face detection means for detecting the user's face on the screen, and the line-of-sight position specified by the line-of-sight position specifying means are detected by the face detection means. The line-of-sight matching detection means for detecting a state within the predetermined area of the user's face as a line-of-sight matching state, and the line-of-sight matching state is continued when the line-of-sight matching detection is detected by the line-of-sight matching detection means Gaze coincidence time measuring means for measuring gaze coincidence time, which is a time to perform, and first predetermined time judgment for judging whether or not the gaze coincidence time measured by the gaze coincidence time measuring means is equal to or longer than a first predetermined time And a terminal that transmits the image data of the other user who is the target of the line-of-sight matching state when the line-of-sight matching time is determined to be equal to or longer than the first predetermined time by the first predetermined time determination unit. A first signal transmitting means for transmitting a first signal indicating that the line-of-sight matching time is equal to or longer than the first predetermined time to a counterpart terminal; and transmitted from the counterpart terminal. First signal receiving means for receiving the first signal, and photographing position changing means for changing the photographing position of the user photographing means with respect to the user when the first signal is received by the first signal receiving means; It has.

  According to a second aspect of the present invention, in addition to the configuration of the first aspect of the invention, the photographing position changing means moves the photographing means to a position where the photographing means is retracted from the user's line-of-sight direction. It is characterized by.

  In addition to the configuration of the invention described in claim 1 or 2, the communication terminal device of the invention according to claim 3 further includes irradiation means for irradiating the user's eyeball with invisible light, and the line-of-sight direction detection means Detects the line-of-sight direction from the relative position between the virtual image (Purkinje image) of the invisible light of the user's eyeball cornea in the image photographed by the user photographing means and the eyeball pupil center of the user. It is characterized by doing.

  According to a fourth aspect of the present invention, in addition to the configuration of the third aspect of the invention, the communication terminal device further includes a positional relationship storage unit that stores a positional relationship among the user's eyeball, the screen, and the irradiation unit. The line-of-sight position specifying unit specifies the line-of-sight position on the screen from the positional relationship stored in the positional relationship storage unit and the line-of-sight direction detected by the line-of-sight direction detecting unit. Features.

  According to a fifth aspect of the present invention, there is provided the communication terminal device according to the third or fourth aspect, wherein the user photographing unit is arranged behind the screen with respect to the user, and the irradiation unit. Is attached to the user photographing means.

  In addition to the configuration of the invention according to any one of claims 1 to 5, a communication terminal device according to a sixth aspect of the invention includes a different image different from the user image on the screen together with the image. Another image display determination means for determining whether or not the image is displayed, and the first predetermined time determination means determines that the line-of-sight coincidence time has reached the first predetermined time or more, and the other image determination means determines the screen. Another image enlargement display means for enlarging and displaying the another image on the screen in comparison with a current display size and displaying the image in a transparent manner on the screen when it is determined that the other image is displayed on the screen. It has.

  According to a seventh aspect of the present invention, in the communication terminal device according to the sixth aspect, in addition to the configuration of the sixth aspect, the separate image enlargement display unit displays the separate image on the predetermined area of the user image. It is characterized by doing.

  According to an eighth aspect of the present invention, there is provided a communication terminal apparatus according to any one of the first to seventh aspects, in addition to the time elapsed since the first signal was transmitted by the first signal transmission means. A post-transmission elapsed time measuring means for measuring the post-transmission elapsed time, and a second predetermined for determining whether or not the post-transmission elapsed time measured by the post-transmission elapsed time measuring means is equal to or longer than a second predetermined time. When the elapsed time after transmission is determined to be equal to or longer than the second predetermined time by the time determination means and the second predetermined time determination means, the post-transmission elapsed time is transmitted to the second predetermined time The second signal transmitting means for transmitting a second signal indicating that the time is over, the second signal receiving means for receiving the second signal, and the second signal receiving means. The second signal If it is, further comprising an imaging position returning means for returning the photographing position to the original position of the imaging means.

  According to a ninth aspect of the present invention, in addition to the configuration of the first aspect of the present invention, the communication terminal device further comprises a user voice detecting means for detecting the voice of the user, and a voice of the counterpart user. The other party voice detecting means for detecting, and the silent time for measuring the silent time, which is the time during which the silent state continues, with the silent state being the state where none of the user voice detecting means and the other user voice detecting means detects the voice It further comprises a measuring means and a first second predetermined time adjusting means for adjusting the second predetermined time based on the silent time measured by the silent time measuring means.

  According to a tenth aspect of the present invention, in addition to the configuration of the ninth aspect of the invention, the first second predetermined time adjusting means includes the silent time measured by the silent time measuring means. Is provided with a third predetermined time determining means for determining whether or not the third predetermined time is exceeded, and the third predetermined time determining means determines that the silent time has become the third predetermined time or longer. Further, the second predetermined time is extended.

  According to an eleventh aspect of the present invention, there is provided a communication terminal apparatus according to any one of the first to tenth aspects of the present invention, the blink detection means for detecting the blink of the user, and the blink detection means. Further, a blink counting means for counting the number of blinks, and a second second predetermined time adjusting means for adjusting the second predetermined time based on the number of times counted by the counting means.

  According to a twelfth aspect of the present invention, in addition to the configuration of the eleventh aspect of the invention, the second second predetermined time adjustment means detects the line-of-sight matching state by the line-of-sight matching detection means. Measuring means for measuring an elapsed time since being done, fourth predetermined time determining means for determining whether or not the elapsed time measured by the measuring means is equal to or longer than a fourth predetermined time, and the fourth predetermined time A blink number determination means for determining whether or not the number of times counted by the blink count means is equal to or greater than a predetermined number when the elapsed time is determined by the determination means to be equal to or greater than the fourth predetermined time; And the second predetermined time is extended when it is determined by the blink number determination means that the number of times is a predetermined number or more.

A communication control method for a communication terminal device according to a thirteenth aspect of the present invention is a communication control method for a communication terminal device that performs communication with other terminals connected via a network via images and sounds. An image data transmitting step for transmitting to the other terminal an image of the user taken by the user photographing means for photographing the user, a display step for displaying the image transmitted from the other terminal on the screen, and the line-of-sight direction of the user A line-of-sight position detecting step, a line-of-sight position identifying step for identifying a line-of-sight position on the screen displayed by the display means ahead of the line-of-sight direction detected in the line-of-sight direction detecting step, The face detection step of detecting a user's face and the eye gaze position specified in the eye gaze position specifying step are detected in the face detection step. The line-of-sight matching state is continued when it is determined in the line-of-sight matching state in the line-of-sight matching determination step that determines whether or not the line-of-sight matching state is within a predetermined region of the user's face. A line-of-sight coincidence time measuring step for measuring a line-of-sight coincidence time, and a first predetermined time determination for determining whether or not the line-of-sight coincidence time measured in the line-of-sight coincidence time measuring step is equal to or longer than a first predetermined time And the partner side that transmits the image data of the partner user who is the subject of the line-of-sight matching state when it is determined in the step and the first predetermined time determination step that the line-of-sight matching time is equal to or longer than the first predetermined time A first signal transmission step of transmitting to the terminal a first signal indicating that the line-of-sight coincidence time is equal to or longer than the first predetermined time; A first signal receiving step of receiving,
In the first signal receiving step, when the first signal transmitted from the counterpart terminal is received, a photographing position changing step of changing a photographing position of the photographing unit with respect to the user is provided. To do.

  According to a fourteenth aspect of the present invention, a communication control program causes a computer to function as various processing means of the communication terminal device according to any one of the first to twelfth aspects.

  In the communication terminal device according to the first aspect of the present invention, communication via the image and sound can be performed with the counterpart terminal. The user is photographed by the user photographing means. The user image taken by the user photographing means is transmitted to the counterpart terminal by the image transmitting means. The image of the partner user transmitted from the partner terminal is displayed on the screen by the display means. The face of the opponent user is detected by the face detection means from the image of the opponent user displayed on the screen by the display means. On the other hand, the gaze direction of the user is detected by the gaze direction detection unit from the image captured by the user imaging unit. The line-of-sight position on the screen ahead of the line-of-sight direction detected by the line-of-sight direction detecting means is specified by the line-of-sight position specifying means. When the line-of-sight position specified by the line-of-sight position specifying unit is within a predetermined area of the face of the opponent user detected by the face detection unit, the line-of-sight matching detection unit detects the line-of-sight matching state.

  When the line-of-sight matching state is detected by the line-of-sight matching detection unit, the line-of-sight matching time is measured by the line-of-sight matching time measuring unit. Then, the first predetermined time determining means determines whether or not the measured line-of-sight matching time is equal to or longer than the first predetermined time. When it is determined by the first predetermined time determination means that the line-of-sight coincidence time is equal to or longer than the first predetermined time, the first signal transmitting means confirms that the line-of-sight coincidence time is equal to or longer than the first predetermined time. The first signal shown is transmitted. The first signal transmitted from the counterpart terminal is received by the first signal receiving means. When the first signal is received by the first signal receiving unit, the shooting position of the user shooting unit with respect to the user is changed by the shooting position changing unit.

  When the user has been looking at the other user on the screen for an unnatural long time, the photographing position of the photographing means for photographing the other user can be changed. Can be shifted. Therefore, the line of sight of the partner user displayed on the screen can be shifted, and an uncomfortable and unnatural situation where the line of sight continues to match between the user and the partner user can be avoided.

  In the communication terminal device according to the second aspect of the invention, in addition to the effect of the invention according to the first aspect, the photographing means is moved to a position where it is retracted from the direction of the user's line of sight by the photographing position changing means. Since the other user's line of sight can be reliably removed from the photographing means, the other user's line of sight reflected on the screen can be reliably shifted.

  According to a third aspect of the present invention, in addition to the effect of the first or second aspect of the invention, the irradiating means irradiates the user's eyeball with invisible light, and the gaze direction detecting means The line-of-sight direction is detected from the relative position of the virtual image (Purkinje image) on the eyeball cornea reflection surface of the user's eyeball cornea in the image photographed by the means and the user's eyeball pupil center. Therefore, the user's line-of-sight direction can be detected with high accuracy without using a large-scale device. Moreover, since the irradiated light is invisible light, a user's gaze direction can be detected without giving a user a sense of incongruity.

  According to a fourth aspect of the present invention, in addition to the effect of the third aspect of the invention, the positional relationship storage unit stores the positional relationship among the user's eyeball, the screen, and the irradiation unit. The line-of-sight position specifying unit specifies the line-of-sight position on the user's screen from the positional relationship stored in the positional relationship storage unit and the line-of-sight direction detected by the line-of-sight direction detecting unit. Therefore, the line-of-sight position on the user's screen can be specified without using a large-scale device.

  According to a fifth aspect of the present invention, in addition to the effect of the third aspect of the present invention, the user photographing means is arranged behind the screen with respect to the user. Therefore, when the user looks at the screen, the user's line of sight is directed to the user photographing means through the screen. Therefore, the line of sight between the user and the other user can be naturally matched. Further, since the irradiating means is attached to the user photographing means, the Purkinje image and the eyeball pupil center coincide with each other when the user's line of sight is directed to the user photographing means. Therefore, the user's line-of-sight direction can be detected with reference to the case where the user's line-of-sight is directed to the user photographing means.

  In addition to the effect of the invention according to any one of claims 1 to 5, the communication terminal device according to a sixth aspect of the invention has a different image different from the image of the other user on the screen by the different image display determining means. It is determined whether the image is displayed together with the image. When the first predetermined time determining means determines that the line-of-sight matching time has reached the first predetermined time and the other image determining means determines that another image is displayed on the screen, Different different images are enlarged and displayed on the screen, and are displayed transparently on the other user's image. Therefore, when the line of sight continues to match unnaturally, the image of the partner user who is the target of line-of-sight matching becomes inconspicuous. Therefore, an unnatural unnatural situation can be avoided.

  In addition to the effect of the invention according to claim 6, the communication terminal device according to claim 7 displays another image superimposed on a predetermined area of the image of the other user by the different image enlargement display means. Therefore, the image of the other user who is the subject of line-of-sight matching is certainly not conspicuous. Thus, uncomfortable and unnatural situations can be reliably avoided.

  According to an eighth aspect of the present invention, in addition to the effect of the first aspect, the post-transmission elapsed time measuring means transmits the first signal by the first signal transmitting means. Measure the elapsed time after transmission, which is the elapsed time since then. The second predetermined time determining means determines whether or not the post-transmission elapsed time measured by the post-transmission elapsed time measuring means is equal to or longer than the second predetermined time. When the second predetermined time determining means determines that the elapsed time after transmission has become equal to or longer than the second predetermined time, the second signal transmitting means notifies the counterpart terminal that the elapsed time after transmission has exceeded the second predetermined time. A second signal indicating that this is transmitted. When the second signal receiving means receives the second signal, and the second signal receiving means receives the second signal transmitted from the counterpart terminal, the photographing position returning means has the photographing position of the photographing means. Return to the position.

  Therefore, when the time from when the shooting position of the shooting means to the user is changed is equal to or longer than the second predetermined time, the shooting position of the shooting means can be returned to the position before the change. Therefore, after the unnatural line-of-sight matching state between the user and the partner user is avoided, the lines of sight of the user and the partner user can be matched again. Therefore, the users can in principle meet each other and hold a meeting only when there is an awkward atmosphere.

  In the communication terminal device according to the ninth aspect of the invention, in addition to the effect of the invention according to any one of the first to eighth aspects, the user voice detecting means detects the voice of the user. The partner user voice detecting means detects the voice of the partner user. The silent time measuring means measures the silent time, which is the time during which the silent state continues, with the state where neither the user voice detecting means nor the partner user voice detecting means detects the voice as the silent state. The first second predetermined time adjusting means adjusts the second predetermined time based on the silent time measured by the silent time measuring means.

  In general, in a silent state where neither the user nor the other user speaks, it can be said that the atmosphere is awkward as compared to the case where a conversation is being performed between the user and the other user. Since the communication terminal device of the invention according to claim 9 can adjust the time to remove the line of sight according to the time during which the silent state continues, the time to remove the line of sight according to the atmosphere of the users during the meeting. Can be adjusted.

  According to a tenth aspect of the present invention, in addition to the effect of the ninth aspect of the invention, the third predetermined time determining means determines whether or not the silent time has reached the third predetermined time or more. To do. The first second predetermined time adjusting means extends the second predetermined time when the third predetermined time determining means determines that the silent time has become equal to or longer than the third predetermined time. Therefore, when the silent state continues for a long time, the time for removing the line of sight can be extended. Therefore, when the line-of-sight matching state is unnatural between users in an awkward atmosphere, it is possible to extend the time for removing the line of sight and reliably avoid the awkward atmosphere.

  According to an eleventh aspect of the present invention, in addition to the effect of the invention according to any one of the first to tenth aspects, the blink detection means detects a user's blink, and the blink count means uses the blink detection means. Count the number of detected blinks. The second second predetermined time adjusting means adjusts the second predetermined time based on the number of blinks counted by the counting means.

In general, it is said that the number of blinks of the user increases when the user is in tension.
Since the communication terminal device of the invention according to claim 11 can adjust the time to remove the line of sight according to the number of blinks of the user, the time to remove the line of sight is adjusted according to the tension state of the user during the meeting. be able to.

  In addition to the effect of the invention according to claim 11, the communication terminal device of the invention according to claim 12 measures the elapsed time after the line-of-sight matching state is detected by the line-of-sight matching detection means. The fourth predetermined time determining means determines whether or not the elapsed time has reached or exceeded the fourth predetermined time. The blink number determination means determines whether or not the counted number of blinks is equal to or greater than a predetermined number when the elapsed time is equal to or greater than a fourth predetermined time. The second second predetermined time adjusting means extends the second predetermined time when the number of blinks is equal to or greater than the predetermined number.

  When the number of blinks of the user is large, it is possible to extend the time to remove the line of sight. Therefore, when the user who unnaturally matched the line of sight with the other user is in a tension state, the time to remove the line of sight Can be extended. Therefore, when there is an awkward atmosphere with the other user as the user becomes nervous, it is possible to extend the time to remove the line of sight and reliably avoid the awkward atmosphere.

  In the communication control method for a communication terminal device according to the thirteenth aspect of the invention, first, in the image data transmission step, the user image taken by the user photographing means for photographing the user is transmitted to another terminal. Next, in the display step, an image transmitted from another terminal is displayed on the screen. Next, in the gaze direction detection step, the user's gaze direction is detected. The line-of-sight position on the screen ahead of the detected line-of-sight direction is specified in the line-of-sight position specifying step. In the face detection step, the face of the partner user on the screen is detected. In the line-of-sight matching determination step, it is determined whether or not the line-of-sight position specified in the line-of-sight position specifying step is in a line-of-sight matching state within a predetermined area of the face of the opponent user detected in the face detection step. When it is determined in the line-of-sight matching determination step that the line-of-sight coincidence state is reached, the line-of-sight matching time is measured in the line-of-sight matching time measurement step. In the first predetermined time determination step, it is determined whether or not the measured line-of-sight matching time is equal to or longer than the first predetermined time.

  When the line-of-sight matching time is equal to or longer than the first predetermined time, in the first signal transmission step, the first signal is transmitted to the partner terminal that transmits the image data of the partner user who is the target of the line-of-sight matching state. In the first signal receiving step, the first signal is received. When the first signal transmitted from the counterpart terminal is received, the photographing position of the photographing unit with respect to the user is changed in the photographing position changing step.

  When the user has been looking at the other user on the screen for an unnatural long time, the photographing position of the photographing means for photographing the other user can be changed. Can be shifted. Therefore, the line of sight of the partner user displayed on the screen can be shifted, and an uncomfortable and unnatural situation where the line of sight continues to match between the user and the partner user can be avoided.

  A communication control program according to a fourteenth aspect causes the computer to function as various processing means of the communication terminal device according to any one of the first to twelfth aspects. An effect can be obtained.

1 is a block diagram showing a configuration of a video conference system 1. FIG. 2 is a diagram showing a physical configuration of a terminal device 3. FIG. It is a figure which shows the imaging position of the video camera. FIG. 4 is a diagram showing one display mode on a screen 32. It is a figure explaining the method to detect a gaze direction (when the infrared light 33 is not located ahead of a user's gaze). It is a figure explaining the method to detect a gaze direction (when the infrared light 33 is located ahead of a user's gaze). It is a figure explaining the area | region which connects the other user's right eye 721, left eye 722, and nose 723 displayed on the screen 32. FIG. 3 is a block diagram showing an electrical configuration of a terminal device 3. FIG. 2 is a conceptual diagram showing various storage areas of an HDD 31. FIG. 3 is a conceptual diagram showing various storage areas of a RAM 22. FIG. It is a flowchart of the gaze direction detection process by CPU20. FIG. 4 is a diagram showing one display mode on a screen 32. It is a flowchart of the imaging position change process by CPU20. It is a figure which shows the blink detection area 70 (a state where the user's eyes 77 are opened). It is a figure which shows the blink detection area 70 (a state where the user's eyes 77 are closed). It is a block diagram which shows the electric constitution of the terminal device 130 of 2nd embodiment. 3 is a conceptual diagram illustrating various storage areas of an HDD 131. FIG. 3 is a conceptual diagram showing various storage areas of a RAM 122. FIG. It is a flowchart of the gaze direction detection process by CPU120. It is a flowchart of the audio | voice detection process by CPU120. It is a flowchart of the blink count process by CPU120.

  Hereinafter, the terminal device 3 which is 1st embodiment of this invention is demonstrated with reference to drawings. First, the configuration of the video conference system 1 including the terminal devices 3 and 4 as constituent elements will be described with reference to FIG.

  The video conference system 1 includes terminal devices 3 and 4 connected to each other via a network 2. The terminal devices 3 and 4 are provided at different bases. In this video conference system 1, a remote conference between users at different bases is performed by transmitting and receiving images and sounds between the terminal devices 3 and 4 via the network 2. In the present embodiment, the base where the terminal device 3 is provided will be described as its own base, and the base where the terminal device 4 is provided will be described as another base.

  In the present embodiment, during the remote conference, the user of the terminal device 3 matches the line of sight with the user of the terminal device 4 (hereinafter referred to as a partner user) displayed on the screen 32 (see FIG. 2). The characteristic is that the line of sight can be removed by changing the position of the video camera 34 (see FIG. 2) for photographing the other user in the terminal device 4 when the predetermined time is longer than the predetermined time.

  First, the physical configuration of the terminal device 3 will be described with reference to FIG. 2 and FIG. Note that the right side in FIG. 2 is described as the front side of the terminal device 3 facing the user, and the left side is described as the back side of the terminal device 3. The terminal device 3 includes a projector 28 that projects an image, a screen 32 that displays the projected image, and a video camera 34 that is disposed behind the screen 32 and photographs a user located in front of the terminal device 3. . A chair (not shown) whose position is fixed with respect to the screen 32 is provided in front of the terminal device 3. When the user sits on the chair, a straight line connecting the user and the video camera 34 intersects the surface of the screen 32 perpendicularly, and the distance between the user and the screen 32 is a predetermined distance L (see FIG. 5). .

  The projector 28 will be described. As shown in FIG. 2, the projector 28 is disposed on the upper rear side of the screen 32 with respect to the user. A video controller 23 is connected to the projector 28, and the video transmitted from the terminal device 4 is projected obliquely toward the screen 32. By arranging the projector 28 obliquely above the screen 32, the video camera 34 can be arranged behind the screen 32. Note that distortion correction may be performed so that an image projected obliquely with respect to the screen 32 becomes a normal display with no distortion on the screen 32. As a distortion correction method, for example, a distortion correction method described in Japanese Patent Application Laid-Open No. 6-133111 is applicable.

  The distortion correction method will be briefly described with reference to an example in which a square image is displayed on the screen 32. The image projected from the projector 28 increases in magnification as the distance from the projector 28 increases. For this reason, when an image is projected obliquely, the image enlargement ratio is larger at a position far from the projector 28 than at a close position, and the image is distorted. For example, when a square image is projected from the projector 28, it becomes a trapezoid on the screen 32. Therefore, the image projected from the projector 28 is processed into an inverted trapezoid in advance. By projecting an inverted trapezoidal image from the projector 28, the image is displayed in a square shape on the screen 32.

  Next, the screen 32 will be described. As the screen 32, a well-known visual dependent scattering plate is used in which transmission and scattering of incident light depend on an incident angle of incident light. Specifically, the screen 32 transmits incident light from the video camera 34 and scatters incident light from the projector 28. Accordingly, the video camera 34 disposed behind the screen 32 can photograph the user via the screen 32. Further, the user can view the video projected from the projector 28 via the screen 32.

  Next, the video camera 34 will be described. As shown in FIG. 3, a camera support member 342 that supports the video camera 34 is disposed behind the screen 32. The camera support member 342 includes a rail portion 341 formed in an arc shape around a chair on which a user sits, and the video camera 34 is supported so as to be movable along the rail portion 341. A camera moving device 36 (see FIG. 8) is attached to the camera support member 342. The camera moving device 36 includes a motor (not shown) and a transmission mechanism (not shown) connected to the video camera 34. When the motor is driven, the position of the video camera 34 is moved via the transmission mechanism.

  The user's image photographed by the video camera 34 is transmitted to the terminal device 4 arranged at another base and used for detecting the user's line-of-sight direction. In a normal state, the video camera 34 is arranged perpendicular to the screen 32 and photographs the user from the front. In the present embodiment, when the time for which the user matches the line of sight with the counterpart user displayed on the screen 32 is too long, the video camera 34 moves along an arc along the rail portion 341. When the shooting position of the video camera 34 deviates from the front of the user, the line of sight of the partner user can be shifted.

  As shown in FIG. 2, an infrared light 33 that emits infrared light is attached to the lower part of the video camera 34. Infrared light emitted by the infrared light 33 is reflected on the eyeball cornea reflection surface of the user to form a virtual image (Purkinje image) 71 (see FIG. 5). The video camera 34 photographs the Purkinje image 71 and the center of the pupil 75 of the eyeball 72, and detects the user's line-of-sight direction from the relative position. A method for detecting the line-of-sight direction will be described later.

  Next, the screen displayed on the screen 32 of the terminal device 3 will be described with reference to FIG. On the screen 32 of the terminal device 3, a partner user screen 281 on which an image of the partner user who uses the terminal device 4 is displayed, and a shared material screen 282 on which the shared material in the terminal devices 3 and 4 is displayed. The shared material screen 282 is displayed and set, for example, when the material stored in the terminal device 3 on the local site side is shown and explained to the other user at the other site. When the shared material display program is started and a desired material is designated as a shared material, the shared material screen 282 is displayed on the screen 32 of the terminal devices 3 and 4, and the shared material is displayed in the frame.

  For example, the partner user screen 281 is arranged in the substantially left half of the screen 32, and the shared material screen 282 is arranged in the lower right quarter of the screen 32. The display mode is not limited to this, and the arrangement and size of the partner user screen 281 and the shared material screen 282 can be freely changed by the user.

  Next, a method for detecting the line-of-sight coincidence will be described with reference to FIGS. Here, the “line of sight matching state” refers to a state in which the user's line of sight is in a region (hereinafter referred to as region M) connecting the other user's eyes and nose displayed on the screen 32. In this embodiment, first, the user's line-of-sight direction is detected. Next, the line-of-sight position on the screen 32 ahead of the detected line-of-sight direction is specified. Then, it is determined whether or not the specified line-of-sight position is in the area M. If it is in the area M, the line-of-sight state is set. Hereinafter, a method for detecting the line-of-sight matching state will be specifically described.

  First, the gaze direction detection method will be described. The detection of the line-of-sight direction is performed based on a photographed image photographed by the video camera 34. A known method is used as a method for detecting the user's line-of-sight direction. For example, the line-of-sight direction detection method described in Japanese Patent Application Laid-Open No. 10-108843 is applicable.

  A gaze direction detection method to which the above detection method is applied will be described. Here, for convenience of explanation, the video camera 34 and the infrared light 33 will be described as being approximated at the same coordinates.

  As shown in FIG. 5, when the infrared light 33 irradiates the user's eyeball 72 with infrared light, the irradiated infrared light is reflected on the cornea reflection surface of the user's eyeball 72, and a virtual image (Purkinje image) 71 of the infrared light 33 is formed. Arise. As shown in FIG. 6, when the infrared light 33 is positioned ahead of the user's line of sight, the generation position of the Purkinje image 71 coincides with the pupil center 76. Then, as the tip of the user's line of sight moves away from the infrared light 33, the interval between the Purkinje image 71 and the pupil center 76 increases (see FIG. 5).

  Specifically, if the rotation angle of the eyeball 72 from the state in which the Purkinje image 71 and the pupil center 76 coincide is θ, and the interval S between the Purkinje image 71 and the pupil center 76 is S, the interval S is the sine of the rotation angle θ. It spreads almost in proportion to Therefore, by calculating the position of the Purkinje image 71, the position of the pupil center 76, and the interval between them in the captured image captured by the video camera 34, the rotation angle θ of the eyeball 72 and the user's line-of-sight direction can be known. Can do. The detection of the Purkinje image 71 and the pupil 75 in the captured image is performed by a known method for detecting an edge value indicating a density change in each pixel.

Next, a method for specifying the user's line-of-sight position T on the screen 32 ahead of the detected line-of-sight direction will be described. In the present embodiment, a straight line connecting the center of the eyeball 72 of the user sitting facing the screen 32 and the video camera 34 intersects the surface of the screen 32 perpendicularly during the meeting. Further, the distance between the center of the eyeball 72 and the screen 32 is a predetermined distance L. Assuming that the point where the straight line connecting the center of the eyeball 72 and the video camera 34 intersects the display surface of the screen 32 is the center point O and the rotation angle of the eyeball is θ, the line-of-sight position T on the display surface ahead of the user's line-of-sight direction. A distance K between the center point O and the center point O is expressed by the following equation.
K = L · tanθ
From the above formula, the line-of-sight position T on the user's screen 32 can be specified.

  Next, referring to FIG. 6, whether or not the identified line-of-sight position T is in a region (region M) connecting the right eye 721, left eye 722, and nose 723 of the other user displayed on the screen 32. An explanation of the determination method will be given. First, the other user's right eye 721, left eye 722, and nose 723 displayed on the screen 32 are extracted by a known method. As an extraction method, for example, the eye and nose detection method described in JP-A-2008-234208 is applicable.

  This will be specifically described. First, the face of the other user displayed on the screen 32 is detected. The face is detected by calculating the edge of the face outline or extracting the face outline by pattern matching of the face outline and detecting the face within the range of the face outline. Then, two adjacent nostrils are detected in the approximate center of the detected face. In the nostril portion, light from the outside is difficult to be irradiated, so the image is taken darker than other portions. Therefore, the density in each pixel changes near the boundary of the nostril portion. Therefore, the nostril portion can be detected by detecting the edge value indicating the obtained density change. Then, the position of the midpoint between two adjacent nostrils is specified as the position of the nose 723.

  Then, the pupil portion is searched for above the detected position of the nose 723. In general, the pupil part has a lower reflectance than the skin part and the white-eye part, so that the pupil part is photographed dark. Therefore, the density in each pixel changes near the pupil. Accordingly, the pupil portion can be detected by detecting the edge value indicating the obtained density change. Then, the center position of the detected pupil is specified as the positions of the right eye 721 and the left eye 722.

  Then, it is determined whether or not there is a line-of-sight position T in the area connecting the specified right eye 721, left eye 722, and nose 723 (hereinafter referred to as area M). State.

  Next, the electrical configuration of the terminal device 3 will be described with reference to FIG. FIG. 8 is a block diagram showing an electrical configuration of the terminal device 3. Since the terminal devices 3 and 4 have the same configuration, only the configuration of the terminal device 3 will be described here, and the description of the terminal device 4 will be omitted.

  The terminal device 3 is provided with a CPU 20 as a controller that controls the terminal device 3. Connected to the CPU 20 are a ROM 21 that stores BIOS, a RAM 22 that temporarily stores various data, and an I / O interface 30 that mediates data transfer. The I / O interface 30 is connected to a hard disk drive 31 (hereinafter referred to as HDD 31) having various storage areas.

  The I / O interface 30 includes a communication device 25 for communicating with the network 2, a mouse 27, a video controller 23, a key controller 24, the above-described video camera 34 for photographing a user, and a user's A microphone 35 for capturing audio, a CD-ROM drive 26, a timing device 41, a drive circuit 331, and a drive circuit 332 are connected to each other. The projector 28 described above is connected to the video controller 23. A keyboard 29 is connected to the key controller 24. The infrared light 33 described above is connected to the drive circuit 331. The camera moving device 36 described above is connected to the drive circuit 332.

  The CD-ROM 114 inserted into the CD-ROM drive 26 stores the main program of the terminal device 3, the communication control program of the present invention, and the like. When the CD-ROM 114 is introduced, these various programs are set up from the CD-ROM 114 to the HDD 31 and stored in a program storage area 313 (see FIG. 9) described later.

  Next, various storage areas of the HDD 31 will be described with reference to FIG. The HDD 31 has a captured image data storage area 311 for storing captured images captured by the video camera 34 and display screen data for storing screen data displayed on the screen 32 (see FIGS. 2 and 4) of the terminal device 3. A storage area 312, a program storage area 313 for storing various programs, a predetermined time storage area 314 for storing a predetermined time required for executing the program, and a position for storing a positional relationship between the user, the screen 32, and the infrared light 33. At least a relation storage area 315, a predetermined value storage area 316 for storing a predetermined value necessary for executing the program, and another information storage area 317 are provided.

  The display screen data storage area 312 includes at least a partner user image storage area (not shown) and a shared material data storage area (not shown). In the partner user image storage area, image data of the partner user transmitted from the terminal device 4 is stored. In the shared material data storage area, shared material data designated by the user or the other user is stored. When shared material data is stored in the shared material data storage area, a shared material screen 282 is displayed on the screen 32. In the positional relationship storage area 315, a predetermined distance L between the user's eyeball 72 and the screen 32 is stored.

  The program storage area 313 stores a main program of the terminal device 3, a communication control program for executing a remote conference with the terminal device 4, and the like. In the other information storage area 317, other information used in the terminal device 3 is stored. When the terminal device 3 is a dedicated machine that does not include the HDD 31, various programs are stored in the ROM 21.

  Next, various storage areas of the RAM 22 will be described with reference to FIG. The RAM 22 includes at least a line-of-sight storage area 221, a line-of-sight storage area 222, an area M storage area 223, a shared material screen coordinate storage area 224, a connection terminal storage area 225, and a measurement time storage area 229. ing. In the line-of-sight storage area 221, the user's line-of-sight direction detected based on the captured image stored in the captured image data storage area 311 is stored as the rotation angle of the eyeball 72. In the line-of-sight position storage area 222, the line-of-sight position of the user on the screen 32 is stored as position coordinates in a coordinate system with the lower right as the origin from the user of the screen 32. In the area M storage area 223, the area M specified based on the display image stored in the partner user image storage area (not shown) is stored as position coordinates on the screen 32. The shared material screen coordinate storage area 224 stores the position coordinates of the shared material screen 282. The shared material screen coordinate storage area 224 includes a reference size storage area (not shown) and an enlarged size storage area (not shown). In the connection terminal storage area 225, the terminal ID of the connection terminal currently connected via the network 2 is stored. The measurement time storage area 229 stores the time measured by the timing device.

  Next, communication control processing in consideration of line-of-sight matching in the video conference system 1 will be described. In this description, it is assumed that the terminal device 3 on the local site side and the terminal device 4 on the other site side hold a conference. In the terminal devices 3 and 4, both the “line-of-sight detection process” for detecting the line-of-sight direction and monitoring the line-of-sight matching state and the “shooting position changing process” for receiving the line-of-sight match and changing the shooting position of the video camera. Done. Therefore, for convenience of explanation, a case will be described as an example where the gaze direction detection process is executed in the terminal device 3 on the local site side and the shooting position change process is executed on the terminal device 4 on the other site side.

  First, the line-of-sight direction detection process executed by the CPU 20 of the terminal device 3 on the local site side will be described with reference to the flowchart of FIG. When the terminal device 3 and the terminal device 4 are respectively connected to the network and start communication with each other, the gaze direction detection process shown in FIG. 11 is started. When the line-of-sight direction detection process is started, first, the value of the line-of-sight matching time t1 and the value of the post-transmission elapsed time t2 stored in the measurement time storage area 229 are initialized (S1).

  Next, the drive circuit 331 is driven, and the infrared light 33 is emitted. The user's image captured by the video camera 34 is stored in the captured image data storage area 311. The Purkinje image 71 and the pupil center 76 are detected from the captured image stored in the captured image data storage area 311, and the distance between the Purkinje image 71 and the pupil center 76 is calculated. Then, the rotation angle θ of the eyeball is calculated from the distance between the Purkinje image 71 and the pupil center 76, and the user's line-of-sight direction is detected (S2). The detected gaze direction is stored in the gaze direction storage area 221 of the RAM 22.

  Subsequently, the line-of-sight position T of the user on the display image on the screen 32 is specified from the line-of-sight direction stored in the line-of-sight direction storage area 221 and the predetermined distance L stored in the positional relationship storage area 315 (S3). The identified line-of-sight position T is stored in the line-of-sight position storage area 222 of the RAM 22.

  Next, the face of the partner user is detected from the display image transmitted from the terminal device 4 and stored in the partner user image storage area (not shown) of the display screen data storage area 312. And the other user's right eye 721, left eye 722, and nose 723 are detected from the detected face. Then, based on the detected position coordinates of the right eye 721, the left eye 722, and the nose 723, a region M connecting the eyes and the nose is specified on the display screen of the screen 32 (S4). The identified area M is stored in the area M storage area 223.

  Then, it is determined whether or not the line-of-sight position T stored in the line-of-sight position storage area 222 is in the area M stored in the area M storage area 223 (S5). When the line-of-sight position T is in the region M (S5: YES), it is determined that the line-of-sight coincidence state is established, and it is determined whether the line-of-sight coincidence time t1 is measured by the timing device 41 (S6). When the measurement of the line-of-sight matching time t1 has not been started and the line-of-sight matching time t1 is not being measured (S6: NO), the measurement of the line-of-sight matching time t1 is started (S7). The measured line-of-sight matching time t1 is stored in the line-of-sight matching time storage area (not shown) of the measurement time storage area 229. When the measurement of the line-of-sight coincidence time t1 is started, the processes of S2 to S5 are performed again. When the line-of-sight matching time t1 is being measured (S6: YES), it is determined whether or not the measured line-of-sight matching time t1 is equal to or longer than the first predetermined time T1 stored in the predetermined time storage area 314 ( S8). In the present embodiment, the first predetermined time T1 is 5 seconds.

  When the line-of-sight matching time t1 is not equal to or longer than the first predetermined time T1 (S8: NO), the elapsed time since the line-of-sight matching state is reached is less than 5 seconds. In this case, it is assumed that the user has naturally matched his / her line of sight to the other user displayed on the screen 32, and until the line-of-sight matching time t1 is equal to or longer than the first predetermined time T1, The process is repeated.

  On the other hand, when the line-of-sight matching time t1 is equal to or longer than the first predetermined time T1 (S8: YES), the line-of-sight matching state continues for 5 seconds or more. In this case, a first signal for notifying that the line-of-sight matching state has continued unnaturally is transmitted to the terminal device 4 used by the other user (S9). As will be described later, the first signal is transmitted to the terminal device 4 in order to change the shooting position of the video camera 34 in the terminal device 4.

  When the first signal is transmitted (S9), measurement of a post-transmission elapsed time t2, which is an elapsed time from the transmission of the first signal, is started (S10). The measured post-transmission elapsed time t2 is stored in the post-transmission elapsed time storage area (not shown) of the measurement time storage area 229. Subsequently, the shared material data storage area of the display screen data storage area 312 is referred to, and it is determined whether or not the shared material screen 282 is displayed on the screen 32 (S11). If shared material data is not stored in the shared material data storage area, the shared material screen 282 is not displayed on the screen 32 (S11: NO). In this case, the process proceeds to S13.

  On the other hand, when shared material data is stored in the shared material data storage area, the shared material screen 282 is displayed on the screen 32 (S11: YES). In this case, as shown in FIG. 12, the shared material screen 282 is enlarged so as to cover the partner user screen 281 (S12).

  Here, the horizontal length of the display area of the screen 32 is a, the vertical length is b, and the horizontal length of the shared material screen 282 is (a × (1/3)), the vertical length. This will be described in detail by taking the case of (b × (1/4)) as an example. It is assumed that the lower right of the shared material screen 282 matches the lower right of the display area of the screen 32.

  When the shared material screen 282 is displayed on the screen 32 (S11: YES), the reference size storage area (not shown) of the shared material screen coordinate storage area 224 is referred to, and the display area of the shared material screen 282 is acquired. The Further, the ratio of the size of the display area of the screen 32 to the acquired display area of the shared material screen 282 is calculated for each of the horizontal direction and the vertical direction. In the above example, the ratio of the size of the screen 32 to the shared material screen 282 is 3 in the horizontal direction and about 4 in the vertical direction.

  Next, the shared material screen 282 is enlarged with the lower right of itself as the origin (S12), with a smaller ratio of the two calculated ratios as an enlargement ratio. In the above-described example, the shared material screen 282 is enlarged three times vertically and three times horizontally with the lower right of itself as the origin, so the horizontal length of the shared material screen 282 is a (= (a × ( 1/3)) × 3), the length in the vertical direction is (b × (3/4)) (= (b × (1/4)) × 3). Then, the coordinates on the upper right, lower right, and lower left screens 32 of the enlarged shared material screen 282 are the enlarged size storage areas (not shown) of the shared material screen coordinate storage area 224 as data relating to the size of the shared material screen 282. Is remembered. In the above example, the lower left coordinate is (a, 0) and the upper right coordinate is (0, (b × (3/4)) where the left and right direction is the x direction, the vertical direction is the y direction, and the lower right is the origin. )), And the upper left coordinates are stored in the enlarged size storage area (not shown) as (a, (b × (3/4))).

  Then, with reference to the enlarged size storage area and the area M storage area 223, it is determined whether or not the area M is included in the display area of the shared material screen 282. When the area M is included in the display area of the shared material screen 282, the process of S12 is terminated and the process proceeds to S13. When the area M is not included in the display area of the shared material screen 282, the shared material screen 282 is moved so that the area M is included in the display area of the shared material screen 282. Then, the process proceeds to S13.

  The shared material screen 282 is arranged on the upper side of the partner user screen 281, but a portion overlapping the partner user screen 281 is transparently displayed. In other words, the partner user screen 281 arranged on the lower side of the shared material screen 282 is visible to the user of the terminal device 3.

  When it is determined that the shared material screen 282 is not displayed on the screen 32 (S11: NO), or when the shared material screen 282 is enlarged (S12), the measured post-transmission elapsed time t2 is predetermined. It is determined whether or not it is equal to or longer than the second predetermined time T2 stored in the time storage area 314 (S13). In the present embodiment, the second predetermined time T2 is 10 seconds.

  When the post-transmission elapsed time t2 is not equal to or longer than the second predetermined time T2 (S13: NO), the elapsed time since the shooting position of the video camera 34 is changed in the terminal device 4 is less than 10 seconds. In this case, the process of S13 is repeated until the post-transmission elapsed time t2 is equal to or longer than the second predetermined time T2, assuming that a sufficient time has not yet elapsed since the unnatural line-of-sight matching state was avoided. .

  On the other hand, when the post-transmission elapsed time t2 is equal to or longer than the second predetermined time T2 (S13: YES), a sufficient time has elapsed since the shooting position of the video camera 34 in the terminal device 4 is changed. In this case, a second signal indicating that an unnatural line-of-sight matching state is avoided is transmitted to the terminal device 4 used by the partner user (S14). In S14, when the shared material screen 282 is enlarged and displayed on the screen 32 (S11: YES), the second signal is transmitted and the reference size storage area of the shared material screen coordinate storage area 224 is referred to. Thus, the size of the shared material screen 282 is returned to the size before enlargement.

  Then, it is determined whether or not the conference is over (S15). Specifically, it is determined whether or not there are one or more terminals connected to the network 2 other than itself. When the terminal devices 3 and 4 are connected to the network 2, a connection signal indicating connection to the network 2 is transmitted to the partner terminal. When a connection signal is received from another terminal, the terminal ID of the terminal that transmitted the connection signal is stored in the connection terminal storage area 225 of the RAM 22. On the other hand, when the connection to the network 2 is disconnected, a disconnection signal indicating that the connection to the network is disconnected is transmitted to the partner terminal.

  When receiving a disconnection signal from another terminal, the terminal ID of the terminal that has transmitted the disconnection signal is deleted from the connected terminal storage area 225 of the RAM 22. If the connected terminal storage area 225 is referred to and the terminal connected to the network 2 is determined to be 0 in addition to itself (S15: YES), the process ends. On the other hand, when there are one or more terminals connected to the network 2 (S15: NO), the process returns to S1 and the processes of S1 to S15 are repeated.

  Next, the photographing position changing process executed by the CPU 20 of the terminal device 4 on the other base side will be described with reference to the flowchart of FIG. When the terminal device 3 and the terminal device 4 are connected to the network and start communication with each other, the photographing position changing process shown in FIG. 13 is started. When the photographing position changing process is started, it is determined whether or not the first signal transmitted from the terminal device 3 has been received (S51).

  If the first signal has not been received (S51: NO), it is determined whether or not the conference is over (S55). When the first signal is received (S51: YES), it is assumed that the user of the terminal device 3 and the user of the terminal device 4 are in an unnatural line-of-sight match state, and the camera is moved by the drive circuit 332 (see FIG. 8). A motor (not shown) included in the device 36 is driven. Then, the position of the video camera 34 moves according to the driving amount of the motor. Specifically, as shown in FIG. 3, the video camera 34 located on the extension line of the user's line of sight moves along the rail portion 341 on an arc around the user. In this way, the shooting position is changed from the position facing the user's line-of-sight direction to the retracted position (S52).

  It is determined whether or not the second signal transmitted from the terminal device 3 has been received (S53). When the second signal is not received (S53: NO), the process of S53 is repeated until the second signal is received. When the second signal is received (S53: YES), it is assumed that an unnatural line-of-sight matching state between the user of the terminal device 3 and the user of the terminal device 4 can be avoided. A motor (not shown) for changing the position of 34 is driven. Then, the video camera 34 that is located off the extension line of the user's line of sight moves to a position facing the user on the arc with the user at the center while being supported by the camera support member 342. Thus, the photographing position is returned to the original position (S54).

  Then, the connected terminal storage area 225 is referred to, and it is determined whether or not there are one or more terminals connected to the network 2 other than itself (S55). If the terminal connected to the network 2 is 0 in addition to itself (S55: YES), the process is terminated. On the other hand, when there are one or more terminals connected to the network 2 (S55: NO), the process returns to S51, and the processes of S51 to S55 are repeated.

  In the above description, the video camera 34 shown in FIGS. 2 and 8 corresponds to the “user photographing unit” of the present invention. The infrared light 33 corresponds to the “irradiation means” of the present invention. The projector 28 and the CPU 20 that controls the projector 28 correspond to the “display unit” of the present invention. The positional relationship storage area 315 of the HDD 31 shown in FIG. 9 corresponds to the “positional relationship storage means” of the present invention. The CPU 20 that executes the process of S2 shown in FIG. 11 corresponds to the “line-of-sight direction detecting means” of the present invention. The CPU 20 that executes the process of S3 shown in FIG. 11 corresponds to the “line-of-sight position specifying unit” of the present invention. The CPU 20 that detects the face of the partner user in S4 shown in FIG. 11 corresponds to the “face detection means” of the present invention. The CPU 20 that executes the process of S5 shown in FIG. The CPU 20 that executes the process of S7 shown in FIG. The CPU 20 that executes the process of S9 shown in FIG. 11 corresponds to the “first signal transmitting means” of the present invention. The CPU 20 that executes the processing of S10 shown in FIG. 11 corresponds to “post-transmission elapsed time measuring means” of the present invention. The CPU 20 that executes the process of S11 shown in FIG. 11 corresponds to “another image display determination unit” of the present invention. The CPU 20 that executes the process of S13 shown in FIG. 11 corresponds to the “second predetermined time determination means” of the present invention. The CPU 20 that executes the process of S14 shown in FIG. 11 corresponds to the “second signal transmitting means” of the present invention. The CPU 20 that executes the process of S51 shown in FIG. 13 corresponds to the “first signal receiving means” of the present invention. The CPU 20 that executes the process of S52 shown in FIG. 13 corresponds to the “photographing position changing means” of the present invention. The CPU 20 that executes the process of S53 shown in FIG. 13 corresponds to the “second signal receiving means” of the present invention. The CPU 20 that executes the process of S54 shown in FIG. 13 corresponds to the “photographing position returning means” of the present invention.

  As described above, the terminal device 3 according to the first embodiment is connected to other terminal devices 4 via the network 2. A video conference system 1 that implements a remote conference by transmitting and receiving images and sounds between these terminal devices is configured. In the video conference system 1, during the remote conference, the user of the terminal device 3 (or 4) monitors whether or not the line of sight is unnaturally aligned with the counterpart user displayed on the screen 32. When an unnatural line-of-sight matching state is detected, a signal (first signal) indicating that the unnatural line-of-sight matching state is detected is transmitted to the terminal device 4 (or 3) of the other user. . When receiving the first signal, the terminal device 4 (or 3) of the partner user changes the position of the video camera 34 that captures the partner user. Thereby, an unnatural line-of-sight matching state between conference participants can be avoided, and the conference participants can talk smoothly.

  Next, the terminal device 130 which is 2nd embodiment is demonstrated with reference to drawings. In the terminal device 130, when the user and the partner user are in an unnatural line-of-sight matching state, the position of the video camera 34 that captures the partner user is changed to avoid the line-of-sight matching state. Also, the number of blinks of the user is counted, and when the blink is frequently performed, the avoidance time of the line-of-sight matching state is lengthened assuming that the user is in a tension state. Further, when both the user and the other user are in a silent state, it is assumed that there is an awkward atmosphere, and the time for avoiding the line-of-sight matching state is lengthened. Therefore, in order to focus on these different points, a description will be given focusing on communication control processing by the CPU 120 different from the first embodiment. In addition, the terminal device 130 of 2nd embodiment comprises the video conference system 1 shown in FIG. 1 similarly to the terminal device 3 of 1st embodiment.

  First, a blink detection method will be described. As a method for detecting blinking, a known method is used. For example, a blink detection method described in Japanese Patent Application Laid-Open No. 2008-226106 is applicable.

  A blink detection method that uses the above detection method appropriately will be described with reference to FIGS. 14 and 15. First, the user's face is detected from the user's captured image stored in the captured image data storage area 311. The face is detected by calculating the edge of the face outline or extracting the face outline by pattern matching of the face outline.

  Next, an area for detecting blink (blink detection area 70) is extracted from the detected face area. In order to extract the blink detection area 70, first, two adjacent nostrils are detected in the approximate center of the detected face. Then, a horizontally-long rectangular blink detection region 70 is extracted above the detected nostril position.

  In the extracted blink detection area 70, the open / closed state of the eye 77 is determined. In order to determine the open / closed state of the eye 77, first, the eye 77 is detected in the blink detection region 70, and the area of the detected eye 77 is acquired. Next, it is determined whether or not the ratio of the detected area of the eye 77 to the face area is greater than or equal to a threshold value stored in a threshold value storage area 318 of the HDD 131 described later. The threshold value is the ratio (maximum value) of the area of the eye 77 to the face when the user is most open, and the ratio (minimum value) of the area of the eye 77 to the face when the user is pinching the eye 77. And the average value. If it is equal to or greater than the threshold value, it is determined that the eye 77 is open, and if it is equal to or less than the threshold value, it is determined that the eye 77 is closed.

  Determination of the open / closed state of the eye 77 is performed every 1/30 seconds. When the area of the eye 77 changes from the threshold value to less than the threshold value, it is detected that blinking has occurred.

  Next, the electrical configuration of the terminal device 130 will be described with reference to FIG. The terminal device 130 is provided with a CPU 120 as a controller that controls the terminal device 130. Connected to the CPU 120 are a ROM 121 that stores BIOS, a RAM 122 that temporarily stores various data, and an I / O interface 30 that mediates data transfer. Connected to the I / O interface 30 are a hard disk drive 131 (hereinafter referred to as HDD 131) having various storage areas and a card reader controller 38. A card reader 39 for reading a user ID for identifying a user stored in an identification card (not shown) owned by each user is connected to the card reader control unit 38.

  As shown in FIG. 17, the HDD 131 includes a threshold storage area 318, a predetermined value storage area 316, a user ID storage area 320, in addition to various storage areas similar to the HDD 31 of the first embodiment (see FIG. 9). Is provided. The user ID storage area 320 stores a user ID stored in an identification card (not shown) read by the card reader 39 at the start of the conference. The threshold storage area 318 is provided with a threshold table (not shown). In the threshold value table, an average value (threshold value) of the eye area with respect to the face area is stored for each user.

  In the RAM 122, when an eye image storage area 226 for storing an image of the eye 77 in the blink detection area 70, a blink number storage area 227 for storing the counted number of blinks, and when the user or the other user makes a sound, A voice presence / absence storage area 228 for storing the presence of voice. The other electrical configuration of the terminal device 130 is the same as that of the terminal device 3 (see FIG. 8) of the first embodiment.

  Next, communication control processing by the CPU 120 will be described with reference to the flowchart of FIG. Also in the present embodiment, in the terminal devices 3 and 4, “line-of-sight detection processing” that detects the line-of-sight direction and monitors the line-of-sight state, and “shooting position change” that changes the shooting position of the video camera in response to line-of-sight matching Both “processing” and “processing” are performed. The “photographing position change process” is the same as that in the first embodiment, and thus description thereof is omitted.

  When the terminal device 3 and the terminal device 4 are respectively connected to the network and start communication with each other, the gaze direction detection process shown in FIG. 19 is started. When the line-of-sight direction detection process is started, first, the values of the line-of-sight matching time t1 and the post-transmission elapsed time t2 stored in the measurement time storage area 229 are initialized. Further, the value of the blink count Y stored in the blink count storage area 227 and the value stored in the voice presence / absence storage area 228 are initialized (S21).

  Next, the user's line-of-sight direction is detected by the above-described method (S22). The detected line-of-sight direction is stored in the line-of-sight direction storage area 221 of the RAM 122. Subsequently, the user's line-of-sight position T on the display image on the screen 32 is specified from the line-of-sight direction stored in the line-of-sight storage area 221 and the predetermined distance L stored in the positional relationship storage area 315 (S23). The identified line-of-sight position T is stored in the line-of-sight position storage area 222 of the RAM 122.

  Next, the region M connecting the right eye 721, the left eye 722, and the nose 723 on the screen 32 of the other user is specified by the above-described method (S24). The identified area M is stored in the area M storage area 223.

  Then, it is determined whether or not the line-of-sight position T stored in the line-of-sight position storage area 222 is in the area M stored in the area M storage area 223 (S25). When the line-of-sight position T is in the region M (S25: YES), it is determined that the line-of-sight coincidence state is present, and it is determined whether the line-of-sight coincidence time t1 is being measured (S26). When the line-of-sight matching time t1 is not being measured (S26: NO), the measurement of the line-of-sight matching time t1 is started (S27).

  When measurement of the line-of-sight coincidence time t1 is started (S27), a sound detection process for detecting sound is started (S28). The voice detection process will be described with reference to FIG. The voice detection process is a process activated in the gaze direction detection process shown in FIG. 19 and is performed in parallel with the gaze direction detection process.

  In the voice detection process, first, it is determined whether the user's voice is input from the microphone 35 or the voice data of the other user is transmitted from the terminal device 4 (S101). When the user's voice is input from the microphone 35 or the voice data of the other user is transmitted from the terminal device 4 (S101: YES), it is stored in the voice presence / absence storage area 228. (S102). Specifically, “1” is stored in the voice presence / absence storage area 228. Then, the process returns to S101.

  When the user's voice is not input from the microphone 35 or the voice data of the other user is not transmitted from the terminal device 4 (S101: NO), the stored content of the voice presence / absence storage area 228 is not changed, and the process of S101 Return to. The initial value of the voice presence / absence storage area 228 is “0”. The voice detection process ends at the same time as the line-of-sight direction detection process ends.

  In the line-of-sight direction detection process shown in FIG. 19, when the voice detection process is activated (S28), next, the blink count process for counting the number of blinks is activated (S29). The blink count process will be described with reference to FIG. The blink count process is a process activated in the gaze direction detection process shown in FIG. 19, and is performed in parallel with the gaze direction detection process.

  In the blink count processing, first, the user ID storage area 320 and the threshold storage area 318 are referred to, and the average value (threshold) of the ratio of the eye area to the face area of the user using the terminal device 130 is specified. (S131). Then, the photographed image data storage area 311 is referred to, and the presence or absence of blinking is detected by the above-described method (S132). Then, it is determined whether or not there has been a blink (S133). If it is determined that there is a blink (S133: YES), the value of the blink count Y stored in the blink count storage area 227 is incremented by 1 (S134), and the process returns to S131. If it is determined that there is no blink (S133: NO), the process returns to S131 without performing the process of S134. Note that the blink count processing ends at the same time when the gaze direction detection processing ends.

  In the gaze direction detection process shown in FIG. 19, when the blink count processing is started (S29), the processes of S22 to S25 are performed again.

  When the line-of-sight matching time t1 is being measured (S26: YES), it is determined whether or not the measured line-of-sight matching time t1 is equal to or longer than the first predetermined time T1 stored in the predetermined time storage area 314 ( S30). In the present embodiment, the first predetermined time T1 is 5 seconds.

  When the line-of-sight coincidence time t1 is not equal to or longer than the first predetermined time T1 (S30: NO), the elapsed time from the line-of-sight coincidence state is less than 5 seconds. In this case, it is assumed that the user is in a state where the line of sight is naturally matched with the other user displayed on the screen 32, and until the line-of-sight matching time t1 becomes equal to or longer than the first predetermined time T1, the processes of S22 to S30 are performed. The process is repeated.

  On the other hand, when the line-of-sight coincidence time t1 is equal to or longer than the first predetermined time T1 (S30: YES), the line-of-sight coincidence state continues for 5 seconds or more. In this case, a first signal for notifying that the line-of-sight matching state has continued unnaturally is transmitted to the terminal device 4 used by the other user (S31). Similarly to the first embodiment, the first signal is transmitted to the terminal device 4 in order to change the shooting position of the video camera 34 in the terminal device 4.

  When the first signal is transmitted (S31), measurement of a post-transmission elapsed time t2, which is an elapsed time from the transmission of the first signal, is started (S32). Subsequently, the shared material data storage area of the display screen data storage area 312 is referred to, and it is determined whether or not the shared material screen 282 is displayed on the screen 32 (S33). When the screen data of the shared material is not stored in the shared material data storage area, the shared material screen 282 is not displayed on the screen 32 (S33: NO). In this case, the process proceeds to S35.

  On the other hand, when the screen data of the shared material is stored in the shared material data storage area, the shared material screen 282 is displayed on the screen 32 (S33: YES). In this case, the shared material screen 282 is enlarged so as to cover the partner user screen 281 (S34). Since the enlargement method is the same as that of the first embodiment, the description thereof is omitted.

  Then, it is determined whether or not the measured line-of-sight matching time t1 is equal to or longer than a third predetermined time T3 stored in the predetermined time storage area 314 (S35). In the present embodiment, the third predetermined time T3 is 7 seconds.

  When the line-of-sight matching time t1 is not equal to or longer than the third predetermined time T3 (S35: NO), the process of S35 is repeated until the line-of-sight matching time t1 becomes equal to or longer than the third predetermined time T3. When the line-of-sight matching time t1 is equal to or longer than the third predetermined time T3 (S35: YES), the user and the other party are referred until the line-of-sight matching time t1 becomes equal to or longer than the third predetermined time T3 with reference to the voice presence / absence storage area 228. It is determined whether or not the user is silent (S36).

  If “0” is stored in the voice presence / absence storage area 228, it is determined that the silent state has continued, in which neither the user nor the other user utters voice (S36: YES). There is a high possibility that the user who has remained silent and the other user were in an awkward atmosphere. Subsequently, with reference to the blink count storage area 227 and the predetermined value storage area 316, the number of blinks Y performed by the user until the line-of-sight matching time t1 becomes equal to or greater than the third predetermined time T3 is the predetermined value U. It is determined whether or not this is the case (S37). In general, the number of blinks increases when a user becomes nervous. If the number of blinks Y stored in the blink count storage area 227 is equal to or greater than the predetermined value U stored in the predetermined value storage area 316 (S37: YES), the user is likely to be in a tension state.

  In this way, when the unnatural line-of-sight matching state continues, the silent state continues between users (S36: YES), and the number of blinks Y of the user is equal to or greater than the predetermined value U (S37: YES), the user Is in a very uncomfortable situation. In this case, the value of the second predetermined time T2 stored in the predetermined time storage area 314 is changed to the predetermined time a (S38). The predetermined time a is longer than the second predetermined time T2 in the initial state. In the present embodiment, the second predetermined time in the initial state is 10 seconds, and the predetermined time a is 20 seconds.

  On the other hand, when the number Y of blinks stored in the blink number storage area 227 is less than the predetermined value U stored in the predetermined value storage area 316 (S37: NO), the user is likely not in a tension state. Thus, although the unnatural line-of-sight matching state continues and the silent state continues between users (S36; YES), when the number of blinks Y of the user is less than the predetermined value U (S37: NO), The value of the second predetermined time T2 stored in the predetermined time storage area 314 is changed to the predetermined time b (S39). The predetermined time b is longer than the second predetermined time T2 in the initial state and is shorter than the predetermined time a. In the present embodiment, the predetermined time b is 15 seconds.

  In S36, when “1” is stored in the voice presence / absence storage area 228, it is determined that the silent state has not been continued because either the user or the other user is uttering voice (S36: NO). In this case, although the line-of-sight matching state has continued for a long time between the user and the partner user, there is a high possibility that there was a conversation between the user and the partner user and there was no awkward atmosphere.

  In this case, subsequently, the blink count storage area 227 and the predetermined value storage area 316 are referred to, and the blink count Y performed by the user until the line-of-sight matching time t1 becomes equal to or greater than the third predetermined time T3. It is determined whether or not the predetermined value U is exceeded (S40). When the blink count Y stored in the blink count storage area 227 is equal to or greater than the predetermined value U stored in the predetermined value storage area 316 (S40: YES), the user is likely to be in a tension state.

  As described above, when the unnatural line-of-sight coincidence state continues and there is an utterance to any of the users (S36: NO), the number of blinks Y of the user is equal to or greater than the predetermined value U (S40: YES). The value of the second predetermined time T2 stored in the time storage area 314 is changed to the predetermined time c (S41). The predetermined time c is longer than the second predetermined time T2 in the initial state, and is shorter than the predetermined time a and the predetermined time b. In the present embodiment, the predetermined time c is 12 seconds.

  On the other hand, when the blink count Y stored in the blink count storage area 227 is less than the predetermined value U stored in the predetermined value storage area 316 (S40: NO), the user does not feel much uncomfortable. The value of the second predetermined time T2 is not changed.

  When the process of any one of S38, S39, and S41 ends, it is determined whether or not the measured post-transmission elapsed time t2 is equal to or greater than the second predetermined time T2 stored in the predetermined time storage area 314 (S42). ). As described above, the second predetermined time T2 stored in the predetermined time storage area 314 is set according to the detected degree of tension or uncomfortableness of the user. Specifically, the second predetermined time T2 is set to 20 seconds when the silent state continues between the user and the other user and the number of blinks of the user is large. If the silent state continues between the user and the other user, but the number of blinks of the user is not large, it is set to 15 seconds. If there is an utterance between the user and the other user, and the number of blinks of the user is large, it is set to 12 seconds. If there is an utterance between the user and the other user and the number of blinks of the user is not large, the initial state of 10 seconds remains.

  If the post-transmission elapsed time t2 is not equal to or longer than the second predetermined time T2 (S42: NO), the post-transmission elapsed time t2 is set to the second after assuming that sufficient time has not passed since the user's uncomfortable feeling is avoided. The process of S42 is repeated until the predetermined time T2 is reached.

  On the other hand, when the post-transmission elapsed time t2 is equal to or longer than the second predetermined time T2 (S42: YES), a sufficient time has elapsed since the shooting position of the video camera 34 in the terminal device 4 is changed. In this case, a second signal indicating that the uncomfortable feeling of the user of the terminal device 3 is avoided is transmitted to the terminal device 4 used by the partner user (S43).

  Then, with reference to the connected terminal storage area 225, it is determined whether there are one or more terminals connected to the network 2 other than itself (S44). If it is determined that the terminal connected to the network 2 is 0 in addition to itself, it is determined that the conference has ended (S44: YES), and the process ends. On the other hand, if there are one or more terminals connected to the network 2, the conference is continued (S44: NO), and the process returns to S21 and is repeated.

  As described above, in the terminal device 130 of the second embodiment, when the silent state continues for the third predetermined time T3 or more between the user and the partner user, the position of the video camera 34 that captures the partner user is changed. Extend the time from return to return. Therefore, when the silent state continues for a long time, the time for removing the line of sight can be extended. Therefore, when the line-of-sight matching state is unnatural between users in an awkward atmosphere, the awkward atmosphere can be surely avoided.

  If the number of blinks Y performed by the user during the third predetermined time T3 is equal to or greater than the predetermined value U, the time from the change of the position of the video camera 34 that captures the other user to the return Is extended. For this reason, when the user who unnaturally matches his / her line of sight with the other user is in a tension state, the time for removing the line of sight can be extended. Therefore, a user's tension state can be eased.

  In the above description, the CPU 120 that performs the process of S101 shown in FIG. 20 corresponds to the “user voice detection means” and the “other user voice detection means” of the present invention. The CPU 120 that executes the processes of S27 and S28 shown in FIG. 19 corresponds to the “silent time measuring means” of the present invention. The CPU 120 that executes the processes of S38 and S39 shown in FIG. 19 corresponds to the “first second predetermined time adjusting means” of the present invention. The CPU 120 that executes the process of S35 shown in FIG. 19 corresponds to the “third predetermined time determination means” of the present invention. The CPU 120 that performs the process of S132 shown in FIG. 21 corresponds to the “blink detector” of the present invention. The CPU 120 that performs the process of S134 shown in FIG. 21 corresponds to the “blink counting means” of the present invention. The CPU 120 that executes the processes of S30 and S41 shown in FIG. 19 corresponds to the “second second predetermined time adjusting means” of the present invention. The CPU 120 that executes the process of S27 shown in FIG. 19 corresponds to the “measurement unit” of the present invention. The CPU 120 that executes the process of S35 shown in FIG. 19 corresponds to the “fourth predetermined time determination means” of the present invention. The CPU 120 that executes the process of S37 shown in FIG. 19 corresponds to the “blink number determination means” of the present invention.

  In addition, it cannot be overemphasized that a various change is possible for the said embodiment in the range which does not deviate from invention. For example, although the video conference system of the embodiment includes two terminal devices, the present invention can also be applied to a video conference system including three or more devices. If the time for which the user matches the line of sight with respect to any one of a plurality of other users displayed on the screen is too long, the shooting position of the video camera 34 of the other user who is the target of the line-of-sight match is changed. do it. Even when an awkward atmosphere is created with a specific partner user among a plurality of partner users, the awkward atmosphere can be avoided and the conference can be smoothly advanced.

  In addition, the video camera 34 that captures an image to be transmitted to the terminal device 4 of the other user is used to capture an image for detecting the user's gaze direction. However, a video camera for detecting the gaze direction is provided separately. May be.

Further, the installation position of the infrared light 33 is not limited to the embodiment. In this case, the arrangement position of the infrared light 33 with respect to the screen 32 is stored in the HDD 31. The irradiation angle of the infrared light 33 with respect to the surface of the screen 32 is α, the rotation angle of the eyeball 72 detected by the above method is θ, the intersection point perpendicular to the screen 32 from the center of the eyeball 72 is the center point O, and the eyeball 72 Let L be the distance between the center of and the center point O. A distance K between the line-of-sight position on the display 28 ahead of the user's line-of-sight direction and the center point O is expressed by the following equation.
K = L · tan (θ− (90−α))
When the space on the back surface of the screen 32 is limited, the installation position of the infrared light 33 can be changed.

  In the above-described embodiment, when an unnatural line-of-sight matching state is detected, if the shared material screen 282 exists on the screen 32, the shared material screen 282 is enlarged on the screen 32, but the screen to be enlarged is shared. It is not limited to the document screen 282. It is determined whether another screen of the partner user screen 281 is displayed on the screen 32 with reference to the display screen data storage area 312, and if it is determined that another screen is displayed, another screen is displayed. The image may be enlarged and displayed. Even if the shared material screen 282 exists on the screen 32, the shared material screen 282 does not have to be enlarged.

  In the gaze direction detection process of the second embodiment shown in FIG. 19, the voice detection process is started (S28) and the blink count processing is started (S29), but the voice detection process and the blink count process are performed. Either one may be activated.

  In the above-described embodiment, the distance between the user's eyeball 72 and the screen 32 is approximated to be constant and is stored in advance as the predetermined distance L, but is not limited thereto. For example, the distance between the user's eyeball 72 and the screen 32 is calculated at the start of a teleconference or when the line-of-sight direction is detected, and the user's line-of-sight position T on the screen 32 is specified with the calculated value as the predetermined distance L. You may do it. As a method for calculating the predetermined distance L, a known method can be applied. For example, the following method can be applied.

  In this method, the user is photographed in advance in a state where the video camera 34 and the user (the user's eyeball 72) are at a predetermined distance. The position of the pupil 75 is extracted for each eyeball 72 of the user from the photographed user image, and the extracted interval between the pupils 75 is stored in the HDD 31 in advance as a reference pupil interval. In general, the pupil interval in the captured image is inversely proportional to the distance between the pupil 75 and the video camera 34. Therefore, the distance between the video camera 34 and the eyeball 72 can be calculated from the actual pupil interval photographed by the video camera 34 and the reference pupil interval stored in advance. By storing the positional relationship between the video camera 34 and the screen 32 in the HDD 31 in advance, the distance between the screen 32 and the eyeball 72 can be calculated, and the calculated value can be used as the predetermined distance L.

DESCRIPTION OF SYMBOLS 1 Video conference system 2 Network 3, 4 Terminal device 32 Screen 33 Infrared light 34 Video camera 35 Microphone 36 Camera moving device 71 Purkinje image 72 Eyeball 75 Pupil 76 Pupil center 130 Terminal device 131 Hard disk drive 221 Gaze direction storage area 222 Gaze position memory Area 281 Counterparty user screen 282 Shared material screen 310 Captured image data storage area 312 Display screen data storage area 315 Position relationship storage area t1 Line-of-sight matching time T1 First predetermined time t2 Second transmission time T2 Second predetermined time T3 Third predetermined time

Claims (14)

A communication terminal device that communicates with other terminals connected via a network via images and sounds,
User photographing means for photographing a user;
Image transmitting means for transmitting the user's image captured by the user photographing means to another terminal;
Display means for displaying the image transmitted from another terminal on a screen;
Gaze direction detection means for detecting the gaze direction of the user from an image photographed by the user photographing means;
A line-of-sight position specifying unit for specifying a line-of-sight position on the screen displayed by the display unit located ahead of the line-of-sight direction detected by the line-of-sight direction detection unit;
Face detection means for detecting the user's face on the screen;
A line-of-sight coincidence detecting unit that detects a state where the line-of-sight position identified by the line-of-sight position identifying unit is within a predetermined region of the user's face detected by the face detecting unit;
A line-of-sight matching time measuring unit that measures a line-of-sight matching time, which is a time during which the line-of-sight matching state continues when the line-of-sight matching state is detected by the line-of-sight matching detection unit;
First predetermined time determination means for determining whether or not the line-of-sight coincidence time measured by the line-of-sight coincidence time measurement means is equal to or longer than a first predetermined time;
When the first predetermined time determination means determines that the line-of-sight matching time is equal to or longer than the first predetermined time, the partner side is a terminal that transmits the image data of the partner user who is the target of the line-of-sight matching state First signal transmitting means for transmitting a first signal indicating that the line-of-sight matching time is equal to or longer than the first predetermined time to the terminal;
First signal receiving means for receiving the first signal transmitted from the counterpart terminal;
A communication terminal apparatus comprising: a shooting position changing unit that changes a shooting position of the user shooting unit with respect to the user when the first signal is received by the first signal receiving unit.   The communication terminal apparatus according to claim 1, wherein the photographing position changing unit moves the photographing unit to a position where the photographing unit is retracted from the sight line direction of the user. Further comprising irradiation means for irradiating the user's eyeball with invisible light,
The line-of-sight direction detecting means includes
Detecting the line-of-sight direction from a relative position between a virtual image (Purkinje image) of the invisible light of the user's eyeball cornea reflection surface in the image photographed by the user photographing means and the eyeball pupil center of the user. The communication terminal device according to claim 1, wherein: A positional relationship storage unit that stores a positional relationship between the user's eyeball, the screen, and the irradiation unit;
The line-of-sight position specifying unit specifies the line-of-sight position on the screen from the positional relationship stored in the positional relationship storage unit and the line-of-sight direction detected by the line-of-sight direction detecting unit. The communication terminal device according to claim 3. The user photographing means is arranged behind the screen with respect to the user,
The communication terminal apparatus according to claim 3 or 4, wherein the irradiation unit is attached to the user photographing unit. Another image display determination means for determining whether another image different from the image of the user is displayed on the screen together with the image;
When it is determined by the first predetermined time determination means that the line-of-sight coincidence time has reached the first predetermined time or more and the separate image determination means determines that the separate image is displayed on the screen, 6. The image display apparatus according to claim 1, further comprising: another image enlargement display unit configured to enlarge and display the different image on the screen in comparison with a current display size and to display the image in a transparent manner on the image. The communication terminal device according to 1. The separate image enlargement display means includes
The communication terminal apparatus according to claim 6, wherein the different image is displayed so as to overlap the predetermined area of the user's image. A post-transmission elapsed time measuring means for measuring an elapsed time after transmission, which is an elapsed time since the first signal was transmitted by the first signal transmitting means;
Second predetermined time determination means for determining whether or not the post-transmission elapsed time measured by the post-transmission elapsed time measurement means is equal to or longer than a second predetermined time;
When the second predetermined time determining means determines that the post-transmission elapsed time has become the second predetermined time or more, the post-transmission elapsed time has become the second predetermined time or more to the counterpart terminal. Second signal transmission means for transmitting a second signal indicating that;
Second signal receiving means for receiving the second signal;
When the second signal transmitted from the counterpart terminal is received by the second signal receiving means, the photographing position returning means for returning the photographing position of the photographing means to the original position is further provided. The communication terminal device according to claim 1, wherein the communication terminal device is a device. User voice detecting means for detecting the voice of the user;
Partner user voice detecting means for detecting the voice of the partner user;
A silent time measuring means for measuring a silent time, which is a time during which the silent state continues, with a state in which neither of the user voice detecting means and the counterpart user voice detecting means detects a voice as a silent state;
The first second predetermined time adjusting means for adjusting the second predetermined time based on the silent time measured by the silent time measuring means. The communication terminal device according to 1. The first second predetermined time adjusting means is
A third predetermined time judging means for judging whether or not the silent time measured by the silent time measuring means has become a third predetermined time or more;
10. The communication terminal device according to claim 9, wherein the second predetermined time is extended when the third predetermined time determination unit determines that the silent time has become equal to or longer than the third predetermined time. . Blink detection means for detecting the blink of the user;
Blink counting means for counting the number of blinks detected by the blink detection means;
11. The apparatus according to claim 1, further comprising: a second second predetermined time adjusting unit that adjusts the second predetermined time based on the number of times counted by the counting unit. Communication terminal device. The second second predetermined time adjusting means is
Measuring means for measuring an elapsed time after the line-of-sight matching state is detected by the line-of-sight matching detection means;
A fourth predetermined time determining means for determining whether or not the elapsed time measured by the measuring means has become a fourth predetermined time or more;
A blink that determines whether or not the number of times counted by the blink counting means is greater than or equal to a predetermined number of times when the elapsed time is determined to be greater than or equal to the fourth predetermined time by the fourth predetermined time determination means. Means for determining the number of times;
With
The communication terminal apparatus according to claim 11, wherein the second predetermined time is extended when the blink number determination means determines that the number of times is equal to or greater than a predetermined number. A communication terminal device communication control method for communicating with other terminals connected via a network via images and sounds,
An image data transmission step of transmitting the image of the user taken by the user photographing means for photographing the user to another terminal;
A display step of displaying the image transmitted from another terminal on a screen;
A gaze direction detection step of detecting the gaze direction of the user;
A line-of-sight position specifying step for specifying the line-of-sight position on the screen displayed by the display means ahead of the line-of-sight direction detected in the line-of-sight direction detection step;
A face detection step for detecting the user's face on the screen and a line-of-sight match in which the line-of-sight position specified in the line-of-sight position specifying step is within a predetermined region of the user's face detected in the face detection step. A line-of-sight coincidence determining step for determining whether or not a state is present;
A line-of-sight matching time measuring step of measuring a line-of-sight matching time, which is a time during which the line-of-sight matching state continues when it is determined in the line-of-sight matching determination step;
A first predetermined time determination step for determining whether or not the line-of-sight matching time measured in the line-of-sight matching time measurement step is equal to or longer than a first predetermined time;
In the first predetermined time determination step, when it is determined that the line-of-sight matching time is equal to or longer than the first predetermined time, to the partner terminal that transmits the image data of the partner user who is the target of the line-of-sight matching state, A first signal transmission step of transmitting a first signal indicating that the line-of-sight coincidence time is equal to or longer than the first predetermined time;
A first signal receiving step for receiving the first signal;
In the first signal receiving step, when the first signal transmitted from the counterpart terminal is received, a photographing position changing step of changing a photographing position of the photographing unit with respect to the user is provided. Communication control method for a communication terminal device.   A communication control program for causing a computer to function as various processing means of the communication terminal device according to claim 1.

Images