JP2010154387A - Communication terminal device, communication control method, and communication control program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a communication terminal device capable of making a smooth conversation with a listening participant of an opposite-side terminal, and to provide a communication control method and a communication control program. <P>SOLUTION: A terminal device 3 is interconnected to other terminal devices 4-6 via a network 2 and constitutes a video conference system 1. In this system, when it is detected that a listening participant nods at a specific terminal device during a remote conference, an opposite-side terminal device of a speaking participant is notified regarding the detection of nodding of the listening participant. At the opposite-side terminal device notified of the nodding, a nodding image of the listening participant, stored beforehand, is displayed. Since the present system does not require encoding and decoding of image data differently from a streaming system, nodding of the listening participant can be displayed without delay. Consequently, deviation between timing of speaking and a reaction of the listening participant can be reduced, and thereby smooth conversations are provided. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a communication terminal device capable of bidirectionally transmitting and receiving images and sound to and from a counterpart terminal, a communication control method for the communication terminal device, and a communication control program.

2. Description of the Related Art Conventionally, there has been known a video conference system in which a plurality of communication terminal devices are connected via a network and images and sound are bidirectionally transmitted and received to realize a conference between persons in remote locations. For example, in order to improve mental stability, concentration and motivation, an electronic conference apparatus and an electronic conference system are known in which a predetermined odor, sound, image, or the like is output from a terminal to improve the efficiency of the conference. (For example, refer to Patent Document 1). In such a system, the “streaming method” is often adopted when transmitting and receiving images. The streaming method is a method of simultaneously reproducing data while receiving it when viewing multimedia data such as an image or sound over a network.
JP-A-7-107453

  However, in the above streaming method, it takes time to encode and decode image data when transmitting and receiving image data to and from the partner terminal. That is, a delay time occurs when displaying an image. For example, when the listener at the other terminal agrees with the content spoken by the speaker during the conference, the image data at that time is encoded. The encoded image data is received by another terminal device via the network. In each terminal device, the received image data is decoded, and the image data is displayed on the display of the terminal device. In such a system, the whirling reaction is displayed with a delay from the actual time. Therefore, there is a problem that it is difficult to speak because the timing of speaking and the reaction of the listener may be slightly different.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a communication terminal device, a communication control method, and a communication control program that can smoothly talk with a listener of a counterpart terminal.

  In order to achieve the above object, a communication terminal apparatus according to claim 1 is a communication terminal apparatus that communicates with a counterpart terminal via an image via a network, and is transmitted from the counterpart terminal. Display means for displaying image data; reaction state detection means for detecting a reaction state of a user; and when the reaction state is detected by the reaction state detection means, Reaction signal transmitting means for transmitting a reaction signal indicating that the reaction is detected; reaction signal receiving means for receiving the reaction signal transmitted by the reaction signal transmitting means; When the reaction signal is received by the reaction signal receiving means, and when the reaction signal is received by the reaction signal receiving means, it is recorded in the reaction image storage means. The reaction time image of the counterpart user is, and an reaction time of image display control means for displaying on the display means.

  According to a second aspect of the present invention, in addition to the configuration of the first aspect of the present invention, the communication terminal device is configured such that the display means transmits the partner transmitted from the partner terminal and photographed by the partner terminal. An interrupt display means for interrupting the streaming image displayed on the display means and displaying the reaction time image when a streaming image of a user is displayed and the reaction signal is received by the reaction signal receiving means; And a first streaming image cut means for cutting the streaming image for a time corresponding to the time of the reaction image from when the reaction image is interrupted by the interrupt display means.

  According to a third aspect of the present invention, in addition to the configuration of the first aspect of the invention, the communication terminal device transmits the user's streaming image to be transmitted to the counterpart terminal by the reaction-time image data transmission means. A second streaming image cutting means for cutting the time corresponding to the time of the reaction image data and transmitting the cut image data to the counterpart terminal.

  According to a fourth aspect of the present invention, there is provided a communication terminal device according to any one of the first to third aspects, in addition to the reaction when the reaction state of the user is detected by the reaction state detection unit. Reaction image data transmitting means for compressing the reaction image data and transmitting it to the counterpart terminal, and Reaction image data for receiving the reaction image data transmitted by the reaction image data transmission means of the counterpart terminal Receiving means; and decompression storage processing means for decompressing the reaction image data received by the reaction image data receiving means and storing the decompressed image data in the reaction image storage means.

  In addition to the configuration of the invention according to any one of claims 1 to 4, the communication terminal device according to the invention according to claim 5 is connected to the plurality of counterpart terminals via the network. A terminal specifying means for specifying one terminal for detecting the reaction state of the counterpart user based on a predetermined condition from the plurality of the counterpart terminals, and the one specified by the terminal specifying means A notification signal transmitting unit that transmits a notification signal for notifying the terminal that the reaction state has been identified, and a notification signal receiving unit that receives the notification signal transmitted from the counterpart terminal; The reaction state detection unit detects the reaction state of the user when the notification signal is received by the notification signal reception unit.

  According to a sixth aspect of the present invention, there is provided a communication terminal device according to the sixth aspect, in addition to the configuration of the fifth aspect of the invention, a CPU load state detecting means for detecting a load state of the CPU of the counterpart terminal via the network; CPU load status storage means for storing the CPU load status detected for each counterpart terminal by the CPU load status detection means, and the terminal specifying means is stored in the CPU load status storage means. With reference to the CPU load status, a first predetermined condition for specifying the counterpart terminal with the lowest CPU load as the one terminal is provided.

  According to a seventh aspect of the present invention, in addition to the configuration of the fifth aspect of the invention, the communication terminal apparatus detects a transmission time of data in communication with the counterpart terminal via the network. And a transmission time storage means for storing the detection result by the transmission time detection means, the terminal specifying means refers to the detection result stored in the transmission time storage means, and the transmission time is shortest. A second predetermined condition for specifying the counterpart terminal as the one terminal is provided.

  According to an eighth aspect of the present invention, there is provided a communication terminal device according to the fifth aspect of the present invention, in addition to the configuration of the fifth aspect of the present invention, a login number detection means for detecting the number of logged-in partner users for each counterpart terminal; Login number storage means for storing the number of logins detected by the login number detection means, wherein the terminal specifying means is the largest number of logins among the number of logins for each of the other party terminals stored in the login number storage means. A third predetermined condition for specifying the counterpart terminal as the one terminal.

  According to a ninth aspect of the present invention, in addition to the configuration of the fifth aspect of the invention, the communication terminal device further includes an utterance detection unit for detecting the utterance of the counterpart user, and the counterpart user detected by the utterance detection unit. An elapsed time measuring means for measuring an elapsed time from the time of each utterance, and the terminal specifying means, the partner terminal of the partner user with the shortest elapsed time measured by the elapsed time measuring means, A fourth predetermined condition for specifying the one terminal is provided.

  According to a tenth aspect of the present invention, there is provided the communication terminal device according to any one of the first to ninth aspects, wherein the reaction state detecting means sway the user's head in a predetermined direction. A whirling state is detected as the reaction state.

  According to an eleventh aspect of the present invention, there is provided a communication terminal device according to any one of the first to ninth aspects, wherein the reaction state detecting means rejects the user's head swinging laterally. The rejection state to be detected is detected as the reaction state.

  According to a twelfth aspect of the present invention, there is provided a communication terminal apparatus according to any one of the first to ninth aspects, further comprising operation means operated by the user when the user is satisfied. The reaction state detection means detects the user's consent state as the reaction state by detecting an operation by the operation means.

  According to a thirteenth aspect of the present invention, in addition to the configuration of the invention according to any one of the first to twelfth aspects, the communication terminal device according to the thirteenth aspect is the first time that the reaction state of the user detected by the reaction state detection unit is Initial reaction state determination means for determining whether or not the reaction state is stored in the reaction image storage means when the reaction state is determined to be the first time by the initial reaction state determination means. The reaction image data transmission means transmits the reaction image data stored in the reaction image storage means to the counterpart terminal in a compressed state.

  According to a fourteenth aspect of the present invention, there is provided a communication terminal device according to the first aspect, wherein the reaction signal is received when the reaction signal is received by the reaction signal receiving means. A response-time image storage determination unit that determines whether or not the response-time image data of the partner user is stored in the image storage unit, and that the response-time image storage determination unit does not store the response-time image. When judged, an alternative image display for displaying, on the display means, an alternative image indicating that the counterpart user indicates the reaction state by using a character, a figure, a symbol or the like instead of the reaction time image Control means.

  A communication control method according to a fifteenth aspect of the present invention is a communication control method for a communication terminal apparatus that communicates with an opponent terminal via an image via a network, the image data transmitted from the opponent terminal. When the reaction state is detected in the reaction state detection step, the reaction state detection step for detecting the reaction state of the user, and the reaction state to the user when the reaction state is detected in the reaction state detection step. A reaction signal transmitting step for transmitting a reaction signal indicating that the reaction signal has been detected; a reaction signal receiving step for receiving the reaction signal transmitted in the reaction signal transmitting step; and the reaction signal received in the reaction signal receiving step. The reaction time image for storing the response time image when the other party user of the counterpart terminal indicates the reaction state The reaction time images stored in the other user to 憶 means, and a reaction time of image display control step of displaying on said display means for displaying said received image data in the image data receiving step.

  A communication control program according to a sixteenth aspect is caused to be executed by a computer as various processing steps of the communication control method according to the fifteenth aspect.

  In the communication terminal device according to the first aspect of the present invention, communication via the image can be performed with the counterpart terminal. The image data transmitted from the counterpart terminal is displayed on the display means. The reaction state of the user is detected by the reaction state detection means. When a reaction state of the user is detected by the reaction state detection unit, a reaction signal is transmitted to the counterpart terminal by the reaction signal transmission unit. On the other hand, a response image when the partner user of the partner terminal indicates a response state is stored in the response image storage means. Then, when the reaction signal is received by the reaction signal receiving means, the reaction time image display control means controls so that the reaction time image of the opponent user stored in the reaction time image storage means is displayed on the display means. That is, unlike the streaming method, the other party's reaction time image is stored in the reaction time image storage means, and when there is a reaction, the reaction time image is displayed on the display means, so the encoding and decoding of the image data Is unnecessary. Accordingly, since the difference between the speaking timing and the listener's reaction becomes small, it is possible to smoothly talk with the listener at the other terminal.

  In the communication terminal device according to the second aspect of the invention, in addition to the effect of the first aspect of the invention, the display means transmits a streaming image of the other user transmitted from the other party terminal and photographed by the other party terminal. Is displayed. Then, when the reaction signal is received by the reaction signal receiving means, the interrupt display means interrupts and displays the reaction time image on the streaming image displayed on the display means. Furthermore, from the time when the response image is interrupted by the interrupt display means, the streaming image corresponding to the time of the response image is cut by the first streaming image cut means. Thereby, since the image at the time of reaction is not displayed on the display means in an overlapping manner, there is no sense of incongruity.

  Moreover, in the communication terminal device of the invention according to claim 3, in addition to the effect of the invention of claim 1, the second streaming image cut means converts the streaming image of the user transmitted to the counterpart terminal into the response image data. Cut by the time corresponding to the time of the image data at the time of reaction transmitted by the transmission means, and transmit to the counterpart terminal. As a result, the images at the time of reaction are not displayed on the display means in a duplicated manner, so that the feeling of strangeness is eliminated.

  In addition, in the communication terminal device of the invention according to claim 4, in addition to the effect of the invention according to any one of claims 1 to 3, image data at the time of reaction when the reaction state of the user is detected by the reaction state detection means Is compressed by the reaction image data transmission means and transmitted to the counterpart terminal. The reaction time image data transmitted from the counterpart terminal is received by the reaction time image data receiving means. The received reaction image data is stored in the reaction image storage unit in a state where it has been decompressed by the decompression storage processing unit. As a result, the reaction time image of the counterpart terminal can be stored at the beginning of the conversation, so that the response image during the subsequent conversation can be quickly displayed on the display means.

  Further, in the communication terminal device of the invention according to claim 5, in addition to the effect of the invention according to any one of claims 1 to 4, when the terminal is connected to a plurality of counterpart terminals, the terminal specifying means includes the plurality of terminals. One terminal that detects the reaction state of the other user is specified based on a predetermined condition. When the terminal is specified, the notification signal transmitting means transmits a notification signal for notifying that the terminal is specified as the terminal that detects the reaction state toward the specified one terminal. The notification signal transmitted from the counterpart terminal is received by the notification signal receiving means. The reaction state detection unit detects the reaction state of the user when the notification signal is received by the notification signal reception unit. That is, even when connected to a plurality of counterpart terminals via a network, one terminal for detecting a reaction state is specified, so that the communication load and delay time in the network can be minimized.

  In the communication terminal device according to the sixth aspect of the invention, in addition to the effect of the fifth aspect of the invention, the CPU load status detecting means detects the CPU load status of the counterpart terminal via the network. The detected CPU load status is stored for each counterpart terminal in the CPU load status storage means. The terminal specifying means has a first predetermined condition for referring to the CPU load status stored in the CPU load status storage means and specifying the counterpart terminal with the lowest CPU load as one terminal. Thereby, the influence of the load by detecting a user's reaction state can be suppressed to the minimum.

  In the communication terminal device of the invention according to claim 7, in addition to the effect of the invention of claim 5, the transmission time of the data in communication with the counterpart terminal via the network is detected by the transmission time detection means. The The detection result is stored in the transmission time storage means. The terminal specifying means has a second predetermined condition for referring to the detection result stored in the transmission time storage means and specifying the counterpart terminal having the shortest transmission time as one terminal. Thereby, since image data can be transmitted quickly, the image at the time of reaction can be quickly displayed on the display means.

  In the communication terminal device according to the eighth aspect of the invention, in addition to the effect of the fifth aspect, the number of logged-in partner users is detected for each partner-side terminal by the log-in number detecting means. The number of logins detected by the login number detection means is stored in the login number storage means. The terminal specifying means has a third predetermined condition for specifying, as one terminal, the partner terminal that has the largest number of logins among the number of logins for each partner terminal stored in the login number storage means. Thereby, since more reaction states can be detected at one site (one terminal), the conversation can proceed more smoothly.

  In the communication terminal device according to the ninth aspect of the invention, in addition to the effect of the fifth aspect of the invention, the speech of the other user is detected by the speech detection means. Furthermore, the elapsed time from the utterance time for each partner user detected by the utterance detection means is measured by the elapsed time measurement means. The terminal specifying unit includes a fourth predetermined condition for specifying the partner terminal of the partner user having the shortest elapsed time measured by the elapsed time measuring unit as one terminal. Thereby, the terminal of the speaker who talked last time can be identified with priority.

  Moreover, in the communication terminal device of the invention according to claim 10, in addition to the effect of the invention according to any one of claims 1 to 9, the reaction state detection means is a state in which the user's head is swung in a predetermined direction. Is detected as a reaction state. As a result, the speaker can quickly recognize the whisper of the listener, so that the speaker is not anxious due to the delay of the listener's reaction. Therefore, a smooth conversation between the speaker and the listener is possible.

  Moreover, in the communication terminal device of the invention according to claim 11, in addition to the effect of the invention according to any one of claims 1 to 9, the reaction state detection means rejects that the user's head shakes in the horizontal direction and rejects The state is detected as a reaction state. As a result, the speaker can quickly recognize the listener's rejection reaction, so that the speaker is not disturbed by the delay in the listener's response.

  In the communication terminal device according to the twelfth aspect of the present invention, in addition to the effects of the invention according to any one of the first to ninth aspects, the user operates the operation means when the user is satisfied. The reaction state detection means detects the user's satisfaction state as the reaction state by detecting an operation by the operation means. This allows the speaker to quickly recognize the listener's satisfaction, so that the speaker is not disturbed by the delay in the listener's reaction.

  In addition, in the communication terminal device of the invention according to claim 13, in addition to the effect of the invention according to any one of claims 1 to 12, the initial reaction state determination means is the user reaction state detected by the reaction state detection means Determine whether is the first time. When the reaction state is determined to be the first time by the initial reaction state determination means, there is a high possibility that no reaction time image is stored in the reaction time storage means. Therefore, since the reaction time image storage processing means stores the reaction time image in the reaction time image storage means, the reaction time image can be stored during the conversation. That is, there is no need to store the reaction time image in advance in the reaction image storage means.

  Further, in the communication terminal device of the invention according to claim 14, in addition to the effect of the invention according to any of claims 1 to 13, when a reaction signal is received by the reaction signal receiving means, a reaction time image storage judging means However, it is determined whether or not the other party user's reaction time image is stored in the reaction time image storage means. When it is determined that the response image is not stored, the alternative image display control means uses a character, figure, symbol or the like instead of the response image to indicate that the other user shows the response state The image is displayed on the display means. As a result, even when no response time image is stored in the response time image storage, the substitute image can be displayed instead of the response time image, so that a smooth conversation can be provided.

  In the communication control method according to the fifteenth aspect of the present invention, first, image data transmitted from the counterpart terminal is received in the image data receiving step. Next, in the reaction state detection step, the reaction state of the user is detected. When a reaction state is detected in the reaction state detection step, a reaction signal indicating that the reaction state has been detected is transmitted to the user terminal in the reaction signal transmission step. The reaction signal transmitted from the counterpart terminal is received in the reaction signal receiving step. When the reaction signal is received in the reaction signal reception step, the reaction time image of the opponent user stored in the reaction time image storage means is displayed on the display means in the reaction time image display control step. That is, unlike the streaming method, the other party's reaction time image is stored in the reaction time image storage means, and when there is a reaction, the reaction time image is displayed on the display means, so the encoding and decoding of the image data Is unnecessary. Accordingly, since the difference between the speaking timing and the listener's reaction becomes small, it is possible to smoothly talk with the listener at the other terminal.

  The communication control program according to the sixteenth aspect of the present invention is caused to be executed by a computer as various processing steps of the communication control method according to the fifteenth aspect, whereby the effect of the invention of the fifteenth aspect can be obtained.

  Hereinafter, the terminal device 3 which is 1st Embodiment of this invention is demonstrated with reference to drawings. First, the configuration of the video conference system 1 including the terminal device 3 as a component will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of the video conference system 1.

  The video conference system 1 includes a network 2 and a plurality of terminal devices 3, 4, 5, and 6 that are connected to each other via the network 2 and provided at each base. In the video conference system 1, a remote conference is performed by transmitting and receiving images and sounds between the terminal devices 3, 4, 5, and 6 via the network 2.

  In this embodiment, when it is detected that a listener is speaking at a specific terminal device during a remote conference, the other terminal device on which the speaker is present is notified that the listener's whisper has been detected. The partner terminal device that is notified of the whispering displays a pre-stored listener whispering image. Since this method does not require encoding and decoding of a whispered image, it is characterized in that the listener's whisper can be displayed quickly.

  Next, the electrical configuration of the terminal device 3 will be described with reference to FIG. FIG. 2 is a block diagram showing an electrical configuration of the terminal device 3. In addition, since all the terminal devices 3-6 are the same structures, only the structure of the terminal device 3 is demonstrated here, and description is abbreviate | omitted about the other terminal devices 4-6.

  The terminal device 3 is provided with a CPU 20 as a controller that controls the terminal device 3. Connected to the CPU 20 are a ROM 21 that stores BIOS, a RAM 22 that temporarily stores various data, and an I / O interface 30 that mediates data transfer. The I / O interface 30 is connected to a hard disk drive 31 (hereinafter referred to as HDD 31) having various storage areas.

  The I / O interface 30 includes a communication device 25 for communicating with the network 2, a mouse 27, a video controller 23, a key controller 24, a card reader control unit 32, and a camera 34 for photographing a user. A microphone 35 for capturing the user's voice and a CD-ROM drive 26 are connected to each other. A display 28 is connected to the video controller 23. A keyboard 29 is connected to the key controller 24. A card reader 33 for reading a user ID for identifying a user stored in an identification card (not shown) owned by each user is connected to the card reader control unit 32.

  The CD-ROM 114 inserted into the CD-ROM drive 26 stores the main program of the terminal device 3, the communication control program of the present invention, and the like. When the CD-ROM 114 is introduced, these various programs are set up from the CD-ROM 114 to the HDD 31 and stored in a program storage area 315 (see FIG. 3) described later.

  Next, various storage areas of the HDD 31 will be described with reference to FIG. FIG. 3 is a conceptual diagram showing various storage areas of the HDD 31. The HDD 31 manages a login table storage area 311 for storing a login table 3111 (see FIG. 4) for managing users who have logged into the network 2 and terminal IDs of the terminal devices 3 to 6 connected to the network 2. , A terminal status table storage area 312 for storing a terminal status table 3121 (see FIG. 5) for storing each operation status of the connected terminal devices 3 to 6, and a shooting time when each of the terminal devices 3 to 6 is photographed. A fired image data table storage area 313 for storing a fired image data table 3131 (see FIG. 6) for storing and managing image data (hereinafter referred to as fired image data) and a camera image taken by the camera 34 are stored. Camera image data storage area 314 to be stored, and program storage area 31 to store various programs When, with the other information storage area 316, a waveform pattern storage area 317 for previously storing a waveform pattern when nodding users, content image storage area 318 for storing the content image for nodding it is at least provided.

  The program storage area 315 stores a main program of the terminal device 3, a communication control program for executing a remote conference with the other terminal devices 4, 5, and 6 and the like. In the other information storage area 316, other information used in the terminal device 3 is stored. When the terminal device 3 is a dedicated machine that does not include the HDD 31, various programs are stored in the ROM 21.

  Next, the login table 3111 will be described with reference to FIG. FIG. 4 is a conceptual diagram of the login table 3111. The login table 3111 stores the user ID of the user who has logged into the network 2 and the terminal IDs of the terminal devices 3 to 6 in which the user ID is registered. Specifically, the user ID stored in the identification card (not shown) read by the card reader 33 is stored in the user ID column. The terminal ID column stores the terminal IDs of the terminal devices 3 to 6 that transmitted the user ID. The terminal ID is a Mac address of the terminal device 3 or the like. Further, the user ID and the terminal ID are included in terminal information transmitted from other terminal devices 4 to 6 connected via the network 2 and are registered in the login table 3111 based on the terminal information.

  For example, as shown in FIG. 4, when Mr. B who is the user of the terminal device 4 logs in, he causes the card reader 33 to read his own identification card. Then, a login signal is transmitted to the other party's terminal device, and the other party is notified that the user has logged in. In this case, the user ID = “B0001” stored in the identification card and the terminal ID = “0002” of the terminal device 4 that transmitted the user ID are stored in the login table 3111, respectively. The same is set for other users.

  Note that the login table 3111 shown in FIG. 4 includes one person (user ID = A0001) from the terminal apparatus 3 (terminal ID = 0001) and one person (user ID = B0001) from the terminal apparatus 4 (terminal ID = 0002). Two users (user ID = C0001, C0002) logged in from the terminal device 5 (terminal ID = 0003), and three people (user ID = D0001, D0002, D0003) logged in from the terminal device 6 (terminal ID = 0004), respectively. Indicates the state.

  Next, the terminal status table 3121 is stored with reference to FIG. FIG. 5 is a conceptual diagram of the terminal status table 3121. The terminal status table 3121 is a table that records the operating status of its own terminal and each operating status of a partner terminal device (hereinafter referred to as a connected terminal) connected via the network 2. The terminal status table 3121 includes a terminal ID column 61 for storing the terminal ID, a conversation direction column 62 for storing the direction of conversation in which the speaker speaks to the listener between the bases for each terminal, and the CPU load. CPU load column 63 for storing the status for each terminal, delay column 64 for storing the delay time of data transmission for each terminal, a soot detection function column 65 for storing the presence or absence of a soot detection function for each terminal, and a login table There are provided a log-in number column 66 for storing the number of users (log-in number) registered in 3111 for each terminal, and a post-speech elapsed time column 67 for storing the elapsed time since the last utterance for each terminal. It has been.

  The data transmission time delay refers to the time required for data to reach the destination from the transmission side. The last utterance elapsed time column 67 stores the elapsed time since the utterance was detected by the microphone 35 in each of the terminal devices 3 to 6. In the conversation direction column 62, a terminal device that detects the user's utterance with the microphone 35 is registered as a speaker, and other terminal devices are registered as listeners.

  Further, each value stored in the terminal status table 3121 is stored based on terminal information transmitted from each terminal device via the network 2. The terminal information includes the terminal ID of each terminal, CPU load (%), data transmission time delay (ms), presence / absence of a whisper detection function, elapsed time after utterance, and the like. In the login number column 66, the number of user IDs stored in the login table 3111 is stored for each terminal.

  For example, as illustrated in FIG. 5, the operation status of the terminal device 4 is stored in the second row of the terminal status table 3121. That is, the terminal ID column 61 = “0002”, the conversation direction column 62 = “listener”, the CPU load column 63 = “50%”, the delay column 64 = “10 ms”, the whisper detection function column 65 = “Yes”, the number of login users Column 66 = “one person” and post-speech elapsed time column 67 = “one second ago” are stored.

  Next, the roaring image data table 3131 will be described with reference to FIG. FIG. 6 is a conceptual diagram of the roaring image data table 3131. The whispered image data table 3131 stores the terminal ID, the whispered image data of the user corresponding to the terminal ID, and the recording time for each terminal. As will be described later, the hand-drawn image captured by each terminal device is transmitted in an encoded and compressed state. The received hand-drawn image data is managed for each terminal ID in a decoded state and stored in the hand-drawn image data table 3131.

  For example, as shown in FIG. 6, the second row of the whirling image data table 3131 contains whispered image data = bbb. avc and recording time = 2.4 seconds are stored in association with each other. Note that bbb. avc is stored in an uncompressed state.

  Next, an image displayed on the display 28 of the terminal device 3 will be described with reference to FIG. FIG. 7 is a diagram showing one display mode on the display 28. During the conference, three divided screens 281, 282, and 283 are displayed on the display 28 of the terminal device 3 in order to display each user of the other terminal devices 4, 5, and 6. For example, the divided screen 281 is arranged on the substantially left half of the display 28, the divided screen 282 is arranged on the upper right side of the display 28, and the divided screen 283 is arranged on the lower side of the right half of the display 28. Is displayed. An image of the user of the terminal device 4 is displayed on the divided screen 281. An image of the user of the terminal device 5 is displayed on the divided screen 282. On the divided screen 283, an image of the user of the terminal device 6 is displayed. Note that the display mode is not limited to this, and the arrangement and size of each of the divided screens 281 to 283 can be freely changed. FIG. 7 shows a state in which the user of the terminal device 4 is talking on the split screen 281.

  Next, a method for detecting a user's whispering will be described with reference to FIGS. FIG. 8 is an explanatory diagram of the characteristic amount d indicating the amount of change (before turning). FIG. 9 is an explanatory diagram of the feature amount d indicating the amount of change (after the turn). FIG. 10 is a conceptual diagram of the camera image data 40. FIG. 11 is a graph showing a detected waveform pattern (when blinking). FIG. 12 is a graph showing registered whispering waveform patterns.

  Here, the “whispering state” refers to a state in which the listener's head swings upward or downward by a predetermined amount or more when the listener is satisfied with the content spoken by the speaker. In this embodiment, the shake of the user's head is detected by well-known image processing. However, for example, an identification method using a state identification device described in Japanese Patent Application Laid-Open No. 2007-97668 is applicable.

  Here, a specific example of the whirl detection method to which the identification method is applied will be described. First, the camera image data transferred from the camera 34 is stored in the camera image data storage area 314 (see FIG. 3) of the HDD 31. Then, a person image is detected from the camera images stored in the camera image data storage area 314. Next, a facial feature amount is calculated for each detected person. In the present embodiment, the position coordinates between the eyebrows are acquired by detecting the space between the eyebrows or the eyes, and the position coordinates of the lowermost end portion of the face shown in the image are acquired from the detected face outline. Then, a difference value between the two acquired position coordinates is calculated.

  For example, when the face shown in the camera image is a front face, as shown in FIG. 8, the position coordinates of the chin are acquired as the position coordinates of the lowest end part shown in the face image. On the other hand, when the face shown in the camera image is a face that looks down, as shown in FIG. 9, the coordinates of the position closer to the eyes, such as the nose, are acquired as the position coordinates of the lowest end part shown in the face image. The As is clear from the comparison between FIG. 8 and FIG. 9, the distance d from the space between the eyebrows to the lowermost end portion of the face shown in the image is the longest in the front face, and becomes shorter as the degree of depression is increased. Therefore, it is possible to determine whether the face is muted or not based on the difference value between the two position coordinates. Various techniques are known for identifying a face based on feature amount extraction, and any of these techniques can be employed in the present embodiment.

  Then, camera image data 40 (see FIG. 10) in which the calculated feature amount d is added with information on the shooting time included in the management information of the camera image and the user ID assigned when the face is detected and identified. ) And is stored in the camera image data storage area 314 (see FIG. 3). Then, by repeating the above processing, a plurality of camera image data 40 representing the degree of depression of the listener at each time is accumulated in the camera image data storage area 314.

  Furthermore, the camera image data 40 for the shooting time of 10 seconds generated immediately before is read from the camera image data storage area 314 and classified by user based on the user ID. Subsequently, the data of each listener is arranged in time series based on the time information. A detection waveform pattern (see FIG. 11) representing a change with time of the feature amount (distance d) is generated from the data group arranged in time series.

  Then, the generated detected waveform pattern is collated with a waveform pattern (see FIG. 12) registered in advance in the waveform pattern storage area 317 (see FIG. 3) of the HDD 31. In the present embodiment, a short waveform pattern (see FIG. 12) of about 1 second, which is a waveform indicating that the lightly nailing operation has been performed, is stored. This waveform pattern is referred to as a “whit pattern”. That is, if the detected waveform pattern matches the whirling pattern, it can be determined that it is scooping. Note that the waveform of the whirling pattern is not limited to this pattern and can be freely changed.

  Next, communication control processing executed by the CPU 20 of the terminal device 3 having the above configuration will be described with reference to the flowcharts of FIGS. 13 to 17 and FIG. FIG. 13 is a flowchart of the communication control process. FIG. 14 is a flowchart showing a continuation of FIG. FIG. 15 is a flowchart showing a continuation of FIG. FIG. 16 is a flowchart showing a continuation of FIG. FIG. 17 is a flowchart showing a continuation of FIG. FIG. 18 is a timing chart for explaining the image cut processing at the time of detection of a whirl.

  This communication control process is performed not only in the terminal device 3 but also in the other terminal devices 4 to 6. Therefore, only the communication control process executed by the CPU 20 of the terminal device 3 will be described here.

  As shown in FIG. 13, first, various data are initialized (S1). And a user operates the recording switch (not shown) provided in the terminal device 3 in order to record a rolling image before a meeting. Along with this, it is determined whether or not there is an instruction to record a whispered image by operating the recording switch (S2). While the recording switch is not operated (S2: NO), the operation of the recording switch is continuously monitored (S2). When the recording switch is operated (S2: YES), a content image that can be displayed on the display 28 is reproduced (S3). The content image is stored in the content image storage area 318 of the HDD 31, and for example, an image that sympathizes with the user or an image that encourages the user to use it can be adopted.

  Then, it is determined whether or not the user's whirling photographed by the camera 34 has been detected (S4). In addition, a user's whisper is detected by the above-described detection method. Until the user asks (S4: NO), the process returns to S4, and it is continuously determined whether or not the user's whisper is detected. When the user hears the content image (S4: YES), the whispered image is recorded (S5). Note that the recorded image data of the recorded image is stored in the expanded image data table 3131 stored in the expanded image data table storage area 313 of the HDD 31 together with the recording time.

  Subsequently, it is determined whether or not it is connected to at least one of the other terminal devices 4 to 6 via the network 2 (S6). Until connected to any of the other terminal devices 4 to 6 (S6: NO), the process returns to S6 and the process is repeated. When connected to another terminal device (S6: YES), based on the terminal information transmitted from the connected terminal device, the terminal status table in which the operating status of the terminal device is stored in the terminal status table storage area 312 of the HDD 31 It is stored in 3121 (see FIG. 5) (S7). Further, the load on the CPU 20 of the terminal device 3 is measured and stored in the terminal status table 3121 (see FIG. 5) (S8). Further, the data transmission time delay of the terminal device 3 is measured and stored in the terminal status table 3121 (see FIG. 5) (S9).

  Furthermore, the presence or absence of the whirl detection function of the terminal device 3 is stored in the terminal status table 3121 (see FIG. 5) (S10). In the present embodiment, since the terminal device 3 has a whirl detection function, “whether” is stored in the whirl detection function column 65 of the terminal status table 3121. More specifically, when the presence / absence of the presence / absence detection function is registered, the presence / absence function flag “1” is stored. In the case of “None”, the rolling function flag “0” is stored.

  Further, although not shown, the number of login users who have logged into the network 2 from the terminal device 3 is detected, and the detected number of login users is stored in the terminal status table 3121 (see FIG. 5). Further, the user's utterance is detected, and the elapsed time from the utterance is stored in the terminal status table 3121 (see FIG. 5). The CPU 20 that detects the number of logins corresponds to the “login number detection means” of the present invention, and the CPU 20 that stores the detected number of logins corresponds to the “login number storage means” of the present invention. The CPU 20 that measures the elapsed time after the utterance is detected by the microphone 35 corresponds to the “elapsed time measuring means” of the present invention.

  Next, it is determined whether or not a speaker has been detected (S12). When the user's voice is detected from the microphone 35, it is determined as a speaker, and when the voice is not detected, it is determined as a listener. Here, since the subsequent processing changes depending on whether it is a speaker or a listener, the following will be described in order for a case where a speaker is detected and a case where a speaker is detected without being detected. .

  First, a case where a speaker is detected by the terminal device 3 will be described. As shown in FIG. 14, when a speaker is detected at the base of the terminal device 3 (S12: YES), one terminal device that detects whispering is identified from other terminal devices that have become listeners ( S13). Here, one terminal device that detects whispering is specified based on the condition determined based on the operation status of each of the terminal devices 4 to 6. In addition, when there is only one other terminal device as a listener, the process of specifying one terminal device that detects whispering is not performed.

  Here, conditions for specifying a terminal will be described. In the present embodiment, there are four types of first condition, second condition, third condition, and fourth condition. In the first condition, a terminal device with the least CPU load is specified. If the terminal device has a low CPU load, it can be quickly processed even when the whisper detection function is used. The second condition specifies the terminal device with the shortest data transmission time delay. The shorter the data transmission time delay, the shorter the time required to send and receive the whispering signal, and the faster it can be processed. The third condition specifies a terminal device with the largest number of logins. A base with a large number of logged-in persons can increase the rate of detection of whispering, so that the conversation can proceed more smoothly. The fourth condition specifies a terminal device that has the shortest elapsed time since the last utterance (elapsed time after utterance). That is, the terminal device of the speaker who was speaking last time can be identified with priority. Since it is possible to detect the whispering of the user who has spoken most recently in the current conversation, an effective impression can be given to the current speaker and a smoother conversation can be provided.

  In the present embodiment, the terminal device 3 can select any one of these first to fourth conditions. Based on the condition selected by the user, the CPU 20 identifies one terminal device that detects the whisper. It should be noted that when one terminal device that detects whispering is specified, those that do not have the whisper detection function are excluded. In the terminal status table 3121 shown in FIG. 5, the terminal device 6 (terminal ID = 0004) does not have a whisper detection function. Therefore, one terminal is specified from any of the terminal devices 4 and 5 on the other party side that has a whisper detection function and is a listener. Then, when one terminal device that detects whispering is identified based on the above-described conditions, a whispering detection instruction signal for instructing to detect whispering is transmitted to the identified terminal device (S14). ).

  Subsequently, it is determined whether or not a whisper detection instruction signal has been received from another terminal device (S15). As described above, when the terminal device 3 is a speaker, since it is a side that transmits a whisper detection instruction signal and not a receiver (S15: NO), the process proceeds to the flow shown in FIG. It is determined whether or not a fired image has been received from the counterpart terminal device (S19). As described above, the whispered image is transmitted together with the terminal ID of the sender and the recording time of the whispered image. When the whispered image is received (S19: YES), the whispered image is in an encoded and compressed state, so that the whispered image data is decoded (S20). Then, the decoded image data that has been decoded and is in an uncompressed state, the terminal ID, and the recording time are registered in the distributed image data table 3131 stored in the HDD 31 (S21). In other words, since the whispered image data is stored in an uncompressed state, when the whispering signal is received, it can be displayed on the display 28 promptly.

  By the way, in the video conference system 1, a conference is performed by transmitting and receiving an image and sound between terminal devices. As for the image, as shown in FIG. 18, the image data of the recorded image is transmitted from the terminal device that recorded the image to the partner terminal device in the streaming method, and is buffer-played on the display 28. In the streaming method, it takes time to encode and decode. Accordingly, a delay occurs in the display on the display 28 of the terminal device on the playback side. For example, image data recorded at timing t0 is reproduced at timing t1 after a delay time P. Further, the image data recorded at the timing t1 is similarly reproduced at the timing t2 after the delay time P.

  Then, as shown in FIG. 16, in the terminal device 3, it is determined whether or not a whispering signal has been received from the terminal device that is the listener and on the recording side (S22). The amount of information of the whispering signal is much smaller than that of the image data. Therefore, the whispering signal transmitted from the terminal device specified as the terminal that detects whispering is promptly transmitted to the terminal device 3 that is a speaker via the network 2.

  Here, for example, when a whisper is detected at the timing t3 (see FIG. 18) in the terminal device on the recording side, in a time shorter than the delay time P, it is directed to the terminal device 3 that is a speaker and on the playback side. A whispering signal is sent. Then, the terminal device 3 on the reproduction side receives the whisper signal at the timing t4 (see FIG. 18). Next, it is determined whether or not the rolled image data corresponding to the terminal device that has transmitted the rolled signal is stored in the rolled image data table 3131 stored in the HDD 31 (S23).

  When it is determined that a whispered image is stored (S23: YES), at the timing t4 (see FIG. 18), based on the decoded whispered image data, the fired image is reproduced by interrupting the image being played on the display 28. (S25). At this time, in the terminal device on the recording side that distributes the streaming image, the streaming image is cut for the reproduction time Q of the rolled image. The CPU 20 that performs this cut processing corresponds to the “second streaming image cutting means” of the present invention.

  Further, at time t6 when the reproduction time Q of the whispered image has elapsed, the R time of the image data remaining in the buffer when the whispered image is interrupted is delayed and reproduced (S26). Then, normal streaming image buffer playback is performed from timing t7 when playback for R hours is completed (S27).

  If it is determined that a fired image is not stored in the fired image data table 3131 of the HDD 31 (S23: NO) even if the fired signal is received (S22: YES), an alternative image stored in advance in the HDD 31 is stored. It is displayed (S24). The substitute image may be, for example, a character, graphic, or the like that indicates to the speaker that the listener is speaking.

  Next, as shown in FIG. 17, it is determined whether an image call is in progress between the terminal devices (S28). When the image call is in progress (S28: YES), the image data of the camera image at the local site is encoded (S29), and the encoded image data is streamed to the other terminal device. (S30). Subsequently, it is determined whether or not all the connections with the terminal device have been disconnected (S31). If all the connections are disconnected (S31: YES), the process is terminated. If the connection still remains (S31: NO), the process returns to S6 in FIG. 14 to determine which terminal is connected. Then, the operation status of the terminal device in which the connection is maintained is rewritten, the operation status of the newly connected terminal device is stored, and the process is repeated in the same manner.

  Next, a case where a speaker is not detected by the terminal device 3 will be described. As shown in FIG. 14, when no speaker is detected at the base of the terminal device 3 (S12: NO), the terminal device 3 becomes a listener. Therefore, as shown in FIG. 15, it is determined whether or not a whisper detection instruction signal has been received from the partner terminal device that has become the speaker (S15). If a whisper detection instruction signal is received, whisper detection processing is executed (S16). In this whispering detection process, a person whispering from a camera image is detected according to the whirling detection method described above, and whispering is detected from the camera image. Then, a whisper signal is transmitted toward the other terminal device that becomes the speaker.

  Thereafter, as shown in FIG. 16, it is determined whether or not a whispered image has been received (S19). When the whispered image is received (S19: YES), the whispered image is in an encoded and compressed state, so that the whispered image data is decoded (S20). Then, the decoded image data that has been decoded and is in an uncompressed state, the terminal ID, and the recording time are registered in the distributed image data table 3131 stored in the HDD 31 (S21).

  Further, it is determined whether or not a whispering signal has been received (S22). At present, since the terminal device 3 is a listener who transmits a whisper signal (S22: NO), subsequently, as shown in FIG. 17, whether or not an image call is in progress between the terminal devices is determined. Determination is made (S28). When the image call is in progress (S28: YES), the image data of the camera image at the local site is encoded (S29), and the encoded image data is streamed to the other terminal device. (S30). Subsequently, it is determined whether or not all the connections with the terminal device have been disconnected (S31). If all the connections are disconnected (S31: YES), the process is terminated. If the connection still remains (S31: NO), the process returns to S6 in FIG. 14, the connection status with the terminal is determined, and the operation status of the connected terminal device is rewritten to the latest (S7 to 10). ), The process is repeated as described above.

  In the above description, the display 28 shown in FIGS. 2 and 7 corresponds to the “display means” of the present invention. The microphone 35 shown in FIG. 2 corresponds to the “speech detection means” of the present invention. The whirling image data table storage area 313 of the HDD 31 shown in FIG. 3 corresponds to the “reaction image storage means” of the present invention. The CPU 20 that executes the process of S8 shown in FIG. 14 corresponds to the “CPU load detection means” and the “CPU load status storage means” of the present invention. The CPU 20 that executes the process of S9 shown in FIG. 14 corresponds to the “transmission time detection means” and the “transmission time storage means” of the present invention. The CPU 20 that executes the process of S11 shown in FIG. 14 corresponds to the “reaction time image data transmitting means” of the present invention. The CPU 20 that executes the process of S13 shown in FIG. 14 corresponds to the “terminal specifying unit” of the present invention. The CPU 20 that executes the process of S14 shown in FIG. 14 corresponds to the “notification signal transmission means” of the present invention. The CPU 20 that executes the process of S15 shown in FIG. 14 corresponds to “notification signal receiving means” of the present invention. The CPU 20 that executes the process of S16 shown in FIG. 15 corresponds to the “reaction state detection means” of the present invention. The CPU 20 that executes the process of S17 shown in FIG. 15 corresponds to the “reaction signal transmission means” of the present invention. The CPU 20 that executes the process of S19 shown in FIG. 16 corresponds to the “reaction time image data receiving means” of the present invention. The CPU 20 that executes the processes of S20 and S21 shown in FIG. 16 corresponds to the “decompressing storage processing means” of the present invention. The CPU 20 that executes the process of S22 shown in FIG. 16 corresponds to the “reaction signal receiving means” of the present invention. The CPU 20 that executes the process of S23 shown in FIG. 16 corresponds to the “reaction-time image storage determination unit” of the present invention. The CPU 20 that executes the processing of S24 shown in FIG. 16 corresponds to “alternative image display control means” of the present invention. The CPU 20 that executes the process of S25 shown in FIG. 16 corresponds to the “reaction-time image display control means” of the present invention.

  As described above, the terminal device 3 according to the first embodiment is connected to the other terminal devices 4 to 6 via the network 2. A video conference system 1 that implements a remote conference by transmitting and receiving images and sounds between these terminal devices is configured. In this video conference system 1, during a remote conference, when it is detected that a listener is speaking in a specific terminal device (any one of the terminal devices 3 to 6), the terminal device (terminal) of the other party where the speaker is located Any device 3 to 6) is notified that the listener's whisper has been detected. The partner terminal device that is notified of the whispering displays a pre-stored listener whispering image. In other words, unlike the streaming method, this method does not require encoding and decoding of image data, so that the listener's whisper can be displayed without delay. Therefore, since the gap between the speaking timing and the listener's reaction can be reduced, a smooth conversation can be provided.

  Next, the terminal device 130 which is 2nd Embodiment is demonstrated with reference to drawings. In the first embodiment, in the communication control process by the CPU 20, the whispered image is recorded before the meeting, and the recording distribution is stopped for the time of the whispered image. On the other hand, the communication control processing of the CPU 120 in the second embodiment is different in that a whispered image is recorded during a meeting and a streaming image is cut by the playing time of the whispered image. Therefore, in order to focus on these different points, the communication control process by the CPU 120 that is different from the first embodiment will be mainly described. In addition, the terminal device 130 of 2nd Embodiment comprises the video conference system 1 shown in FIG. 1 similarly to the terminal device 3 of 1st Embodiment.

  First, the configuration of the terminal device 130 will be described with reference to FIG. FIG. 19 is a block diagram illustrating an electrical configuration of the terminal device 130. The terminal device 130 is provided with a CPU 120 as a controller that controls the terminal device 130. Connected to the CPU 120 are a ROM 121 that stores BIOS, a RAM 122 that temporarily stores various data, and an I / O interface 30 that mediates data transfer. The I / O interface 30 is connected to a hard disk drive 131 (hereinafter referred to as HDD 131) having various storage areas. The HDD 131 includes various storage areas (see FIG. 3) similar to the HDD 31 of the first embodiment. Other configurations are the same as the configuration of the terminal device 3 (see FIG. 2) of the first embodiment, and thus description thereof is omitted.

  Next, communication control processing by the CPU 120 will be described with reference to the flowcharts of FIGS. 20 to 23 and FIG. FIG. 20 is a flowchart of communication control processing by the CPU 120. FIG. 21 is a flowchart showing a continuation of FIG. FIG. 22 is a flowchart showing a continuation of FIG. FIG. 23 is a flowchart showing a continuation of FIG. FIG. 24 is a timing chart for explaining the image cut processing at the time of detection of a whirl.

  As shown in FIG. 20, first, various data are initialized (S40). Subsequently, it is determined whether or not it is connected to at least one of other terminal devices via the network 2 (S41). Until it is connected to any other terminal device (S41: NO), the process returns to S41 and the process is repeated. When connected to another terminal device (S41: YES), the terminal status is stored in the terminal status table storage area 312 of the HDD 31 from the terminal information transmitted from the connected other terminal device. It is stored in the table 3121 (see FIG. 5) (S42). Further, the load on the CPU 120 of the terminal device 130 is measured and stored in the terminal status table 3121 (see FIG. 5) (S43). Further, the data transmission time delay of the terminal device 130 is measured and stored in the terminal status table 3121 (see FIG. 5) (S44).

  Further, the presence / absence of the terminal device detection function is stored in the terminal status table 3121 (see FIG. 5) (S45). When the terminal device has a whirl detection function, “present” is stored in the whirl detection function column 65 of the terminal status table 3121. More specifically, when the presence / absence of the presence / absence detection function is registered, the presence / absence function flag “1” is stored. In the case of “None”, the rolling function flag “0” is stored.

  Next, it is determined whether or not a speaker has been detected (S47). When the user's voice is detected from the microphone 35, it is determined as a speaker, and when the voice is not detected, it is determined as a listener. Here, since subsequent processing changes depending on whether the speaker is a speaker or a listener, the following description will be made separately for a case where a speaker is detected and a case where a speaker is detected without being detected.

  First, a case where a speaker is detected will be described. As shown in FIG. 19, when a speaker is detected at the base of the terminal device (S47: YES), a terminal device that detects whispering is specified (S48). Here, one terminal device that detects whispering is specified based on conditions determined based on the operating status of each terminal device. In addition, the conditions for specifying the terminal are any of the first to fourth conditions described above, as in the first embodiment.

  Similarly in the second embodiment, the terminal device 130 can select any one of the first to fourth conditions. Based on the condition selected by the user, the CPU 120 identifies one terminal device that detects whispering. It should be noted that when one terminal device that detects whispering is specified, those that do not have the whisper detection function are excluded. Then, when one terminal device that detects whispering is identified based on the above-described conditions, a whispering detection instruction signal for instructing to detect whispering is transmitted to the identified terminal device (S49). ).

  Subsequently, it is determined whether or not a whisper detection instruction signal is received from another terminal device (S50). If the terminal device 130 is a speaker, the whisper detection signal is on the transmitting side and not on the receiving side (S50: NO), the process proceeds to the flow shown in FIG. It is determined whether or not a whispered image has been received (S57). As described above, the whispered image is transmitted together with the terminal ID of the sender and the recording time of the whispered image. When the whispered image is received (S57: YES), the whispered image is in an encoded and compressed state, so that the whispered image data is decoded (S58). Then, the decoded image data that has been decoded and is in an uncompressed state, the terminal ID, and the recording time are registered in the distributed image data table 3131 stored in the HDD 31 (S59).

  By the way, also in 2nd Embodiment, a meeting is performed by transmitting and receiving an image and an audio | voice between terminal devices. As for the image, as shown in FIG. 24, the image data of the recorded image is transmitted from the terminal device that recorded the image to the partner terminal device in a streaming manner, and is buffer-reproduced on the display 28. In the streaming method, it takes time to encode and decode. Accordingly, a delay occurs in the display on the display 28 of the terminal device on the playback side. For example, image data recorded at the r0 timing is reproduced at the r1 timing after the delay time P. Furthermore, the image data recorded at the r1 timing is also reproduced at the r2 timing after the delay time P.

  Then, as shown in FIG. 22, in the terminal device 130, it is determined whether or not a whispering signal has been received from the terminal device that is the listener and on the recording side (S60). The amount of information of the whispering signal is much smaller than that of the image data. Therefore, the whispering signal transmitted from the terminal device identified as the terminal that detects whispering is promptly notified to the terminal device 130 that is the speaker via the network 2.

  Here, when a whisper is detected at the r3 timing (see FIG. 24) in the terminal device on the recording side, a whisper signal is sent to the terminal device that is the speaker and on the playback side in a time shorter than the delay time P. Is sent. Then, the playing signal is received at the r4 timing (see FIG. 24) in the terminal device on the playback side. Next, it is determined whether or not a rolled image corresponding to the terminal device that has transmitted the rolled signal is stored in the rolled image data table 3131 (see FIG. 6) stored in the HDD 131 (S61).

  When it is determined that a whispered image is stored (S61: YES), at the r4 timing (see FIG. 24), based on the decoded whispered image data, the display 28 interrupts and plays the image being played (S63). ). Further, for streaming delivery from the terminal device on the recording side, the reproduction time (Q) of the reproduced rolled image is cut (S64). In other words, the time T1 of the streamed image and the time T2 of the image remaining in the buffer when interrupted are cut. Further, at the r5 timing when the reproduction time Q of the whispered image has passed, the time T2 of the image remaining in the buffer when the whispered image is interrupted is delayed and reproduced. Then, from the r6 timing when the reproduction for T2 hours ends, normal streaming image buffer reproduction is performed (S65).

  If it is determined that a fired image is not stored in the fired image data table 3131 of the HDD 31 (S61: NO) even if the fired signal is received (S60: YES), an alternative image stored in advance in the HDD 31 is stored. It is displayed (S62). The substitute image may be any image that indicates to the speaker that he / she is scolding with characters, graphics, or the like.

  Next, as shown in FIG. 23, it is determined whether an image call is in progress between the terminal devices (S66). When the image call is in progress (S66: YES), the image data of the camera image is encoded at the local site (S67), and the encoded image data is streamed to the partner terminal device. (S68). Subsequently, it is determined whether or not all connections with other terminal devices have been disconnected (S69). If all the connections are disconnected (S69: YES), the process is terminated. If the connection still remains (S69: NO), the process returns to S41 in FIG. 20, the operation status of the connected terminal device is rewritten to the latest one, and the process is repeated.

  Next, a case where a speaker is not detected by the terminal device 130 will be described. As shown in FIG. 20, when no speaker is detected at the base of the terminal device 130 (S47: NO), the terminal device 130 becomes a listener. Accordingly, as shown in FIG. 21, it is determined whether or not a whisper detection instruction signal has been received from the terminal device on the other side who has become the speaker (S50). When a whisper detection instruction signal is received, whisper detection processing is executed (S51). In this whispering detection process, a person whispering from a camera image is detected according to the whirling detection method described above, and whispering is detected from the camera image. Then, a whisper signal is transmitted toward the other terminal device that becomes the speaker.

  Subsequently, it is determined whether or not the first detection has been performed (S52). In the case of the first detection (S52: YES), the terminal device on the speaker side does not store the whispered image data of the terminal device 130 that is the listener. Therefore, the whispered image of the user who detected the whisper is encoded (S53) and stored in the HDD 131 (S54). Further, the encoded sooted image data is transmitted to another terminal device connected via the network 2 (S55). When the whispering is not detected for the first time (S52: NO), the whispering signal is transmitted toward the terminal device that is the speaker.

  Thereafter, it is determined whether or not a whispered image has been received (S57). When the whispered image is received (S57: YES), the whispered image is in an encoded and compressed state, so that the whispered image data is decoded (S58). Then, the decoded image data that has been decoded and is in an uncompressed state, the terminal ID, and the recording time are registered in the distributed image data table 3131 stored in the HDD 131 (S59).

  Further, it is determined whether or not a whispering signal has been received (S60). Since the terminal device 130 is a listener who transmits a whispering signal (S60: NO), whether or not an image call is currently in progress between the terminal devices as shown in FIG. Determination is made (S66). When the image call is in progress (S66: YES), the image data of the camera image is encoded at the local site (S67), and the encoded image data is streamed to the partner terminal device. (S68). Subsequently, it is determined whether or not all the connections with the terminal device have been disconnected (S69). If all the connections are disconnected (S69: YES), the process is terminated. If the connection still remains (S69: NO), the process returns to S41 in FIG. 19, returns to S41 in FIG. 20, the connection status with the terminal is determined, and the operation status of the connected terminal device is the latest. (S42-45), and the process is repeated in the same manner as described above.

  As described above, in the terminal device 130 according to the second embodiment, in the communication control process of the CPU 120, a whispered image can be recorded during the conference, so that there is no need for troubles such as preparation for recording before the conference. Furthermore, when playing the whispered image, the streaming image is cut for the playing time of the whispered image. In this case, in the terminal device on the streaming delivery side, processing for temporarily stopping streaming is not required as in the first embodiment.

  In the above description, the CPU 20 that executes the process of S52 shown in FIG. 21 corresponds to the “initial reaction state detecting means” of the present invention. The CPU 20 that executes the processes of S53 and S54 shown in FIG. 21 corresponds to the “reaction image storage processing means” of the present invention. The CPU 20 that executes the process of S63 shown in FIG. 22 corresponds to the “interrupt display means” of the present invention. The CPU 20 that executes the process of S64 shown in FIG. 22 corresponds to the “first streaming image cutting means” of the present invention.

  Next, a terminal device 230 according to a third embodiment of the present invention will be described with reference to FIG. FIG. 25 is a block diagram illustrating an electrical configuration of the terminal device 230. In the first and second embodiments, a user's whisper is detected from a camera image using image processing. In the third embodiment, when the listener is satisfied with the speaker's story, the whisper button 70 provided on the terminal device 230 is pressed.

  The configuration of the terminal device 230 will be described. As illustrated in FIG. 25, the terminal device 230 is provided with a CPU 220 as a controller that controls the terminal device 230. Connected to the CPU 220 are a ROM 221 that stores BIOS and the like, a RAM 222 that temporarily stores various data, and an I / O interface 30 that mediates data transfer. A hard disk drive 231 having various storage areas is connected to the I / O interface 30. A whirl button 70 is connected to the I / O interface 30 while having the same configuration as the terminal device 3 (see FIG. 2) of the first embodiment.

  The whisper button 70 is pressed when the listener is satisfied with the speaker's story. Therefore, when the whisper button 70 is pressed, the whisper signal similar to that of the first embodiment is transmitted to the terminal device that is the speaker. That is, the communication control process by the CPU 220 is performed when the whisper button 70 is selected in the whirl detection process (FIG. 15: S16) in the case of being specified as one terminal device that detects whisper in the communication control process of the first embodiment. The difference is that it is determined whether or not the button has been pressed. When the whispering button 70 is pressed, a whispering signal is transmitted (FIG. 15: S17), so that the same effect as in the first embodiment can be obtained.

  As described above, in the terminal device 230 according to the third embodiment, when the listener is satisfied with the speaker's story, the whisper button 70 provided on the terminal device 230 is pressed. Compared with image processing, it is possible to reliably convey the will that the listener is satisfied to the speaker. Note that the S button 70 shown in FIG. 25 corresponds to the “operation means” of the present invention.

  The present invention is not limited to the first to third embodiments described above, and various modifications can be made. For example, in the above-described embodiment, the whirling of the user's head is detected, but the reaction state of denying the speaker by shaking his / her head left and right is detected by the image processing described above. Is also possible. In this case, as in the case of the whispered image, if the image of shaking the head is memorized, it is possible to quickly inform the speaker of the intention of denying the listener.

  In addition, in order to convey a person's own feelings to the other person, there are cases where not only the motion of shaking the head but also various gestures are used. By detecting the characteristics of the listener's reaction state, various reactions can be detected, and the present invention can be applied.

1 is a block diagram showing a configuration of a video conference system 1. FIG. 3 is a block diagram showing an electrical configuration of a terminal device 3. FIG. 3 is a conceptual diagram showing various storage areas of an HDD 31. FIG. 3 is a conceptual diagram of a login table 3111. FIG. It is a conceptual diagram of the terminal status table 3121. It is a conceptual diagram of the roaring image data table 3131. It is a figure which shows one display mode in the display. It is explanatory drawing (before whispering) of the feature-value d which shows the amount of depression. It is explanatory drawing (after a whisper) of the feature-value d which shows depressing adjustment. 3 is a conceptual diagram of camera image data 40. FIG. It is a graph which shows a detection waveform pattern (at the time of whispering). It is a graph which shows the registered whispering waveform pattern. It is a flowchart of the communication control process by CPU20. It is a flowchart which shows the continuation of FIG. It is a flowchart which shows the continuation of FIG. It is a flowchart which shows the continuation of FIG. It is a flowchart which shows the continuation of FIG. It is a timing chart for demonstrating the image cut process at the time of a whirling detection. It is a block diagram which shows the electric constitution of the terminal device 130 which is 2nd Embodiment. It is a flowchart of the communication control process by CPU120. It is a flowchart which shows the continuation of FIG. It is a flowchart which shows the continuation of FIG. It is a flowchart which shows the continuation of FIG. It is a timing chart for demonstrating the image cut process at the time of a whirling detection. It is a block diagram which shows the electric constitution of the terminal device 230 which is 3rd Embodiment.

Explanation of symbols

1 Video conference system 2 Network 3 Terminal device 20 CPU
28 Display 31 Hard Disk Drive 34 Camera 35 Microphone 70 Sparkling Button 120 CPU
130 Terminal device 220 CPU
230 Terminal device 311 Login table storage area 312 Terminal status table storage area 313 Fired image data table storage area 314 Camera image data storage area

Claims (16)

A communication terminal device that communicates with a counterpart terminal via an image via a network,
Display means for displaying the image data transmitted from the counterpart terminal;
Reaction state detection means for detecting the reaction state of the user;
When the reaction state is detected by the reaction state detection unit, a reaction signal transmission unit that transmits a reaction signal indicating that the reaction state is detected to the user to the counterpart terminal;
Reaction signal receiving means for receiving the reaction signal transmitted by the reaction signal transmitting means;
A reaction-time image storage means for storing a reaction-time image when the partner user of the counterpart terminal indicates the reaction state;
A reaction time image display control means for displaying, on the display means, the reaction time image of the counterpart user stored in the reaction time image storage means when the reaction signal is received by the reaction signal receiving means; A communication terminal device comprising the communication terminal device. The display means displays the streaming image of the partner user transmitted from the partner terminal and photographed by the partner terminal,
Interrupt display means for interrupting the streaming image displayed on the display means and displaying the reaction time image when the reaction signal is received by the reaction signal receiving means,
And a first streaming image cut means for cutting the streaming image for a time corresponding to the time of the reaction image from when the reaction image is interrupted by the interrupt display means. The communication terminal device according to claim 1.   The streaming image of the user to be transmitted to the counterpart terminal is cut by a time corresponding to the time of the response image data transmitted by the response image data transmission means, and transmitted to the counterpart terminal. The communication terminal apparatus according to claim 1, further comprising: 2 streaming image cut means. Reaction time image data transmitting means for compressing the reaction time image data when the reaction state of the user is detected by the reaction state detection means, and transmitting the compressed image data to the counterpart terminal;
Reaction image data receiving means for receiving the reaction image data transmitted by the reaction image data transmitting means of the counterpart terminal;
4. A decompression storage processing means for decompressing the reaction image data received by the reaction image data receiving means and storing the decompressed image data in the reaction image storage means. The communication terminal device according to claim 1. One terminal that detects the reaction state of the counterpart user based on a predetermined condition from the plurality of counterpart terminals when connected to the counterpart terminals via the network A terminal identification means for identifying
To the one terminal specified by the terminal specifying means, a notification signal transmitting means for transmitting a notification signal notifying that it has been specified as a terminal for detecting the reaction state;
Notification signal receiving means for receiving the notification signal transmitted from the counterpart terminal,
The communication terminal according to any one of claims 1 to 4, wherein the reaction state detection unit detects the reaction state of the user when the notification signal is received by the notification signal reception unit. apparatus. CPU load status detecting means for detecting the CPU load status of the counterpart terminal via the network;
CPU load status storage means for storing the CPU load status detected for each counterpart terminal by the CPU load status detection means,
The terminal specifying means includes
A first predetermined condition is provided for referring to the CPU load status stored in the CPU load status storage means to identify the counterpart terminal with the lowest CPU load as the one terminal. The communication terminal device according to claim 5. A transmission time detecting means for detecting a transmission time of data in communication with the counterpart terminal via the network;
Transmission time storage means for storing the detection result by the transmission time detection means,
The terminal specifying means includes
6. The second predetermined condition for specifying the counterpart terminal with the shortest transmission time as the one terminal with reference to the detection result stored in the transmission time storage means. The communication terminal device according to 1. For each counterpart terminal, a login number detection means for detecting the number of logged-in counterpart users,
Login number storage means for storing the number of logins detected by the login number detection means,
The terminal specifying means includes
A third predetermined condition for specifying, as the one terminal, the partner terminal having the largest number of logins among the number of logins for each partner terminal stored in the login number storage unit is provided. The communication terminal device according to claim 5. Utterance detection means for detecting the utterance of the other user;
Elapsed time measuring means for measuring the elapsed time from the time of utterance for each partner user detected by the utterance detecting means,
The terminal specifying means includes
6. The fourth predetermined condition for specifying, as the one terminal, the counterpart terminal of the counterpart user with the shortest elapsed time measured by the elapsed time measuring means. Communication terminal device. The reaction state detecting means includes
10. The communication terminal device according to claim 1, wherein a state in which the user's head shakes in a predetermined direction is detected as the reaction state. 11. The reaction state detecting means includes
The communication terminal device according to claim 1, wherein a rejection state in which the user's head shakes and rejects in a horizontal direction is detected as the reaction state. Comprising operating means operated by the user when the user is convinced,
The reaction state detecting means includes
The communication terminal device according to claim 1, wherein the user's satisfaction state is detected as the reaction state by detecting an operation performed by the operation unit. Initial reaction state determination means for determining whether or not the reaction state of the user detected by the reaction state detection means is the first time;
A reaction time image storage processing means for storing the reaction time image in the reaction time image storage means when the reaction state is determined to be the first time by the initial reaction state determination means;
13. The reaction image data transmitting unit transmits the reaction image data stored in the reaction image storage unit to the counterpart terminal in a compressed state. The communication terminal device described. A reaction time image storage judging means for judging whether or not the reaction time image data of the counterpart user is stored in the reaction time image storage means when the reaction signal is received by the reaction signal receiving means; ,
When it is determined by the response image storage determination means that the response image is not stored, the display means that the other user indicates the response status instead of the response image. 14. The communication terminal apparatus according to claim 1, further comprising: a substitute image display control unit that displays a substitute image represented by characters, figures, symbols, and the like. A communication control method for a communication terminal device that performs image communication with a counterpart terminal via a network,
An image data receiving step of receiving image data transmitted from the counterpart terminal;
A reaction state detection step for detecting a reaction state of the user;
When the reaction state is detected in the reaction state detection step, a reaction signal transmission step of transmitting a reaction signal indicating that the reaction state is detected to the user to the counterpart terminal;
A reaction signal receiving step for receiving the reaction signal transmitted in the reaction signal transmitting step;
When the reaction signal is received in the reaction signal reception step, the counterpart user stored in the reaction-time image storage means for storing a response-time image when the counterpart user of the counterpart terminal indicates the reaction state And a reaction-time image display control step of displaying the reaction-time image on the display means for displaying the image data received in the image data reception step.   A communication control program for causing a computer to execute various processing steps of the communication control method according to claim 15.

Images