JP2005328484A - Video conference system, information processing apparatus, information processing method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video conference system having communication properties and overflowing with presence. <P>SOLUTION: The video conference system, comprising one or more terminal devices installed at different spots and an information processing apparatus connected with the terminal devices via a predetermined network, is provided with a video compositing means for creating a composited video image by compositing a video image of a user, based on the video data transmitted from a terminal device via the network and an image of a required document; a control means for controlling the video compositing means to enlarge or reduce the image of the document or the corresponding video image of the user in the composited video image, based on document operation using a document operating means and/or the presence/absence of the user's voice detected based on the audio data; and a transmission means for transmitting to terminal devices via the network composited video data comprising video data of the composited video image created by the video compositing means. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a video conference system, an information processing apparatus, an information processing method, and a program, and is suitable for application to a video conference with, for example, a plurality of remote users at remote locations.

  Conventionally, video conferencing terminals installed at a plurality of points are connected via public lines such as ISDN (Integrated Services Digital Network) or dedicated lines, and information such as video data and audio data is transmitted and received mutually. There is a video conference system that can perform a conference with a plurality of other users on the ground.

  In such a video conference system, images of a plurality of other users at the video conference destination are displayed on the display. However, if the other user's remarks and facial expressions at that time can be recognized, more communicability can be achieved. It is considered that a high meeting can be held.

However, in such a video conference system, when a plurality of users meet to conduct a video conference, there is a problem that it is difficult to recognize who is speaking. In order to solve such a problem, a method for identifying a user who detects a speaker by detecting the speaker based on the direction of the voice and adding a cursor to the head of the user's video displayed on the display. Has been proposed (see, for example, Patent Document 1).
JP 2003-189273 A

  By the way, in recent video conference systems, a personal computer installed at a plurality of points is used, for example, a remote desktop function of Windows XP (Windows (registered trademark) XP) or a remote access system such as VNC (Virtual Network Computing). By connecting via a network such as the Internet line, a video conference in a multi-window / multi-display environment using a personal computer can be performed.

  At this time, when a free discussion is performed with the other party user of the video conference destination, the video of the other party user is enlarged and displayed so that the state of the utterance and the facial expression at that time can be easily recognized. When the other user explains the contents of the document, it is considered that if the corresponding part of the document is enlarged and displayed, a teleconference with high communication and high presence can be performed.

  However, in such a case, it is necessary to perform complicated operations such as enlarging or reducing the other user's video and document video, so that the user's consciousness concentrates on the original meeting. Due to the inconvenience of operation and the operation of enlargement / reduction of the other user's image and document image depending on the situation, the presence of reality is lacking and communication is lacking There is a problem of doing.

  The present invention has been made in consideration of the above points, and proposes a video conference system, an information processing apparatus, an information processing method, and a program having high communication performance and full of realism.

  In order to solve such a problem, in the present invention, in a video conference system comprising one or more terminal devices installed at different points and an information processing device connected to the terminal device via a predetermined network, The terminal device includes a photographing unit for photographing the user, a sound collecting unit for collecting the user's voice, a document operation unit for performing a predetermined document operation on the document, video data output from the photographing unit, and video data collection. A transmission unit configured to transmit audio data output from the sound unit to the information processing device via the network; and a display unit configured to display a video based on the composite video data transmitted from the information processing device via the network. The processing device includes the user's video based on the video data transmitted from the terminal device via the network and the necessary data. Based on the presence or absence of a user's speech detected based on the document operation using the document operation means and / or the audio data And / or control means for controlling the synthesized video means so as to enlarge or reduce the corresponding user's video, and synthesized video data composed of synthesized video data generated by the video synthesizing means to the terminal device via the network. A transmission means for transmitting is provided.

  As a result, the document image and the user video of the user who performed the operation are mainly captured when the document is operated, and the user video of the user who performed the speech is main when the user speaks without causing the user to perform an extra operation. Can be captured.

  In order to solve such a problem, in the present invention, in one or a plurality of terminal devices installed at different points and an information processing device connected to the terminal device via a predetermined network, a network is connected from the terminal device. Video synthesizing means for synthesizing the user's video based on the photographed user's video data transmitted via the network and a required document image, and a predetermined document for the document transmitted via the network from the terminal device The image of the document in the synthesized video and / or the corresponding video of the user is enlarged or reduced based on the presence or absence of the user's speech detected based on the user's document operation and / or the voice data of the collected user's voice Control means for controlling the video composition means and composition generated by the video composition means The synthetic image data composed of video data of the image, and to provide a transmission means for transmitting to the terminal device via the network.

  As a result, the document image and the user video of the user who performed the operation are mainly captured when the document is operated, and the user video of the user who performed the speech is main when the user speaks without causing the user to perform an extra operation. Can be captured.

  Further, in order to solve such a problem, in the present invention, in an information processing method in one or a plurality of terminal devices installed at different points and an information processing device connected to the terminal device via a predetermined network, A first step of generating a synthesized video obtained by synthesizing a user's video and a necessary document image based on the captured video data of the user transmitted from the terminal device via the network; and transmitting from the terminal device via the network The document image in the synthesized video and / or the corresponding user video is enlarged based on the presence or absence of the user's speech detected based on the predetermined document operation and / or the voice data of the collected user's voice Or a second step to reduce and composition composed of composite video data Image data, and to provide a third step of transmitting to the terminal device via the network.

  As a result, the user image of the user who operated the document image is mainly captured at the time of the document operation without causing the user to perform an extra operation, while the user image of the user who performed the statement is the main image when the user speaks. Can be captured.

  Furthermore, in order to solve such a problem, in the present invention, a terminal is provided for one or a plurality of terminal devices installed at different points in a program and an information processing device connected to the terminal device via a predetermined network. A first step of generating a synthesized video obtained by synthesizing a user's video and a necessary document image based on the captured video data of the user transmitted from the device via the network, and transmitted from the terminal device via the network The document image in the synthesized video and / or the corresponding user video is enlarged or reduced based on the presence or absence of the user's speech detected based on the predetermined document operation and / or the voice data of the collected user's voice. The second step of reduction and the composite video data composed of the video data of the composite video The data were so as to execute a third step of transmitting to the terminal device via the network.

  As a result, the user image of the user who operated the document image is mainly captured at the time of the document operation without causing the user to perform an extra operation, while the user image of the user who performed the statement is the main image when the user speaks. Can be captured.

  As described above, according to the present invention, in a video conference system including one or a plurality of terminal devices installed at different points and an information processing device connected to the terminal device via a predetermined network, the terminal device A photographing means for photographing the user, a sound collecting means for collecting the user's voice, a document operating means for performing a predetermined document operation on the document, video data output from the photographing means, and a sound collecting means An information processing apparatus comprising: transmission means for transmitting audio data output from the information processing apparatus via the network; and display means for displaying video based on the composite video data transmitted from the information processing apparatus via the network. The user's video based on the video data transmitted from the terminal device via the network and the necessary document The image of the document in the synthesized video and / or the correspondence based on the presence or absence of the user's speech detected based on the document operation using the document operation unit and the above-described audio data Transmitting the control means for controlling the composite video means so as to enlarge or reduce the user's video, and the composite video data composed of the video data of the composite video generated by the video composite means to the terminal device via the network By providing the means, the user can mainly capture the document image and the user video of the user who performed the operation during the document operation without causing the user to perform an extra operation. Can capture the user image of the user, and thus Myunikeshon resistance is high, it is possible to realize a immersive video conferencing system.

  Further, as described above, according to the present invention, in one or a plurality of terminal devices installed at different points and an information processing device connected to the terminal device via a predetermined network, from the terminal device via the network A video synthesizing unit that generates a synthesized video by synthesizing a user's video and a necessary document image based on the captured user's video data to be transmitted, and a predetermined document for the document transmitted from the terminal device via the network Based on the presence or absence of the user's speech detected based on the voice data of the user's voice that has been operated and / or collected, synthesized video synthesis so as to enlarge or reduce the document image and / or the corresponding user video in the synthesized video Control means for controlling the means, and video data of the composite video generated by the video composition means By providing a transmission means for transmitting the composite video data to the terminal device via the network, the document image and the user video of the user who performed the operation at the time of the document operation are performed without causing an extra user operation. The user image of the user who made the utterance can be captured mainly when the user utters, thus realizing a highly communicative and realistic information processing apparatus.

  Furthermore, as described above, according to the present invention, in the information processing method in one or a plurality of terminal devices installed at different points and the information processing device connected to the terminal device via a predetermined network, the terminal device A first step of generating a synthesized video obtained by synthesizing a user's video and a necessary document image based on the captured user's video data transmitted via the network from the terminal device, and transmitted from the terminal device via the network Enlarge or reduce a document image and / or a corresponding user video in the synthesized video based on a predetermined document operation on the document and / or presence or absence of a user's speech detected based on voice data of the collected user's voice The second step of generating the synthesized video data comprising the synthesized video data, The third step of transmitting to the terminal device via the network is provided, so that the document image and the user image of the user who has performed the operation are mainly displayed during the document operation without causing the user to perform an extra operation. On the other hand, when the user speaks, the user's video of the user who made the speech can be captured mainly, thus realizing an information processing method that is highly communicative and full of realism.

  Further, as described above, according to the present invention, one or a plurality of terminal devices installed at different points in the program and an information processing device connected to the terminal device via a predetermined network are transmitted from the terminal device. A first step of generating a synthesized video obtained by synthesizing a user's video and a necessary document image based on the captured user's video data transmitted via the network, and a document transmitted from the terminal device via the network The document image and / or the corresponding user video in the synthesized video is enlarged or reduced based on the presence or absence of the user's speech detected based on the predetermined document operation and / or the voice data of the collected user's voice In the second step, the synthesized video data composed of the synthesized video data is transferred to the network. The third step of transmitting to the terminal device via the network is executed, so that the document image and the user video of the user who performed the operation at the time of the document operation can be displayed without causing the user to perform an extra operation. On the other hand, when the user speaks, the user's video of the user who made the speech can be captured mainly, and thus a program with high communication and full of realism can be realized.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

(1) Configuration of Video Conference System According to the Present Embodiment In FIG. 1, reference numeral 1 denotes the video conference system according to the present embodiment as a whole, and a plurality of video conference terminals 2 (2A, 2B) installed at different points, respectively. And the video conference server 3 are connected to each other via a network NT such as an Internet line.

  In this case, as shown in FIG. 2, each video conference terminal 2 is provided with a camera unit 4 made of, for example, a CCD (Charge Coupled Device), a microphone 5, and a pointing device 6 made of a mouse or the like. .

  The camera unit 4 captures an image of the user 7 (7A, 7B) who uses the video conference terminal 2, and sends the video signal S1 thus obtained to the video processing unit 8. The video processing unit 8 performs predetermined signal processing such as analog / digital conversion processing on the supplied video signal S1, and sends the obtained video data D1 to the video compression unit 9. Further, the video compression unit 9 compresses the supplied video data D1 by a predetermined compression method according to a predetermined standard such as H.323 standardized by the International Telecommunication Union (ITU). The obtained compressed video data D2 is sent to the packetizing unit 10.

  In addition, the microphone 5 collects the voice when the user 7 speaks and sends the obtained voice signal S2 to the input voice processing unit 11. The input audio processing unit 11 performs predetermined signal processing such as analog / digital conversion processing on the supplied audio signal S2, and sends the obtained audio data D3 to the audio compression unit 12. Further, the audio compression unit 12 performs compression processing of the compression method based on a predetermined standard such as H.323 similar to the above on the supplied audio data D3, and the obtained compressed audio data D4 is packetized. To send.

  Further, the pointing device 6 sends an operation information signal S3 representing the content of the user operation to the situation determination unit 13, while the situation determination unit 13 determines the current cursor position and the like determined based on the operation information signal S3. The situation determination information is sent to the packetizing unit 10 as the situation determination data D5.

  The packetizing unit 10 packetizes the compressed video data D2 supplied from the video compression unit 9, the compressed audio data D4 supplied from the audio compression unit 12, and the operation data D5 supplied from the situation determination unit 13 in a predetermined format. The received packet data D6 is sent to the network control unit 14.

  Then, the network control unit 14 performs predetermined modulation processing such as intermediate frequency modulation on the supplied packet data D6, and transmits the transmission signal S4 thus obtained to the video conference server 3 via the network NT.

  The video conference server 3 has a configuration as shown in FIG. 3, and the network control unit 16 receives a transmission signal S 4 transmitted from each video conference terminal 2. Then, the network control unit 16 performs predetermined demodulation processing such as detection processing on the transmission signal S4 from each of the video conference terminals 2, and sends the obtained packet data D7 to the depacketizing unit 16.

  The depacketizing unit 16 extracts the compressed video data D8, the compressed audio data D9, and the operation data D10 included in the packet from the supplied packet data D7, and sends the compressed video data D8 to the video decompressing unit 17 and the compressed audio data. D9 is sent to the voice decompression unit 18, and the situation judgment data D10 is sent to the situation judgment control unit 19.

  The video decompression unit 17 performs decompression processing by the corresponding decoding method on the supplied compressed video data D8, and sends the obtained baseband video data D11 to the video composition unit 20.

  At this time, similarly to the video conference terminal 2, the video conference server 3 is provided with a camera unit 21 made of a CCD camera or the like, a microphone 22, and a pointing device 23 made of a mouse or the like.

  Then, the camera unit 21 sends the video signal S5 obtained by imaging the user 7 (7C) who uses the video conference server 3 to the video processing unit 25. The video processing unit 25 performs predetermined signal processing such as analog / digital conversion processing on the supplied video signal S5, and sends the obtained video data D12 to the video synthesis unit 20.

  The pointing device 23 also sends an operation information signal S6 representing the content of the user operation to the situation determination control unit 19. Then, the situation determination control unit 19 is associated with each video conference terminal 2 and the video conference server 3 based on the operation information signal S6 and the situation judgment data D10 transmitted from each video conference terminal 2, respectively. Image data (hereinafter referred to as “cursor image data”) D13 of each cursor that moves in response to the operation of the pointing devices 6 and 23 of the video conference terminal 2 or the video conference server 3 is generated, and this is generated as a video composition unit. 20 to send.

  Further, the video conference server 3 is provided with a document storage unit 26 in which image data of a document transmitted in advance from the video conference terminal 2 or input by the user 7C of the video conference server 3 is stored. The situation determination control unit 19 is stored in the document storage unit 26 based on the operation information signal S6 supplied from the pointing device 23 of the video conference server 3 and the situation determination data D10 transmitted from each video conference terminal 2. When a document display command is given, the document storage unit 26 is controlled to read the document data. Then, the document data D14 of this document is given to the video composition unit 20.

  Under the control of the situation determination control unit 19, the video composition unit 20 has each user video 27 (27 </ b> A, 27 </ b> B) based on the video data D <b> 11 from each video conference terminal 2 supplied from the video expansion unit 17. User video 27 (27C) based on the video data D12 supplied from the video processing unit 26, cursor images 29 (29A to 29C) based on the cursor image data D13 supplied from the situation determination control unit 19, and the document storage unit 26 For example, a composite video 32 as shown in FIG. 4 is generated by combining the document image 31 based on the document data D14 supplied from the video data, and the composite video data D15 composed of the video data is generated in the display processing unit 33 and the video compression unit 34. Send it out.

  Then, the display processing unit 33 performs predetermined signal processing such as digital / analog conversion processing on the synthesized video data D15, and the obtained video signal S7 is displayed on a display 35 made of, for example, a CRT (Cathode-Ray Tube). Send it out. Thereby, the composite image 32 as described above based on the video signal S7 is displayed on the display 35.

  In addition, the video compression unit 34 performs a predetermined compression process on the synthesized video data D15 supplied from the video synthesis unit 20, and sends the obtained compressed synthesized video data D16 to the packetizing unit 36.

  On the other hand, the voice decompression unit 18 performs a decoding process using a corresponding decoding method on the compressed voice data D9 supplied from the depacketizing unit 16, and sends the obtained baseband voice data D17 to the voice synthesis unit 37. Send it out.

  At this time, the voice synthesizer 36 has a baseband obtained by subjecting the voice signal S8 output from the microphone 22 of the video conference server 3 to predetermined signal processing such as analog / digital conversion in the voice processor 38. Audio data D18 is provided.

  Thus, the voice synthesizer 37 is based on each voice based on the voice data D17 from each video conference terminal 2 (2A and 2B) supplied from the voice expansion unit 18 and on the voice data D18 supplied from the voice processing unit 38. A synthesized voice synthesized with the voice is generated, and synthesized voice data D19 composed of the voice data is sent to the output voice processor 39 and the voice compressor 40.

  The output sound processing unit 39 performs predetermined signal processing such as digital / analog conversion processing on the synthesized sound data D19, and outputs the obtained sound signal S9 to the speaker 41. As a result, the above synthesized voice based on the voice signal S9 is output from the speaker 41.

  The voice compression unit 40 performs a predetermined compression process on the synthesized voice data D19 supplied from the voice synthesis unit 37, and sends the obtained compressed synthesized voice data D20 to the packetizing unit 36.

  The packetizing unit 36 packetizes the compressed synthesized video data D16 supplied from the video compressing unit 34 and the compressed synthesized audio data D20 supplied from the audio compressing unit 40 in a predetermined format, and the obtained packet data D21 is network control unit 15 To send.

  Then, the network control unit 15 performs predetermined modulation processing such as intermediate frequency modulation on the supplied packet data D21, and transmits the transmission signal S10 thus obtained to each video conference terminal 2 via the network NT. .

  In each video conference terminal 2, the network control unit 14 receives the transmission signal S <b> 10 transmitted from the video conference server 3. Then, the network control unit 14 performs predetermined demodulation processing such as detection processing on the transmission signal S10 from the video conference server 3, and sends the obtained packet data D22 to the depacketizing unit 42.

  The depacketizing unit 42 extracts the compressed synthesized video data D23 and the compressed synthesized audio data D24 included in the packet from the supplied packet data D22, sends the compressed synthesized video data D23 to the video decompressing unit 43, and also compresses the synthesized synthesized audio data. D24 is sent to the voice decompression unit 44.

  Then, the video decompression unit 43 performs decompression processing with the corresponding decoding method on the supplied compressed composite video data D23 and sends the obtained baseband composite video data D25 to the display processing unit 45. The display processing unit 45 performs predetermined signal processing such as digital / analog conversion processing on the composite video data D25, and sends the obtained video signal S11 to the display 46 such as a CRT. As a result, the composite image 32 (FIG. 4) based on the video signal S11 is displayed on the display 46.

  Further, the audio decompression unit 44 performs decompression processing by the corresponding decoding method on the supplied compressed synthesized audio data D24, and sends the obtained baseband synthesized video data D26 to the output audio processing unit 47. The output audio processing unit 47 performs predetermined signal processing such as digital / analog conversion processing on the synthesized audio data D26 and sends the obtained audio signal S12 to the speaker 48. As a result, the above-described synthesized voice based on the voice signal S12 is output from the speaker 48.

  Thus, in this video conference system 1, each user video 27, cursor image 29, and document image of each user 7 captured by the camera unit 4 of each video conference terminal 2 and the camera unit 21 of the video conference server 3, respectively. The composite video 32 formed by combining 31 is displayed on each display 46 of each video conference terminal 2 and the display 35 of the video conference server 3, and each microphone 5 of each video conference terminal 2 and microphone 22 of the video conference server 3 are displayed. The synthesized voices obtained by synthesizing the voices of the respective users 7 collected by the above can be output from the speakers 48 of the video conference terminals 2 and the speakers 41 of the video conference server 3, respectively.

(2) Various functions installed in the video conference system 1 Next, various functions installed in the video conference system 1 will be described.

(2-1) Display Control Function During Document Operation, etc. In this video conference system 1, each display 35 and each video conference terminal 2 or the video conference server 3 according to the operation of each pointing device 6 or 23 of the video conference server 3 46, the display position and / or size of the document image 31 in the composite video 32 displayed on the display 46, and the display control function at the time of document operation for changing the display position and / or size of each user video 27 of each user 7. Is installed.

  Actually, as shown in FIG. 5, the situation determination control unit 19 of the video conference server 3 implements a display control function at the time of the document operation, as shown in FIG. Has an additional information table 49 for managing the user 7 who is operating.

  The situation determination control unit 19 then selects one of the video conferences based on the operation input signal S6 supplied from the pointing device 23 of the video conference server 3 and the situation determination data D10 transmitted from each video conference terminal 2. The pointing device 6 or 23 of the terminal 2 or the video conference server 3 is operated to display the document image 31 based on the document data D14 stored in the document storage unit 26, or the currently displayed document image 31 to another When it is recognized that an instruction to change to the document image 31 or to display the next page of the currently displayed document image 31 is input, the file name of the document data D14 is displayed in the additional information table 49. In addition to storing in the document management column 49A, The ID of the video conference terminal 2 or the video conference server 3 used by the user 7 who performed the operation (hereinafter referred to as the terminal / server ID) is stored in the document operation user management column 49B of the additional information table 49. .

  Further, the situation determination control unit 19 controls the document storage unit 26 based on the additional information table 49 to read out the designated document data D14 stored in the document storage unit 26, and this is read out. It is sent to the video composition unit 20.

  Further, the situation determination control unit 19 controls the video composition unit 20 based on the additional information table 49 together with this, so that the document based on the document data D14 supplied from the document storage unit 26 as shown in FIG. The image 31 is enlarged and displayed at the center of the screen, and the user video 27 of the user 7 based on the video data D11 or D12 transmitted from the video conference terminal 2 or the video conference server 3 on which the operation is performed is enlarged to the lower left of the screen. The user video 27 of the user 7 based on the video data D11 or D12 that is displayed and transmitted from the other video conference terminal 2 or the video conference server 3 is not enlarged to the right side of the screen, a predetermined level that can recognize a nod or the like The composite video data D15 of the composite video 32 displayed in size is generated.

  Then, the video composition unit 20 sends the composite video data D15 generated in this way to the display processing unit 33 and the video compression unit 34, respectively. As a result, the composite video 32 as shown in FIG. 6 is displayed on the display 35 of the video conference server 3 and the display 46 of each video conference terminal 2.

  On the other hand, when the situation determination control unit 19 displays the document image 31 in the composite video 32 in this way, the operation input signal S6 supplied from the pointing device 23 of the video conference server 3 and each video conference terminal 2, the positions of the cursor images 29 respectively associated with the pointing device 23 of the video conference server 3 and the pointing device 6 of each video conference terminal 2 are monitored.

  The situation determination control unit 19 then displays each cursor image 29 associated with the video conference terminal 2 or the video conference server 3 different from the video conference terminal 2 or the video conference server 3 on which the cursor operation has been performed on the document. When a cursor operation that moves for 1 to 2 seconds or more is performed, the terminal / server ID stored in the document operation user column 49B of the additional information management table 49 is set to the new video conference terminal 2 or video conference server. The terminal / server ID is rewritten to 3.

  Further, the situation determination control unit 19 thereafter controls the video composition unit 20 based on the additional information table 49, so that, as shown in FIG. The user video 27 based on the video data D11 or the video data D12 from the video processing unit 25 of the video conference server 3 and the video data D11 from the video conference terminal 2 on which the cursor operation is newly performed or the video conference server 3 The composite video data D15 of the composite video 32 in which the user video 27 based on the video data D12 from the video processing unit 25 is replaced is generated, and this is transmitted to the display processing unit 33 and the video compression unit 34, respectively. Accordingly, the composite video 32 as shown in FIG. 7 is displayed on the display 35 of the video conference server 3 and the display 46 of each video conference terminal 2 with respect to the original composite video 32 as shown in FIG.

  On the other hand, when the situation determination control unit 19 controls the video composition unit 20 to display the document image 31 in the composite video 32 in this way, the operation input signal S6 supplied from the pointing device 23 of the video conference server 3 is displayed. Based on the situation determination data D10 transmitted from each video conference terminal 2, the time during which the user 7 performs the cursor operation on the document image 31 is monitored.

  Then, when the same user 7 is continuously performing the cursor operation, the situation determination control unit 19 controls the video composition unit 20 every predetermined time (for example, about 1 to 2 minutes) to thereby perform the predetermined time. Each time the document image 31 is enlarged step by step, the synthesized video data D15 of the synthesized video 32 as shown in FIG. 8 is generated and sent to the display processing unit 33 and the video compression unit 34, respectively. As a result, a composite video 32 is displayed on the display 35 of the video conference server 3 and the display 46 of each video conference terminal 2 so that the document image 31 is enlarged step by step according to the cursor operation of the user 7. The

  On the other hand, when no user 7 is performing such a cursor operation, the situation determination control unit 19 controls the video composition unit 20 every predetermined time (for example, about 1 to 2 minutes) to thereby perform the predetermined time. Each time, the synthesized video data D15 of the synthesized video 32 obtained by sequentially reducing the document image 31 in stages is generated and sent to the display processing unit 33 and the video compression unit 34, respectively. As a result, when there is no cursor operation of the user 7 in either the video conference terminal 2 or the video conference server 3, the document image 31 is sequentially reduced in steps at regular intervals, and finally, as shown in FIG. The composite video 32 in which the document image 30 is reduced and arranged at the left end of the composite video 32 is displayed on the display 35 of the video conference server 3 and the display 46 of each video conference terminal 2.

  In this manner, in the video conference system 1, the document image 30 and the user video 27 of the user 7 can be displayed so as to be enlarged or reduced according to the document operation of the user 7. Thus, the document image 31 and the user video 27 of the user 7 who has performed the operation can be mainly captured at the time of the document operation or the like without performing any operation.

(2-2) Display and Volume Control Function Based on Voice Detection etc. On the other hand, in this video conference system 1, the display 35 of the video conference server 3 and each video conference terminal 2 at that time according to the presence / absence of the speech of the user 7. The display position and size of the user video 27 of each user 7 in the composite video 32 displayed on the display 46 of each of the users 7 and each user 7 output from the speakers 41 and 48 of the video conference terminal 2 and the video conference server 3. It is equipped with a display and volume control function that changes the volume of the voice.

  Actually, in the case of this video conference system 1, in the video conference terminal 2, the audio signal S 2 output from the input audio processing unit 11 is given to the situation determination unit 13.

  And the situation judgment part 13 always monitors whether the user 7 has spoken from the signal level based on the supplied audio signal S2, and when detecting that the user 7 has spoken, this is described above. The determination data D5 is sent to the packetizing unit 10. Thus, the situation determination data D5 is thereafter transmitted to the video conference server 3 via the network NT as described above, and is then given to the situation determination control unit 19.

  At this time, the situation determination control unit 19 receives the audio signal S8 output from the input audio processing unit 38, and constantly monitors whether or not the user 24 of the video conference server 3 has made a speech based on the audio signal S8. It is made like that.

  Then, the situation determination control unit 19 determines whether any user 7 is speaking based on the monitoring result and the situation determination data D10 transmitted from each video conference terminal 2, and at that time, for example, 1 to When a user 7 speaking continuously for 2 seconds or more is detected, the terminal / server ID of the corresponding video conference terminal 2 or video conference server 3 is stored in the additional user management table 49 as shown in FIG. Store in column 49C.

  In addition, the situation determination control unit 19 controls the video composition unit 20 based on the additional information management table 49 thereafter, so that the user video based on the video data D11 and D12 from each video conference terminal 2 and the video conference server 3 is obtained. 27 is enlarged or reduced, and the voice synthesizer 37 is controlled at the same time, so that the volume of the speech voice of the user 7 based on the voice data D17 and D18 from each video conference terminal 2 and the video conference server 3 is controlled. Raise or lower.

  For example, as shown in FIG. 6, the situation determination control unit 19 displays a composite video 32 in which a document image 31 is enlarged and displayed at the center of the screen, and a user video 27 of a user 7 is enlarged and displayed on the left side. In the case where the synthesizing unit 20 generates, when another user 7 speaks, as shown in FIG. 7, the display positions of the user 7 that has been enlarged so far and the user 7 who has spoken and The synthesized video data D15 of the synthesized video 32 as shown in FIG. Thus, the composite image 32 based on the composite video data D15 is displayed on the displays 35 and 46 of each video conference terminal 2 and the video conference server 3.

  In addition, the situation determination control unit 19 continuously monitors the audio signal S2 from the input audio processing unit 38 of the video conference server 3 and the situation determination data D10 from each video conference terminal 2 thereafter, and the user 7 thereafter When it is determined that the user continues to speak, the video synthesis unit 20 and the voice synthesis unit 37 are controlled every predetermined time (for example, about 1 to 2 minutes).

  Thus, at this time, under the control of the situation determination control unit 19, the video composition unit 20 further enlarges the user video 27 of the user 7 that is displayed at that time in a stepwise manner at every predetermined time. Synthetic video data D15 of the synthetic video 32 as shown in FIG. 10 in which the user video 27 of the other user 7 is sequentially reduced step by step for every predetermined time is generated in synchronization with the display processing unit 33 and Each is sent to the video compression unit 34. As a result, the composite video 32 based on the composite video data D15 is displayed on each display 35 and 46 of each video conference terminal 2 and video conference server 3, respectively.

  At this time, under the control of the situation determination control unit 19, the voice synthesis unit 37 gradually increases the volume of the voice of the user 7 displayed at that time step by step in increments of the predetermined time, and synchronizes with this. Then, the synthesized voice data D19 of the synthesized voice in which the volume of the voice of the other user 7 is lowered step by step in a stepwise manner is generated and sent to the output voice processing unit 39 and the voice compression unit 40, respectively. As a result, synthesized speech based on the synthesized speech data D19 is output from each of the speakers 41 and 48 of the video conference terminal 2 and the video conference server 3, respectively.

  Further, for example, as shown in FIG. 9, the situation determination control unit 19 displays the document image 31 in the composite video 32 in a reduced size at the left end of the screen and the user video 27 of each user 7 on the right side thereof with an equal size. When a certain user 7 continues to speak, the video synthesis unit 20 and the voice synthesis unit 37 are controlled every predetermined time (for example, about 1 to 2 minutes).

  Thus, at this time, the video composition unit 20 gradually expands the user video 27 of the user 7 step by step for each predetermined time under the control of the situation determination control unit 19, and synchronizes with this. 7 of the synthesized video 32 obtained by sequentially reducing the 7 user videos 27 in a stepwise manner for each predetermined time, and transmitting the generated video data D15 to the display processing unit 33 and the video compression unit 34, respectively. As a result, a composite image 32 as shown in FIG. 11 based on the composite video data D15 is displayed on the displays 35 and 46 of each video conference terminal 2 and video conference server 3.

  At this time, the voice synthesizing unit 37 gradually increases the volume of the voice of the user 7 step by step for every predetermined time under the control of the situation determination control unit 19 and synchronizes with this. The synthesized voice data D19 of the synthesized voice in which the volume of the voice is gradually reduced step by step for each predetermined time is generated and sent to the output voice processing unit 39 and the voice compression unit 40, respectively. As a result, synthesized speech based on the synthesized speech data D19 is output from each of the speakers 41 and 48 of the video conference terminal 2 and the video conference server 3, respectively.

  On the other hand, the situation determination control unit 19 determines which user based on the audio signal S8 provided from the input audio processing unit 11 of the video conference server 3 and the situation determination information D10 transmitted from each video conference terminal 2. When it is detected that 7 is not speaking, the video synthesis unit 20 and the voice synthesis unit 37 are controlled every predetermined time (for example, about 1 to 2 minutes).

  Thus, at this time, under the control of the situation determination control unit 19, the video composition unit 20 is configured to obtain the user video 27 of each user 7 from the state shown in FIG. 10 or FIG. The user video 27 of the user 7 that has been enlarged and displayed at that time is reduced in stages step by step at the predetermined time until the size is reached, and the user video 27 of the other users 7 at the predetermined time in synchronization with this. The composite video data D15 of the composite video 32 that is sequentially expanded in steps is generated and sent to the display processing unit 33 and the video compression unit 34, respectively. Thus, based on the composite video data D15, the composite video 32 that is reduced or enlarged in stages until the sizes of the user videos 27 are equal to each other as described above becomes the video conference terminals 2 and the televisions. It is displayed on each display 35 and 46 of the conference server 3 respectively.

  At this time, the voice synthesizer 37 controls the voice of the user 7 whose volume has been raised until the volume of the voice of each user 7 becomes equal to a predetermined volume under the control of the situation determination control unit 19. The synthesized voice data D19 of the synthesized voice is sequentially reduced in a stepwise manner for each predetermined time and the volume of the voice of the other user 7 is gradually increased in a stepwise manner in synchronization with this. It is generated and sent to the output audio processing unit 39 and the audio compression unit 40, respectively. Thus, on the basis of the synthesized voice data D19, the synthesized voice that is lowered or raised until each user 7's voice volume becomes equal in a stepwise manner as described above is sent to each video conference terminal 2 or video conference server. 3 is output from each of the three speakers 41 and 48.

  In this manner, in the video conference system 1, the user video 27 of the user 7 can be displayed so as to be enlarged or reduced according to the presence or absence of the user 7's speech, thereby causing the user to perform an extra operation. The user video 27 of the user 7 can be captured as a main.

(2-3) Speech Support Function On the other hand, in this video conference system 1, a support mark image 50 as shown in FIG. 13 and a text image 51 as shown in FIG. Thus, a speech support function for supporting the speech of the users 7 and 24 is installed.

  In practice, in the case of this video conference system 1, each video conference terminal 2 is provided with a dedicated switch group 53 including a plurality of dedicated switches respectively associated with the content of statements such as “speak” or “know”. ing.

  When the user 7 presses a dedicated switch associated with the desired message content in the dedicated switch group 53, a pressing operation signal S13 corresponding to this is output to the situation determination unit 13 and the message image generation unit 54. The

  Each video conference terminal 2 is provided with a keyboard 55. When the user 7 performs a predetermined operation and inputs a character string such as "The cause is OO", it indicates that there is a keyboard operation. The keyboard operation signal S14 is output to the situation determination unit 13, and the text data D27 of the text input at this time is output to the utterance image generation unit 54.

  When the pressing operation signal S13 is given from the dedicated switch group 53 or the keyboard operation signal S14 is given from the keyboard 55, the situation judgment unit 13 sends the situation judgment data D5 corresponding thereto to the packetizing unit 10. Thus, the situation determination data D5 is thereafter transmitted to the video conference server 3 via the network NT as described above, and thereafter provided to the situation determination control unit 19 of the video conference server 3.

  Further, when the pressing operation signal S13 is given from the dedicated switch group 53 or the text data D27 is given from the keyboard 55, the speech image generating unit 54 corresponds to the pressing operation signal S13, for example, a support mark image as shown in FIG. For example, a text image 51 as shown in FIG. 14 is generated based on the text data D27, and image data (hereinafter referred to as a comment image) of the support mark image 50 or the text image 51 (hereinafter collectively referred to as a speech image). D28) is sent to the video compression unit 9.

  Thus, the speech image data D28 is subjected to predetermined compression processing and packetization processing in the video compression section 9 and the packetizing section 10 to the video conference server 3 via the network NT, as in the case of the video data D1. Then, the data is sent to the video composition unit 20 through the depacketizing unit 16 and the video decompression unit 17 in sequence.

  At this time, the video conference server 3 is provided with a dedicated switch group 56 composed of a plurality of dedicated switches similar to the video conference terminal 2, and the user 7 is associated with the desired message content in the dedicated switch group 56. When the dedicated switch is pressed, a pressing operation signal S15 corresponding thereto is given to the situation determination control unit 19 and the speech image generation unit 57.

  The video conference server 3 is also provided with a keyboard 58, which means that when the user 7 performs a predetermined operation and inputs a character string such as “The cause is OO”, there is a keyboard operation. The keyboard operation signal S16 is output to the situation determination unit, and the text data D29 of the text input at this time is output to the speech image generation unit 57.

  When the pressing operation signal S15 is given from the dedicated switch group 56 or the text data D29 is given from the keyboard 58, the speech image generation unit 57, for example, a support mark image 50 as shown in FIG. 13 corresponding to the pressing operation signal S15. Alternatively, for example, a text image 51 as shown in FIG. 14 is generated based on the text data D29, and image data (speech image data) D30 of the comment image formed by the support mark image 50 or the text image 51 is sent to the video composition unit 20.

  At this time, the situation determination control unit 19 sends the pressing operation signal S15 or the keyboard operation signal S16 supplied from the dedicated switch group 56 or the keyboard 58 of the video conference server 3 and the situation determination data D10 transmitted from each video conference terminal 2. Are constantly monitored, and any one of the dedicated switches 53 or 56 in the video conference terminal 2 or the video conference server 3 based on the pressing operation signal S15, the keyboard operation signal S16, or the situation determination data D10. When a dedicated switch is pressed or a character string to be displayed on the composite video 32 is input via the keyboard 55 or 58, a control signal D31 corresponding to this is sent to the video composition unit 20. .

  Thus, at this time, the video composition unit 20 based on the control signal D31, the document image 31 based on the document data D14 from the document storage unit 26, and the video data D12 supplied from the video processing unit 25 of the video conference server 3. Is transmitted from the corresponding video conference terminal 2 on the synthesized video 32 formed by synthesizing the video 28 based on the above and the user video 27 of the user 7 based on the video data D11 transmitted from each video conference terminal 2 or Composite image formed by superimposing and compositing speech images (support mark images 50 or text images 51) based on the speech image data D28 or D30 supplied from the speech image generation unit 54 or 57 of the video conference server 3 at corresponding positions. 32 composite image data D15 is generated, and is displayed as a display processing unit 33 and a video compression unit 34. Each sends. As a result, based on the composite video data D15, a composite image 32 in which a speech image (support mark image 50 or text image 51) as shown in FIG. 13 or 14 is displayed in the vicinity of the user 7 who performed the operation is displayed. It is displayed on each display 35 and 46 of each video conference terminal 2 and video conference server 3, respectively.

  In this manner, in the video conference system 1, it is possible to inform the other users of his / her intention to speak and the content of the desired speech in advance without casually disturbing the speech of the user who is speaking at that time. It is made like that.

(2-4) Illumination Function During Video Conference Further, in this video conference system 1, as shown in FIG. 15, the television screen is illuminated by illuminating the floor surface around the user 7 participating during the video conference. Each video conference terminal 2 and the video conference server 3 are equipped with a lighting function that allows the surrounding people to recognize that the conference is in progress.

  In practice, in the case of this video conference system 1, as is apparent from FIGS. 2 and 3, the video conference server 3 and each video conference terminal 2 are used to illuminate the floor surface around the user 7, respectively. Light sources 59 and 60 are provided.

  Then, when the situation determination control unit 19 of the video conference server 3 is connected to the corresponding video conference terminal 2 in a communicable state by, for example, SIP (Session Initiation Protocol) control according to the user operation, the drive voltage is applied to the light source 59. 15A, the light source 59 is turned on as shown in FIG. 15A. On the other hand, when all the video conference terminals 2 are disconnected in response to a user operation, the drive voltage to the light source 59 is reduced. By stopping the supply, the light source 59 is turned off as shown in FIG.

  Similarly, when the status determination unit 13 of each video conference terminal 2 is connected to the video conference server 3 in a state where it can communicate with the video conference server 3 in response to a user operation, the supply of drive voltage to the light source 60 is started. When the light source 60 is turned on as shown in A) and then the connection with the video conference server 3 is disconnected in response to a user operation, the supply of the drive voltage to the light source 60 is stopped, thereby stopping the light source 60 shown in FIG. In this way, the light source 60 is turned off.

  In this way, in this video conference system 1, by clearly distinguishing whether the user is working or is in video conference, it is easily recognized by the surroundings that the video conference is in progress. It is possible to prevent the surrounding people from speaking during a video conference effectively and effectively so that the video conference can proceed smoothly without interruption.

(3) Specific processing contents of the situation judgment control unit 19 (3-1) Display control processing procedure at the time of document operation Here, at the time of document operation by the situation judgment control unit 19 of the video conference server 3 as described above The display control is performed according to a display control processing procedure RT1 such as during document operation shown in FIG. 16 based on a control program given in advance to the situation determination control unit 19.

  In other words, when the situation determination control unit 19 is connected to the corresponding video conference terminal 2 in a communicable state, it starts the display control processing procedure RT1 at the time of document operation in step SP0, and in the subsequent step SP1, has the document operation been performed? Judge whether or not. Then, while waiting for the user 7 to operate the document in step SP1, the situation determination control unit 19 proceeds to step SP2 only when the document is operated, and determines whether or not the document image 31 is displayed.

  If the result of the determination in step SP2 is negative, the situation determination control unit 19 proceeds to step SP3 to enlarge and display the document image 31, and enlarges the user video 27 of the user 7 who has operated the document beside the document image 31. If an affirmative result is obtained, the process proceeds to step SP4 where the document image 31 is changed and displayed, and the user video 27 of the user 7 who has operated the document is displayed next to the document image 31. .

  Further, the situation determination control unit 19 thereafter proceeds to step SP5 and determines whether or not the cursor is operated within a predetermined time. The situation determination control unit 19 proceeds to step SP7 when an affirmative result is obtained in step SP5, and proceeds to step SP8 when an affirmative result is obtained, and displays the document image 31 in a reduced scale stepwise.

  Further, the situation determination control unit 19 proceeds to step SP7 to determine whether or not the same user 7 has continued to operate the cursor for a predetermined time. Then, when the situation determination control unit 19 obtains a positive result in step SP7, the process proceeds to step SP8, and the document image 31 is enlarged and displayed stepwise. On the other hand, if a negative result is obtained, the process proceeds to step SP9. A user video 27 of the user 7 who performed the cursor operation is displayed beside the image 31.

  Further, the situation determination control unit 19 proceeds to step SP10 to determine whether or not the user 7 has performed an operation (hereinafter referred to as an end operation) to end the video conference, and obtains a negative result. Returning to step SP1, thereafter, the same processing is repeated for step SP1 to step SP10. Then, when the user 7 eventually performs an ending operation and obtains a positive result in step SP10, the situation determination control unit 19 proceeds to step SP11 and ends the display control processing procedure RT1 such as during document operation.

(3-2) Display at the time of voice detection, etc. and volume control processing procedure On the other hand, when the situation determination control unit 19 is connected to the corresponding video conference terminal 2 in a communicable state, the above-mentioned display at the time of detection of the document operation or the like is performed. In parallel with the control processing procedure RT1, the display and sound volume control processing procedure RT2 shown in FIG. 17 is started in step SP12, and in step SP13, whether voice is detected from the user 7 within a predetermined time. Judge whether or not. Then, the situation determination control unit 19 proceeds to step SP15 when a positive result is obtained in step SP13, and proceeds to step SP14 when a negative result is obtained, and displays the user video 27 of the user 7 to be equalized. Let

  Further, the situation determination control unit 19 thereafter determines whether or not the voice from the same user 7 has been detected for a predetermined time or longer in step SP15. The situation determination control unit 19 proceeds to step SP17 when a negative result is obtained at step SP15, and proceeds to step SP16 when an affirmative result is obtained, and enlarges and displays the user video 27 of the user 7 who detected the voice. The user image 27 of the other user 7 is reduced and displayed, and the volume is reduced.

  Further, the situation determination control unit 19 thereafter determines whether or not the document image 31 is displayed in step SP17. The situation determination control unit 19 proceeds to step SP19 when a negative result is obtained in step SP17, and proceeds to step SP18 when an affirmative result is obtained in response to this, and the situation determination control unit 19 proceeds to step SP18. The user video 27 is displayed.

  Further, the situation determination control unit 19 proceeds to step SP19 to determine whether or not the user 7 has performed an end operation. If a negative result is obtained, the process returns to step SP13. Thereafter, the same applies to steps SP13 to SP19. Repeat the process. Then, when the user 7 eventually performs an end operation and obtains a positive result in step SP19, the situation determination control unit 19 proceeds to step SP20 and ends the display and volume control processing procedure RT2 at the time of voice detection.

  In this way, the situation determination control unit 19 can control the video synthesis unit 20 and the voice synthesis unit 37 to change the display position, size, and volume of the user video 27 and the document image 31 of the user 7. ing.

(4) Operation and Effect According to this Embodiment In the above configuration, the video conference system 1 displays the document image 31 and the user video 27 of the user 7 so as to be enlarged or reduced according to the document operation of the user 7. .

  Accordingly, without causing the user 7 to perform an extra operation, the document image 31 can be captured mainly when the document is operated, and the user video 27 of the user 7 who has performed the operation can be captured mainly.

  When the document image 31 is displayed in the composite video 32, the time during which the user 7 performs the cursor operation on the document image 31 is monitored, and the same user 7 continuously performs the cursor operation. The document image 31 is displayed so as to be enlarged step by step every predetermined time (for example, about 1 to 2 minutes), so that the user 7 does not need to perform an extra operation. When the cursor operation is performed on the document image 31, the document image 31 can be captured as a main.

  Furthermore, since the video conference system 1 is configured to display the user video 27 of the user 7 so as to be enlarged or reduced according to the presence or absence of the user 7 utterance, the user 7 is not allowed to perform an extra operation. The user video 27 of the user 7 who made the remark can be captured mainly.

  Furthermore, in this video conference system 1, either one of the video switches in the dedicated switch group 53 or 56 in the video conference terminal 2 or the video conference server 3 is pressed, or the synthesized video 32 is input via the keyboard 55 or 58. A comment image (support mark image 50 or text image 51) is displayed in the vicinity of the user 7 who performed the operation in response to an input of a character string to be displayed above. Without disturbing the user's speech, the user's intention to speak and desired speech content can be communicated to other users in advance.

  Further, when the video conference system 1 is connected to a state where communication is possible according to a user operation, the light source 59 is turned on by starting to supply a drive voltage to the light sources 59 and 60, and thereafter the user operation is performed. When the connection is cut accordingly, the light source 59 and 60 are turned off by stopping the supply of the drive voltage to the light source 59 and the surroundings can be easily recognized by the surroundings. Therefore, it is possible to effectively prevent people from talking during the video conference, and thus the video conference can proceed smoothly without being interrupted.

  According to the above configuration, the document image 31 and the user video 27 of the user 7 are displayed so as to be enlarged or reduced in accordance with the document operation of the user 7, thereby allowing the user 7 to perform the document operation without extra operations. In some cases, the document image 31 can be captured mainly, while the user video 27 of the user 7 who performed the operation can be captured mainly, thus realizing a video conference system with high communication and full of realism. Can do.

  When the document image 31 is displayed in the composite video 32, the time during which the user 7 performs the cursor operation on the document image 31 is monitored, and the same user 7 continuously performs the cursor operation. The document image 31 is displayed so as to be enlarged step by step every predetermined time (for example, about 1 to 2 minutes), so that the user 7 does not need to perform an extra operation. When a cursor operation is performed on the document image 31, the document image 31 can be captured as a main, thus realizing a video conference system with high communication and full of realism.

  Furthermore, since the user video 27 of the user 7 is displayed so as to be enlarged or reduced according to the presence or absence of the user 7's speech, the user 7 who made the speech without causing the user 7 to perform an extra operation. Thus, it is possible to realize a video conference system that is highly communicative and full of realism.

  Furthermore, in any of the video conference terminals 2 or the video conference server 3, any of the dedicated switches in the dedicated switch group 53 or 56 is pressed, or a character string to be displayed on the composite video 32 via the keyboard 55 or 58. The comment image (support mark image 50 or text image 51) is displayed in the vicinity of the user 7 who performed the operation in response to the input of the user, so that the user who is speaking at that time is prevented from the comment. In addition, it is possible to inform the user of his / her intention to speak and the desired content of the speech in a casual manner in advance, thus realizing a video conferencing system with high communicability and full of realism.

  Furthermore, when connected in a communicable state according to a user operation, the light source 59 is turned on by starting the supply of driving voltage to the light sources 59 and 60, and then the connection is disconnected according to a user operation. Since the light sources 59 and 60 are turned off by stopping the supply of the drive voltage to the light source 59 and the video conference, it is possible to easily recognize that the video conference is in progress. Talking with people around you can be prevented in advance and effectively, allowing the video conference to proceed smoothly without interruption, thus realizing a highly conducive and realistic video conference system. .

(5) Other Embodiments In the above-described embodiment, the video conference system 1 including the video conference server 3 and the two video conference terminals 2 (2A, 2B) has been described. The present invention is not limited to this, and the present invention can be applied to various other real-time video and audio transmission / reception systems including one or more terminal devices and an information processing device.

  In the above-described embodiment, when the document image 31 is displayed, the case where the user video 27 of the user 7 is displayed to be enlarged or reduced according to the presence or absence of the user 7 has been described. The present invention is not limited to this, and even when the document image 31 is not displayed, the user video 27 of the user 7 can be displayed so as to be enlarged or reduced according to the presence or absence of the user 7 utterance as described above. it can.

  Furthermore, in the above-described embodiment, the case where the video conference terminal 2 and the video conference server 3 are configured and processed as described above has been described. However, the present invention is not limited thereto, and for example, the video conference server Only the synthesis process and the situation determination control process may be performed on the various types of data to be transmitted, and the data may be transmitted. The control process may be performed to transmit the data, and various other configurations can be widely applied.

  Further, in the above-described embodiment, the case where the speech image generation unit is provided in each of the video conference terminal 2 and the video conference server 3 has been described. However, the present invention is not limited thereto, and for example, only the video conference server is provided. It may be a case where it is provided, and various other configurations can be widely applied.

  Further, in the above-described embodiment, when the cursor operation or the user's utterance in which each cursor image 29 moves on the document for 1 to 2 seconds or more is detected for 1 to 2 seconds or more, the user 7 user Although the case where the video 27 is displayed so as to be enlarged or reduced has been described, the present invention is not limited thereto, and may be made longer or shorter than the time in consideration of various other effects.

  Further, in the above-described embodiment, when the user's 7 cursor operation and / or the user's 7 speech are performed at the same time, the document image 31 and the user's 7 user video 27 are associated based on the operation time and / or detection time. Although the case where the image is displayed so as to be enlarged or reduced to the size has been described, the present invention is not limited to this. For example, the size of the volume, the size of the movement of the cursor, and the number of document operations and voice detection within a unit time. The document image 31 and the user video 27 may be displayed so as to be enlarged or reduced to a corresponding size, and various other determination criteria can be used.

  Furthermore, in the above-described embodiment, any one of the dedicated switches in the dedicated switch group 53 or 56 is pressed by any one of the video conference terminals 2 or the video conference server 3, or the composite video 32 via the keyboard 55 or 58. Although the speech image (the support mark image 50 or the text image 51) is displayed near the user 7 (7A or 7B) or 24 who performed the operation according to the input of the character string to be displayed above, The present invention is not limited to this. For example, a speech image (support mark image 50 or text image 51) is displayed in the vicinity of the user 7 who performed the operation, and the video 27 of the user 7 who performed the operation is enlarged in the center. You may make it display and have it speak preferentially.

  Further, in the above-described embodiment, the volume of the voice of the user 7 who is speaking is increased stepwise in increments of the predetermined time, and the volume of the voice of the other user 7 is increased in synchronism with the predetermined time. However, the present invention is not limited to this, and the volume of the voice of the other user 7 may be lowered step by step at the predetermined time. The sound volume for the user who is present may be increased relatively to the sound volume for the other users.

  The present invention can also be used in the case of transmitting and displaying video and audio in real time on various devices other than the video conference system.

It is a basic diagram which shows the structure of the video conference system by this Embodiment. It is a block diagram which shows the internal structure of the video conference terminal by this Embodiment. It is a block diagram which shows the internal structure of the video conference server by this Embodiment. It is a conceptual diagram with which it uses for description of the display in a display. It is a conceptual diagram with which it uses for description of an additional information management table. It is a conceptual diagram with which it uses for description of the display in a display. It is a conceptual diagram with which it uses for description of the display in a display. It is a conceptual diagram with which it uses for description of the display in a display. It is a conceptual diagram with which it uses for description of the display in a display. It is a conceptual diagram with which it uses for description of the display in a display. It is a conceptual diagram with which it uses for description of the display in a display. It is a conceptual diagram with which it uses for description of the display in a display. It is a conceptual diagram with which it uses for description of the display in a display. It is a conceptual diagram with which it uses for description of the display in a display. It is a conceptual diagram with which it uses for description of the distinction between the time of a video conference, and a self-work. It is a flowchart which shows the display control processing procedure at the time of document operation. It is a flowchart which shows the display at the time of audio | voice detection, etc. and a volume control processing procedure.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Video conference system, 2 ... Video conference terminal, 3 ... Video conference server, 4, 21 ... Camera part, 5, 22 ... Microphone, 6, 23 ... Pointing device, 7 ... User, 13 ...... Situation judgment unit, 19 ... Situation judgment control unit, 20 ... Video composition unit, 27 ... User video, 29 ... Cursor image, 31 ... Document image, 32 ... Composite video, 35, 46 ... Display, 37... Speech synthesis unit, 49... Additional information management table, NT... Network, RT1... Display control processing procedure at the time of document operation, RT2.

Claims (15)

In a video conference system including one or more terminal devices installed at different points and an information processing device connected to the terminal device via a predetermined network,
The terminal device is
Photographing means for photographing the user;
Sound collecting means for collecting the user's voice;
A document operation means for performing a predetermined document operation on the document;
Transmitting means for transmitting video data output from the photographing means and audio data output from the sound collecting means to the information processing apparatus via the network;
Display means for displaying video based on composite video data transmitted from the information processing apparatus via the network,
The information processing apparatus
Video synthesizing means for generating a synthesized video obtained by synthesizing the video of the user based on the video data transmitted from the terminal device via the network and a necessary image of the document;
Based on the document operation using the document operation means and / or presence / absence of the user's speech detected based on the audio data, the image of the document and / or the corresponding user image in the synthesized video is enlarged or reduced. Control means for controlling the composite video means so as to
A video conferencing system comprising: transmission means for transmitting the composite video data, which is video data of the composite video generated by the video synthesis means, to the terminal device via the network. The information processing apparatus
Comprising speech synthesis means for generating synthesized speech formed by synthesizing speech based on the speech data transmitted from each of the terminal devices via the network;
The control means includes
According to the user's speech time, the voice synthesis means is controlled so that the volume for the corresponding user in the synthesized voice is relatively larger than the volume for the other users,
The transmission means is
Transmitting synthesized voice data consisting of voice data of the synthesized voice generated by the voice synthesizing means to the terminal device via the network;
The terminal device is
The video conference system according to claim 1, further comprising audio output means for outputting the synthesized voice based on the synthesized voice data transmitted from the information processing apparatus. The terminal device is
A light source that emits light;
A blinking control means for controlling blinking of the light source,
The blinking control means is
The video conference system according to claim 1, wherein when connected to the information processing apparatus, the light source is turned on, and when the connection with the information processing apparatus is disconnected, the light source is turned off. In one or a plurality of terminal devices installed at different points and an information processing device connected to the terminal device via a predetermined network,
Video synthesizing means for generating a synthesized video obtained by synthesizing the video of the user and a necessary document image based on the video data of the photographed user transmitted from the terminal device via the network;
The synthesis based on the presence or absence of the user's speech detected based on the predetermined document operation on the document transmitted from the terminal device via the network and / or the voice data of the collected user's voice Control means for controlling the composite video composition means so as to enlarge or reduce the image of the document in the video and / or the corresponding video of the user;
An information processing apparatus comprising: transmission means for transmitting synthesized video data composed of video data of the synthesized video generated by the video synthesizing means to the terminal device via the network. The above document operation
An operation for displaying or changing the document or changing the page of the document.
The control means includes
The video synthesizing unit is controlled to enlarge the image of the document in the synthesized video and / or the video of the user who performed the document manipulation when the document operation is performed. 5. The information processing apparatus according to 4. The control means includes
6. The image synthesizing unit is controlled so that when the document operation is not performed for a predetermined time, the image of the document and the corresponding video of the user are gradually reduced every predetermined time. The information processing apparatus described in 1. The above document operation
An operation of moving the cursor displayed on the composite video on the image of the document,
The control means includes
The information processing apparatus according to claim 4, wherein the video composition unit is controlled so as to enlarge the image of the document in the composite video in a stepwise manner according to an operation time of the document operation. The control means includes
8. The video synthesizing unit is controlled so that when the document operation is not performed for a predetermined time, the image of the document and the corresponding video of the user are gradually reduced every predetermined time. The information processing apparatus described in 1. The control means includes
When the document operation using the document operation means and / or the user's speech detected based on the audio data is not performed continuously for a predetermined time or more, the document operation or the user's speech is not performed The information processing apparatus according to claim 4, wherein the image synthesizing unit is controlled. The control means includes
5. The information processing apparatus according to claim 4, wherein the video synthesizing unit is controlled so as to gradually expand the corresponding video of the user in the synthesized video in accordance with the speech time of the user. Comprising speech synthesis means for generating synthesized speech formed by synthesizing speech based on the speech data transmitted from each of the terminal devices via the network;
The control means includes
According to the user's speech time, the voice synthesis means is controlled so that the volume for the corresponding user in the synthesized voice is relatively larger than the volume for the other users,
The transmission means is
5. The information processing apparatus according to claim 4, wherein synthesized voice data including voice data of the synthesized voice generated by the voice synthesizer is transmitted to the terminal device via the network. The control means includes
The information processing apparatus according to claim 4, wherein the video synthesizing unit is controlled to display a predetermined image supporting the user's speech on the synthesized video in response to a predetermined user operation. The control means includes
5. The information processing apparatus according to claim 4, wherein the video synthesizing unit is controlled to display a text image representing a character string input by the user on the synthesized video in response to the user operation. . In an information processing method in one or a plurality of terminal devices installed at different points and an information processing device connected to the terminal device via a predetermined network,
A first step of generating a synthesized video obtained by synthesizing a user's video and a necessary document image based on the captured video data of the user transmitted from the terminal device via the network;
The synthesis based on the presence or absence of the user's speech detected based on the predetermined document operation on the document transmitted from the terminal device via the network and / or the voice data of the collected user's voice A second step of enlarging or reducing the image of the document in the video and / or the corresponding video of the user;
And a third step of transmitting the composite video data composed of the video data of the composite video to the terminal device via the network. For one or a plurality of terminal devices installed at different points and an information processing device connected to the terminal device via a predetermined network,
A first step of generating a synthesized video obtained by synthesizing a user's video and a necessary document image based on the captured video data of the user transmitted from the terminal device via the network;
The synthesis based on the presence or absence of the user's speech detected based on a predetermined document operation on the document transmitted from the terminal device via the network and / or voice data of the collected user's voice A second step of enlarging or reducing the image of the document in the video and / or the corresponding video of the user;
A program for executing a third step of transmitting composite video data composed of video data of the composite video to the terminal device via the network.

Images