JP2005151002A - Remote conference system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a remote conference system capable of realizing a remote conference with full of presence whereby participants during discussion can easily be visually recognized even when a discussion is carried out between a plurality of the participants. <P>SOLUTION: A voice processing module 13 receives voice data of each participant and a voice level measurement means 31 measures a voice level of each participant from the voice data of each participant. The voice processing module 13 inputs information for particularizing the participant during making utterance and information with respect to the voice level to an image processing module 14. The image processing module 14 outputs display menu data on a monitor 20 so that the participant during the utterance is displayed at a magnification factor in response to the voice level of the utterance for a prescribed period. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a remote conference system, and more particularly to a remote conference system that transmits and receives a video signal and an audio signal via a network.

  In a remote communication system (remote conference system) such as a video conference system, video and audio of a plurality of participants are transmitted / received via a network, and each participant checks the video of other participants on a display screen. Take communication. For example, such a remote conference system connects each participant's personal computer to an imaging device such as a camera, a voice input device such as a microphone, a video display device such as a display, and a voice output device such as a speaker. realizable. The remote conference system can be realized by a server / client method or a peer-to-peer method.

  When the remote conference system is configured by a client-server system, the remote conference server for realizing the remote conference system uses the video signal and the audio signal transmitted from the client terminal used by each participant, etc. The video and audio of the other participants received are transmitted to each client terminal. On the other hand, when the teleconferencing system is configured in a peer-to-peer manner, the client terminal used by each participant receives a video signal and a client terminal used by other participants without passing through a server. Send and receive audio signals.

  In a remote conference, a plurality of participants are usually displayed on the same screen in divided areas. For example, if there are four participants displayed on the display screen, the display screen is divided into four, and one participant is assigned to each divided area. In a general conference, when there is a participant who is speaking, other participants can easily grasp who is speaking. However, in a teleconferencing system, the video of the participants displayed on the display screen is generally the same size, so when there are many participants, the video of each participant is small. It often becomes difficult to understand what is inside, and meetings often lack realism.

In a remote conference system, as a technique for giving a sense of reality to a meeting, there are techniques described in Patent Document 1 and Patent Document 2, for example. For example, in Patent Document 1, the server specifies the participant who is speaking by measuring the audio level from the audio signal received from each participant, and displays the participant's video on the client terminal of the other participant. Zoom in on the display screen. At that time, the enlargement ratio when the video of the participant who is speaking is enlarged and displayed is determined based on the measured voice level of the speaker.
JP-A-5-103324 JP-A-9-261608

  In Patent Document 1, the video of the participant who is speaking is displayed larger than the other participants, giving a sense of realism to the conference. However, in Patent Document 1, for example, in the case where a discussion is performed between a plurality of participants, the video of the participant displayed in an enlarged manner is changed rapidly, and which participant is currently discussing. There is a problem that it is difficult to grasp.

  The present invention solves the above-mentioned problems and can realize a remote conference with a sense of reality, and even when discussions are held between a plurality of persons, it is easy to visually recognize participants who are discussing. An object is to provide a conference system.

  In order to achieve the above object, a remote conference system according to the present invention includes three or more terminal devices each having a voice input device, an imaging device, a voice output device, and a video display device, and the terminal device via a communication line. Are connected to each other to transmit and receive audio data and image data to and from each other, each of the terminal devices includes level measuring means for measuring the audio level of audio data received from another terminal device, and the image And an enlargement ratio setting means for setting an enlargement ratio that depends on the sound level of the sound data corresponding to each of the data, and when the sound level of the sound data received from another terminal device increases, the video display device When the corresponding image data is displayed at the enlargement ratio newly set by the enlargement ratio setting means and the sound level of the sound data decreases, the sound level is reduced. From the time when Le is lowered after a predetermined time elapses, and displaying the image data at the newly set magnification at the magnification setting means.

  In the remote conference system of the present invention, the enlargement rate of the user image data displayed on the display screen of the video display device is in accordance with the audio level measured from the audio data of the participant (user) who uses the terminal device. Therefore, for example, a remote conference with a sense of reality can be realized by setting the enlargement ratio higher for a user with a higher voice level. Further, when the user's voice level is changed in a decreasing direction, the image data of the user is enlarged according to the voice level before the decrease until a predetermined period elapses after the voice level is lowered. It is displayed at a rate, and after a predetermined time has elapsed, it is displayed at an enlargement rate according to the reduced audio level. For this reason, for example, when a discussion is performed between a plurality of users, a plurality of users are displayed on the video display device at an enlargement ratio corresponding to each sound level. You can easily see who you are discussing with. The predetermined time is, for example, about 1 to 2 minutes, and can be arbitrarily determined according to the usage status of the remote conference system.

  In the teleconference system of the present invention, it is preferable that the video display device displays image data corresponding to audio data whose audio level has finally increased on the forefront. In this case, each user can easily visually recognize who is the last speaker.

  In the remote conference system according to the present invention, each of the terminal devices is configured to transmit the control information for prohibiting the enlarged display of the image data of the image data transmitted by the terminal device to another terminal. Can be adopted. In this case, for example, in the remote conference, the presenter can display his / her video in the same size on the video display device of another user.

  In the remote conference system of the present invention, each of the terminal devices can adopt a configuration in which enlargement display of an image of image data received from another terminal is prohibited or the enlargement ratio is controlled. In this case, each user can arbitrarily customize the display size of the video data of other users displayed on the video display device used by the user.

  In the remote conference system of the present invention, the video display device can adopt a configuration in which a part of the received image data is extracted and displayed in an enlarged manner. Image data can be enlarged by simply scaling up or down the image size displayed on the video display device, or alternatively or in addition to extracting a part of the image data. This can be done by enlarging the extracted range.

  In the remote conference system of the present invention, the enlargement ratio of the image data is set according to the audio level measured from the audio data, so that a remote conference with a sense of reality can be realized. In addition, when the audio level of the audio data decreases, the video display device displays the image data at an enlargement rate newly set by the enlargement rate setting means after a predetermined time has elapsed from the time when the audio level has decreased. When a discussion is performed between a plurality of persons in a conference, the participant of the conference can easily visually recognize who is participating in the discussion.

  Hereinafter, with reference to the drawings, the present invention will be described in more detail based on exemplary embodiments of the present invention. FIG. 1 shows the configuration of a remote conference system according to an embodiment of the present invention. The remote conference system of the present invention is a communication communication system with audio and video in which a plurality of people participate, and can be applied to, for example, a videophone or a video chat in which a plurality of people make a call. Or it can be applied to a distance education system such as English education by correspondence system. In the figure, the remote conference system is configured as a peer-to-peer remote conference system 100, and the remote conference system 100 includes a plurality of client terminals 10 connected to a network 21.

  Each client terminal 10 includes a computer 11 including an input / output interface 12, an audio processing module 13, an image processing module 14, an input interface 15, and an output interface 16, a microphone 17 connected to the computer 11, a camera 18, and a speaker. 19 and a monitor 20. The audio processing module 13 includes an audio level measurement unit 31, and the image processing module 14 includes an enlargement ratio setting unit 41 and a screen creation unit 42. The input interface 15 inputs the participant's voice from the microphone 17 and inputs the participant's video from the camera 18. The input interface 15 inputs audio input from the microphone 17 to the audio processing module 13 and inputs video input from the camera 18 to the image processing module 14.

  The voice processing module 13 digitizes the input voice, and transmits the digitized voice data to another client terminal 10 via the input / output interface 12 using itself as a transmission source. The image processing module 14 transmits the input video data to another client terminal 10 via the input / output interface 12 with itself as a transmission source. The input / output interface 12 receives video and audio of other participants from other client terminals 10. Audio received from the other client terminal 10 received by the input / output interface 12 is input to the audio processing module 13, and video from the other client terminal 10 is input to the image processing module 14.

  When the voice data of other participants is input, the voice processing module 13 converts the digitized voice data into analog, and the voice level measuring unit 31 uses the voice data amplitude (voice) for each participant. Level). In addition, the audio processing module 13 outputs the audio data subjected to analog conversion to the speaker 19 via the output interface 16. The voice processing module 13 determines the presence / absence of a participant who is speaking from the voice level measured by the voice level measuring unit 31, and when there is a participant who is speaking, information for identifying the participant is measured. The audio level is transmitted to the image processing module 14.

  The image processing module 14 creates display screen data such that the video of each participant is displayed on the display screen of the monitor 20 by the screen creation means 42, and monitors the created display screen data via the output interface. 20 is output. When the image processing module 14 receives the information for identifying the participant who is speaking and the voice level of the participant from the voice processing module 13, the image processing module 14 is displayed on the monitor 20 for a predetermined time according to the voice level. Display screen data is created so that the video of the speaker is displayed larger than the video of the participant who is not speaking. In the image processing module 14, the enlargement rate setting means 41 determines the enlargement rate of the video of the participant who is speaking. The image processing module 14 enlarges and displays the video of the speaker on the monitor 20 as the voice level of the speaker is higher, and highlights the speaker.

  The display screen data output by the image processing module 14 can be customized by a participant who uses the client terminal 10, and each participant can display the display position of each participant in the display screen of his / her client terminal 10. An enlargement ratio for enlargement can be arbitrarily set according to the display size or the audio level. However, for example, a specific participant such as a remote conference presenter or a distance learning instructor wants to display his / her video in a fixed video size at a fixed position in the display screen of each client terminal 10. Can send a signal to the client terminal 10 to prohibit the change of the display position and the display size, and a specific participant is prohibited from changing the display position and the display size of his own video. When transmitting a signal to the effect, each participant cannot arbitrarily set a display position or a display size for a specific participant.

  FIG. 2 shows a processing procedure example of the audio processing module and the image processing module. The audio processing module 13 and the image processing module 14 are activated in a separate thread in the computer 11. In each client terminal 10, the voice processing module 13 waits for processing until the voice of any participant is input, and waits for the start of processing (step S11). When audio data is input from any participant including itself via the input / output interface 12 or the input interface 15, the audio processing module 13 uses the audio level measurement unit 31 to output audio data. The voice level is measured for each person (step S12).

  The voice processing module 13 determines the presence or absence of a participant who is speaking from the measured voice level. When there is a participant who is speaking, information for identifying the participant and the voice level of the participant are determined. It transmits to the image processing module 14 (step S13). In step S13, the voice processing module 13 determines that the participant is speaking when the voice level preset for each participant is exceeded, for example. The audio processing module 13 transmits an audio signal to the speaker 19 via the output interface 16 and outputs audio (step S14).

  In each client terminal 10, the image processing module 14 waits for processing until a video of any participant is input, and waits for the start of processing, similarly to step S11 of the audio processing module 13 (step S21). When video data is input from any of the participants including itself via the input / output interface 12 or the input interface 15, the image processing module 14 uses the screen creation means to make the video of each participant the same size. Display screen data to be displayed on the monitor is created (step S22). The video size of each participant's video in the display screen data created in step S22 is, for example, a video size corresponding to an enlargement rate corresponding to the audio level (level 0) when not speaking.

  3A to 3C show examples of display screen data created in step S22. The video display area of each participant is surrounded by a frame having a predetermined width. In step S22, for example, if there are four participants, the image processing module 14 displays the images of the participants A to D in the same vertical size as shown in FIG. Such display screen data, or display image data that is displayed in the same size in a horizontal row as shown in FIG. Or, for example, when the number of participants is 16, display image data is displayed so that the images of the participants A to P are arranged in the same size vertically and horizontally as shown in FIG. create.

  The image processing module 14 determines whether or not the information for identifying the participant who is speaking and the sound level of the participant who is speaking are received from the sound processing module 13 (step S23). When the image processing module 14 receives the information for identifying the participant who is speaking and the voice level of the participant who is speaking, the image processing module 14 sets the enlargement rate of the participant who is speaking by the enlargement rate setting means 41, and screen creation means 42, the display screen data created in step S22 is changed, and the video of the participant who is speaking is enlarged at the enlargement rate set by the enlargement rate setting means 41, and the participant who is speaking is emphasized. Display screen data to be displayed is created (step S24).

  In step S24, when the voice level of the participant who is speaking falls from level 5 indicating that the speaker is speaking to level 0 indicating silence, for example, the image processing module 14 continues from that point until a predetermined time elapses. Creates display screen data for displaying participants at an enlargement rate corresponding to level 5. In step S24, the image processing module 14 creates display screen data such that the latest speaker image is displayed in the foreground. Further, the image processing module 14 creates display image data in which the color of the frame corresponding to the participant who is speaking is red. Alternatively, the image processing module 14 creates display image data that blinks and displays the color of the frame corresponding to the participant who is speaking.

  FIGS. 4A to 4C respectively show the images of the participants in the display screen data. For example, when the participant is not speaking, it is assumed that the video size (display size) of the participant in the display screen data created in step S22 is 160 × 120, as shown in FIG. When this participant speaks and the voice level of the speech is, for example, 5 in 10 stages, in step S24, the participant's video is shown in FIG. Display screen data that is enlarged 1.5 times vertically and horizontally to a display size of 240 × 180 is created.

  Alternatively, instead of the above, for example, as shown in FIG. 4C, a range surrounded by a dotted line in FIG. 4A is set as a cutout range, and an image in this cutout range is enlarged to 240 × 180 Display screen data that can be displayed at a display size of may be created. In this case, the video enlargement ratio is 1.5 times when the original display size is used as a reference, but when the cutout range is used as a reference, the enlargement ratio is 1.5 times or more. For example, when the audio level of the participant is 5 or less, the image processing module 14 enlarges the entire video of the participant as shown in FIG. 5B, and when the audio level exceeds 5, the image processing module 14 For emphasis, a part of the participant's video can be enlarged and displayed on the monitor 20 as shown in FIG.

  FIGS. 5A to 5C each show an example of display screen data created in step S24. For example, if the display screen data shown in FIG. 3A is created in step S22 and the participants C and D speak, in step S24, as shown in FIG. For C and D, the display position and display size are changed, and display image data is created so that the images of the participants C and D are emphasized compared to the images of the participants A and B. Is done. In the example of FIG. 5A, the latest speaker is the participant D, and the voice level of the participant D is higher than the voice level of the participant C, so the participant D is displayed in the foreground. In addition, the participant D is displayed in a larger size than the participant C.

  Unlike the above, for example, if the display screen data shown in FIG. 3B is created in step S22, and participant B speaks, in step S24, as shown in FIG. Display image data in which the video of the person B is enlarged according to the sound level is created. For example, when the display screen data shown in FIG. 3C is created in step S22, and the participant F, the participant G, and the participant O speak, in step S24, the display screen data shown in FIG. ), The images of the participant F, the participant G, and the participant O are enlarged in accordance with the respective audio levels, and display image data is created such that these speakers pop up on the front side. The

  The image processing module 14 outputs the display screen data created by the screen creation means 42 to the monitor 20 via the output interface 16 (step S25). When the image processing module 14 does not receive the information for identifying the speaking participant and the voice level of the speaking participant from the audio processing module 13 in step S23, the display processing unit 20 monitors the display image data created in step S22. When the information for identifying the speaking participant and the voice level of the speaking participant are received from the speech processing module 13 in step S23, the display image data changed in step S24 is monitored. 20 is output.

  In the present embodiment example, the audio processing module 13 identifies a participant who is speaking according to the sound level of the speaker, and the image processing module 14 expands the video size of the participant who is speaking according to the sound level. Display screen data that displays a blinking red frame that displays the participant who is speaking. Thereby, in the remote conference system 100, it is easy to attract attention to the speaker, and the realism of the remote conference can be enhanced, and the conference can proceed smoothly.

  In the present embodiment example, the video of the participant who has spoken is enlarged and displayed continuously for a predetermined time from the time when the speech was made.For example, when a discussion is performed between a plurality of participants, It is possible to easily determine who and who is currently discussing. At this time, the image processing module 14 creates display screen data that displays the video of the latest speaker in the foreground, so that each participant can say “who speaks after who speaks, and who speaks. Can be easily identified. In addition, when the discussion is incandescent, the video of the participant displayed in the foreground is switched each time the speaker is switched, and a realistic conference can be produced by a visual effect.

  In the present embodiment, the audio processing module 13 and the image processing module 14 are activated as separate threads, and the display screen data output by the image processing module 14 can be customized by a participant using the client terminal 10. is there. In the present embodiment, with this configuration, each participant can display information identifying the participant who is speaking from the voice processing module 13 and what display screen data is received by the image processing module 14 that has received the voice level of the speech. Can be set dynamically.

  For example, consider an example in which participant A, participant B, and participant C are included. In this case, when the participant C is speaking at the audio level 5, the video size of the participant C displayed on the monitor 20 of the client terminal 10 used by the participant A and the client used by the participant B are displayed. The video size of the participant C displayed on the monitor 20 of the terminal 10 can be different from each other. Also, when the participant B feels bothered that the video of the participant C is enlarged, the video of the participant C is set to the audio level on the monitor 20 of the client terminal 10 used by the participant B at any time. Accordingly, it can be set not to enlarge, or the enlargement rate of the video size of the participant C with respect to the audio level can be reduced.

  6 and 7 each show the configuration of a remote conference system according to another embodiment of the present invention. In FIG. 1, each client terminal 10 is connected to the same network 21. However, as shown in FIG. 6, the remote conference system of the present invention includes a network 21 (1) and a plurality of client terminals 10 connected to each other. 21 (2) may be connected to each other via a network 21 (3) such as the Internet.

  The remote conference system of the present invention can also be configured by a server / client system. In this case, as shown in FIG. 7, each of the networks 21 (1) and 21 (2) to which the plurality of client terminals 10 are connected is connected to the network 21 (4) via the network 21 (4) such as the Internet. It can be set as the structure connected to the server connected to 3). When the remote conference system is configured by a server / client system, the audio and video of each participant is collected in the server 22 and distributed from the server 22 to each client terminal 10. Further, the image processing module 14 of each client terminal 10 receives the information for identifying the participant who is speaking and the voice level of the participant from the server 22 and creates display image data.

  As described above, the present invention has been described based on the preferred embodiment. However, the remote conference system of the present invention is not limited to the above embodiment, and various modifications and changes can be made to the configuration of the above embodiment. Changes are also included in the scope of the present invention. For example, the client terminal 10 can be configured as a mobile communication device such as a mobile phone.

1 is a block diagram showing a configuration of a remote conference system according to an embodiment of the present invention. The flowchart which shows the example of a process sequence of an audio | voice processing module and an image processing module. (A)-(c) is a screen example which respectively shows the example of the display screen data produced by step S22. (A)-(c) is an example of a screen which respectively shows the image | video of the participant in display screen data. (A)-(c) is an example of a screen which respectively shows the example of the display screen data produced by step S24. The block diagram which shows the structure of the remote conference system of another embodiment of this invention. The block diagram which shows the structure of the remote conference system of another embodiment of this invention.

Explanation of symbols

10: client terminal 11: computer 12: input / output interface 13: audio processing module 14: image processing module 15: input interface 16: output interface 17: microphone 18: camera 19: speaker 20: monitor 21: network 22: server 31: Audio level measurement means 41: enlargement ratio setting means 42: screen creation means 100: remote conference system

Claims (5)

Each includes three or more terminal devices each having an audio input device, an imaging device, an audio output device, and a video display device, and the terminal devices are connected to each other via a communication line to transmit and receive audio data and image data to and from each other. In the remote conference system,
Each of the terminal devices includes a level measurement unit that measures a sound level of sound data received from another terminal device, and a magnification rate that sets a magnification rate that depends on the sound level of the sound data corresponding to each of the image data. Setting means,
When the audio level of the audio data received from another terminal device rises, the video display device displays the corresponding image data at the enlargement rate newly set by the enlargement rate setting means, and the audio level of the audio data is A teleconferencing system characterized in that, when lowered, image data is displayed at an enlargement rate newly set by the enlargement rate setting means after a predetermined time has elapsed from the time when the audio level has dropped.   The teleconference system according to claim 1, wherein the video display device displays image data corresponding to audio data whose audio level has finally increased on the forefront.   The remote terminal according to claim 1 or 2, wherein each of the terminal devices transmits control information for prohibiting enlarged display of an image of image data transmitted by the terminal device or designating an enlargement rate to another terminal. Conference system.   Each of the said terminal device is a remote conference system as described in any one of Claims 1-3 which prohibits the enlarged display of the image of the image data received from the other terminal, or controls an expansion rate.   The remote conference system according to claim 1, wherein the video display device extracts a part of the received image data and displays the enlarged image data.

Images