JP2013008210A - Conference system, image display device, and image voice processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a conference system for improving convenience.SOLUTION: A conference system 1 displays a synthetic image obtained by synthesizing respective images to be treated by respective terminals 2 with one another on an image display device 3. Each terminal 2 includes: input acceptance means for accepting input of display states of an image to be treated by the terminal 2 and images to be treated by other terminals 2 in the synthetic image; and transmission control means for transmitting voice information regarding voice collected by microphones, display information regarding the display states, and image information regarding the image to be treated by the terminal 2 to the image display device 3. The image display device 3 includes: reception means for receiving the voice information, the display information, and the image information; first control means for generating the synthetic image on the basis of the display information, and making the display means display the synthetic image; and second control means for generating synthetic voice of the respective voices collected by the respective microphones on the basis of the display information, and making a speaker output the synthetic voice.

Description

  The present invention relates to a conference system, an image display device, and an image sound processing method.

2. Description of the Related Art Conventionally, as a conference system used when a conference is performed, a configuration is known in which each display screen of each terminal used for the use of each participant in the conference is simultaneously displayed (divided display) on a screen (for example, , See Patent Document 1).
The conference system described in Patent Document 1 includes a plurality of computers (terminals) provided for use by each participant of a conference, a host computer (server) provided for use by a conference host, and a projector. It has a configuration connected by a network.
The server receives each image data handled by each terminal (image data related to each display screen of each terminal) via the network, and then generates a composite image (divided display screen) by combining the images based on each image data. Then, the composite image data related to the composite image is transmitted to the projector.
The projector receives the composite image data via the network and displays a composite image based on the composite image data on the screen.

JP 2010-193521 A

By the way, in the meeting place, referring to the composite image on the screen, the explanation is mainly based on the participant (the material created by the self included in the composite image (the image handled by the terminal device used by the user) included in the composite image). Participants (hereinafter referred to as main speakers) who perform
However, in the conference system described in Patent Document 1, since each participant's material (each image handled by each terminal device) is divided and displayed on the screen, the other speakers can display the main speaker on the screen. It is difficult to determine which image is used for the explanation among the images that have been displayed.
In addition, when the main speaker is explaining, if some of the other participants have a conversation or the like, the voice of the main speaker is difficult to hear.
From the above, the conference system described in Patent Document 1 has a problem that it is difficult to improve convenience.

  An object of the present invention is to provide a conference system, an image display device, and an image sound processing method capable of improving convenience.

  The conference system of the present invention includes a plurality of terminal devices and an image display device connected to be communicable with the plurality of terminal devices, and displays the combined image obtained by combining the images handled by the plurality of terminal devices. In the conference system to be displayed on the device, the terminal device includes: a sound collection unit that collects ambient sounds; an image handled by the terminal device in the synthesized image; and an image handled by the other terminal device Input receiving means for receiving an input of a display state; audio information relating to the sound collected by the voice collecting means; display information relating to the display state; and image information relating to an image handled by the terminal device are transmitted to the image display device. The image display device includes a display unit that displays an image, a reception unit that receives the audio information, the display information, and the image information, and the table. A first control unit configured to generate the composite image by combining the images based on the image information from the plurality of terminal devices based on the information, and to display the generated composite image on the display unit; Based on the display information, and synthesizing each voice based on the voice information from the plurality of terminal devices to generate a synthesized voice, and generating the synthesized voice as the voice output means And a second control means for outputting the output.

In the present invention, the conference system includes a plurality of terminal devices and image display devices configured as described above. For this reason, as shown below, each image handled by each terminal device is displayed on the image display device, and each sound collected by each sound collecting means such as a microphone of each terminal device is output to the image display device. Can be made.
For example, when one of the participants inputs the display state of the image handled by the terminal device used by the participant and the image handled by another terminal device by the input means of the terminal device used by the participant, the input acceptance of the terminal device is accepted. The means accepts an input of the display state (input acceptance step).
Here, the display state means, for example, the display position and display size of each image handled by each terminal device in a composite image (divided display screen).
And each transmission control means of a plurality of terminal devices displays the display state in addition to the sound information output from each sound collecting means such as a microphone provided for each terminal device and the image information related to the image handled by its own terminal device. Display information (sent only by the terminal device receiving the display state) is transmitted to the image display device (transmission step).

On the other hand, in the image display device, the receiving means receives the audio information, display information, and image information transmitted from each terminal device (reception step).
Further, the first control unit synthesizes the images based on the image information from the terminal devices based on the display information (display position and display size of each image) and displays the combined image on the display unit ( First control step).
Further, the second control means synthesizes each voice based on each voice information from each terminal device based on the display information, and outputs the synthesized voice to a voice output means such as a speaker (second control step).
For example, the first and second control means may display the display information “two images handled by the first and second terminal devices among the images handled by the plurality of terminal devices are divided into left and right parts for use by the main speaker. The display position and display size of the image handled by the first terminal device provided are “left” and “large”, and the display position and display size of the image handled by the second terminal device provided for use by other participants are If the information indicates “right” and “small”, a synthesized image and synthesized speech are generated as shown below.

That is, based on the display information, the first control means enlarges an image based on the image information transmitted from the first terminal device (material of the main speaker, etc.) to the left, and an image transmitted from the second terminal device. A composite image in which (other participants' materials, etc.) is reduced to the right is generated.
Further, the second control means increases the output level of the voice (the voice of the main speaker) based on the voice information transmitted from the first terminal device based on the display information, and the voice transmitted from the second terminal device. The output level of the voice based on the information (voices of other participants) is lowered, and the adjusted voices are synthesized to generate a synthesized voice.

From the above, by inputting the display state as appropriate using the terminal device, the main speaker's material (the image handled by the terminal device used for the main speaker's use) is synthesized (divided display screen). It can be set to any display position inside, any display size, etc. For this reason, other participants can easily determine which image the main speaker is explaining.
In addition, each participant's voice is collected by each microphone, etc., and as described above, a synthesized speech in which the voice of the main speaker is emphasized with respect to the voices of other participants can be generated and output. When the explanation is given, the voice of the main speaker (voice through speakers, etc.) should be easy to hear even if some of the other participants are talking. it can.
Therefore, the convenience can be improved.
In addition, since the image display apparatus itself generates a synthesized image and synthesized speech, a conventional server is not required, and the configuration of the conference system can be simplified.

  The conference system of the present invention includes a plurality of terminal devices, an image display device, the plurality of terminal devices and an information processing device connected to be communicable with the image display device, and each of the plurality of terminal devices handles A conference system that displays a composite image obtained by combining images on the image display device, wherein the terminal device includes a sound collection unit that collects ambient sounds, and an image handled by the terminal device in the composite image. , An input receiving unit that receives an input of a display state with an image handled by the other terminal device, voice information about the sound collected by the voice collecting unit, display information about the display state, and the terminal device Transmission control means for transmitting image information related to an image to the information processing apparatus, and the information processing apparatus receives the images from the plurality of terminal devices based on the display information. First control means for synthesizing each image based on the report to generate the synthesized image, and based on the display information, synthesizing the sounds based on the audio information from the plurality of terminal devices, A second control means for generating; and a third control means for transmitting the synthesized image information relating to the synthesized image and the synthesized voice information relating to the synthesized voice to the image display device via the communication path, and the image display. The apparatus includes display means for displaying the synthesized image based on the synthesized image information from the information processing apparatus, and voice output means for outputting the synthesized voice based on the synthesized voice information from the information processing apparatus. It is characterized by.

In the present invention, the conference system includes a plurality of terminal devices, image display devices, and information processing devices configured as described above.
In other words, the conference system of the present invention omits a part of the functions of the image display device (a function of generating a synthesized image and synthesized speech) in the above-described conference system that does not include the information processing device, and Some of the functions are provided.
Therefore, the same effect as the above-described conference system that does not include the information processing apparatus can be enjoyed.
In addition, since the generation of the synthesized image and the synthesized voice is executed by the information processing apparatus instead of the image display apparatus, it is not necessary to provide a separate function for generating the synthesized image and synthesized voice in the image display apparatus. It is possible to construct a conference system using

In the conference system of the present invention, the display state includes a display size of each image handled by the plurality of terminal devices, and the second control unit determines whether each image is based on the display size of each image. It is preferable to adjust the output level of each voice corresponding to the above and generate the synthesized voice.
In the present invention, since the display state includes the display size described above, if the first and second control means generate a synthesized image and synthesized speech based on the display size, the material of the main speaker (the main speaker's material) It is possible to emphasize the image) and voice of the terminal device used for use with respect to other participants' materials and voice.
Therefore, it becomes easy to view the material and voice of the main speaker, and the convenience can be improved.

The image display device of the present invention is an image display device that is communicably connected to a plurality of terminal devices and displays a combined image in which the images handled by the plurality of terminal devices are combined, and the image display device includes: Display means for displaying an image; audio information relating to sound collected by each of the plurality of terminal devices; display information relating to a display state of an image handled by each of the plurality of terminal devices in the synthesized image; Receiving means for receiving image information handled by each of the terminal devices, and generating the composite image by combining the images based on the image information from the plurality of terminal devices based on the display information, First display means for displaying the synthesized image on the display means, sound output means for outputting sound, and each of the plurality of terminal devices based on the display information. And combines the audio based on the voice information to produce synthesized speech, the synthesized speech generated, characterized in that it comprises a second control means for outputting to the audio output means.
Since the image display device of the present invention is used in the conference system described above, it can enjoy the same operations and effects as the conference system described above.

The video / audio processing method of the present invention includes a plurality of terminal devices and an image display device connected to be communicable with the plurality of terminal devices, and combines the synthesized images obtained by combining the images handled by the plurality of terminal devices. An audio / video processing method using a conference system to be displayed on an image display device, wherein the terminal device displays a display state of an image handled by the terminal device and an image handled by another terminal device in the synthesized image. An input receiving step for receiving an input, and audio information related to the sound collected by the terminal device by the sound collecting means, display information related to the display state, and image information related to an image handled by the terminal device are stored in the image display device. A transmission control step of transmitting, a reception step of receiving the audio information, the display information, and the image information by the image display device; and the image display device of A first control step of generating the composite image by combining the images based on the image information from the plurality of terminal devices based on the display information, and displaying the generated composite image; and the image display A second control step in which an apparatus synthesizes each voice based on each voice information from the plurality of terminal devices based on the display information to generate a synthesized voice, and outputs the generated synthesized voice; It is characterized by including.
Since the audio / video processing method of the present invention is a method using the above-described conference system, it can enjoy the same operations and effects as the above-described conference system.

The image / audio processing method of the present invention is an image / audio processing method for an image display device that is communicably connected to a plurality of terminal devices and displays a combined image obtained by combining the images handled by the plurality of terminal devices, Audio information relating to the sound collected by each of the plurality of terminal devices, display information relating to a display state of an image handled by each of the plurality of terminal devices in the synthesized image, and images handled by each of the plurality of terminal devices A receiving step for receiving information; and a method for generating the composite image by combining the images based on the image information from the plurality of terminal devices based on the display information, and displaying the generated composite image. 1 control step and, based on the display information, synthesize voices based on the voice information from the plurality of terminal devices to generate a synthesized voice, A second control step of outputting the formed sound, characterized in that it comprises a.
Since the image / audio processing method of the present invention is a method implemented by the above-described image display device, it can enjoy the same operations and effects as the above-described image display device.

The block diagram which shows the conference system in 1st Embodiment. The block diagram which shows the structure of the terminal device in 1st Embodiment. FIG. 2 is a block diagram illustrating a configuration of a projector according to the first embodiment. The block diagram which shows the structure of the audio | voice processing means in 1st Embodiment. The flowchart explaining the image audio processing method in 1st Embodiment. The figure which shows an example of the setting window in 1st Embodiment. The block diagram which shows the conference system in 2nd Embodiment. The block diagram which shows the structure of the projector in 2nd Embodiment. The block diagram which shows the structure of the server apparatus in 2nd Embodiment. The flowchart explaining the image audio processing method in 2nd Embodiment.

[First embodiment]
DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, a first embodiment of the invention will be described with reference to the drawings.
[Conference system configuration]
FIG. 1 is a block diagram showing a conference system 1.
The conference system 1 is a system that is used when a conference is performed, and simultaneously displays (divides and displays) each display screen of each terminal that is used by each participant in the conference.
The conference system 1 has a configuration in which a plurality of terminal devices 2 and a projector 3 as an image display device are connected via a predetermined communication path.

In this embodiment, for convenience of explanation, as shown in FIG. 1, the number of terminal devices 2 is four (first to fourth terminal devices 2A to 2D), but the number is not limited to four. Other numbers may be used.
Further, in the present embodiment, as shown in FIG. 1, a local area network LAN (hereinafter referred to as a network LAN) is adopted as the communication path. Depending on the situation, other communication paths may be adopted. The communication method employed in the conference system 1 may be wired communication or wireless communication.

[Configuration of terminal device]
FIG. 2 is a block diagram illustrating a configuration of the terminal device 2.
The terminal device 2 is configured by a personal computer that is used by each participant of the conference (four people in this embodiment).
In addition, each terminal device 2A-2D in this embodiment has the same structure.
As shown in FIG. 2, the terminal device 2 includes a memory 22 such as a hard disk, a CPU (Central Processing Unit) 21 that executes a program stored in the memory 22, and a network interface (network I / F) 23. An input unit 24, an image processing unit 25, a video random access memory (VRAM) 26, a display 27, a microphone 28 as an audio collection unit, an audio processing unit 29, and the like.

The network interface 23 is an interface for connecting to a network LAN.
The input unit 24 includes a mouse, a keyboard, and the like, and outputs an operation signal corresponding to the operation by the participant.
The image processing means 25 is composed of, for example, a GPU (Graphics Processing Unit) or the like, and displays an image to be displayed on the display 27 using the VRAM 26 as a buffer under the control of the CPU 21.
The microphone 28 collects sounds around the terminal device 2 (particularly, voices of participants who use the terminal device 2) and outputs audio signals.
The audio processing means 29 converts the analog audio signal output from the microphone 28 into digital audio data (audio information).

As shown in FIG. 2, the CPU 21 executes a program stored in the memory 22, and as shown in FIG. 2, as an input receiving means 211, a GUI (Graphical User Interface) control means 212, a communication control means 213 as a transmission control means, and the like. Function.
Based on the operation signal from the input unit 24, the input receiving unit 211 recognizes an operation on the input unit 24 by the participant.
The GUI control unit 212 controls the operation of the image processing unit 25 to display a setting window W described later on the display 27.
The communication control unit 213 establishes a connection with the projector 3 via the network LAN and transmits / receives information to / from the projector 3 via the network LAN.
For example, the communication control unit 213 includes the above-described audio data, information recognized by the input receiving unit 211 (hereinafter, display data (display information)), and image data handled by the terminal device 2 (an image displayed on the display 27). Image data (excluding the setting window W) (image data stored in the VRAM 26)) is transmitted to the projector 3.

[Configuration of projector]
FIG. 3 is a block diagram showing the configuration of the projector 3.
The projector 3 projects an image and displays a projected image on the screen.
As shown in FIG. 3, the projector 3 includes a memory 32 including a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and a CPU 31 that executes a program stored in the memory 32. A network interface (network I / F) 33, an image processing means 34, a VRAM 35, an image projection means 36 as a display means, an audio processing means 37, a speaker 38 as an audio output means, and the like.

The network interface 33 is an interface for connecting to a network LAN.
The image processing means 34 is composed of, for example, a GPU or the like, and projects an image on the image projection means 36 using the VRAM 35 as a buffer under the control of the CPU 31.
The image projection means 36 is composed of an optical system used in various general projectors, and includes a light source device, a light modulation device such as a liquid crystal panel that modulates a light beam emitted from the light source device, and a light modulation device. The projection lens etc. which project the modulated light beam (image) are comprised.

FIG. 4 is a block diagram showing the configuration of the audio processing means 37.
The voice processing means 37 synthesizes the voices of the participants collected by the microphones 28 of the terminal devices 2 based on the voice data transmitted from the terminal devices 2 under the control of the CPU 31. The synthesized voice synthesized is output through the speaker 38.
As shown in FIG. 4, the voice processing unit 37 includes a plurality of voice amplification units 371, a voice synthesis unit 372, and the like.
In the present embodiment, since the number of the terminal devices 2 used in the conference system 1 is four, the number of the audio amplifying units 371 is also four (first to fourth audio amplifying units 371A to 371D). The number is not limited to four, and may be provided according to the number of terminal devices 2.

The audio amplifying unit 371 includes an amplifier and the like, converts the audio data transmitted from the target terminal device 2 among the terminal devices 2 into an analog audio signal, and converts it under the control of the CPU 31. The signal level (sound output level) of the audio signal is adjusted (amplified at various amplification levels).
The first to fourth audio amplification units 371A to 371D correspond to the first to fourth terminal devices 2A to 2D, respectively.
The voice synthesis unit 372 generates a synthesized voice signal by synthesizing each voice signal whose amplification degree is adjusted by each voice amplification unit 371, and outputs a synthesized voice based on the synthesized voice signal via the speaker 38. .

As shown in FIG. 3, the CPU 31 functions as a communication control unit 311, a first control unit 312, a second control unit 313, and the like as a reception unit by executing a program stored in the memory 32.
The communication control unit 213 establishes a connection with each terminal device 2 via the network LAN and transmits / receives information to / from each terminal device 2 via the network LAN.

The first control unit 312 controls the operation of the image processing unit 34 based on the display data transmitted from the terminal device 2, combines the images based on the image data from the terminal devices 2, and generates the combined image. Is displayed on the screen.
The second control unit 313 controls the operation of the voice processing unit 37 based on the display data transmitted from the terminal device 2, synthesizes each voice based on each voice data from each terminal device 2, and the synthesized voice. Is output from the speaker 38.

[Operation of the conference system]
Next, the operation (video / audio processing method) of the conference system 1 described above will be described.
FIG. 5 is a flowchart for explaining the image / audio processing method.
In the following, for convenience of explanation, it is assumed that the connection between the terminal device 2 and the projector 3 via the network LAN has already been established.
In the following, for convenience of explanation, the terminal devices 2 that are connected to the projector 3 via the network LAN are referred to as first and second terminal devices 2A and 2B, and third and fourth terminal devices 2C and 2D. It is assumed that no connection has been established.
In other words, the number of participants in the conference using the conference system 1 is the first participant using the first terminal device 2A and the second participant using the second terminal device 2B.

FIG. 6 is a diagram illustrating an example of the setting window W.
For example, when the first participant performs an input operation to “display the setting window W” using the input unit 24 of the first terminal device 2A, the input receiving unit 211 recognizes the input operation (step S101).
After step S101, the GUI control unit 212 controls the operation of the image processing unit 25 to display the setting window W shown in FIG. 6 on the display 27 (step S102).

As shown in FIG. 6, the setting window W includes first to third windows W1 to W3.
The first window W1 is an area in which all participants who use the conference system 1 recognize the participant.
For each participant, when the connection between each terminal device 2 and the projector 3 via the network LAN is established, each terminal device 2 that has already been connected (in this embodiment, the first and second terminal devices). 2A, 2B) information is transmitted from the projector 3 to each terminal device 2. Then, each terminal device 2 (each CPU 21) recognizes each participant set in advance for the terminal device 2 based on the transmitted information (information on each terminal device 2 that has been established), and Participant images relating to each participant are displayed in one window W1. For example, in the present embodiment, the first terminal device 2A recognizes that the connection between the second terminal device 2B and the projector 3 has been established based on the information transmitted from the projector 3, and the second terminal device Recognize the participant set for 2B.
In the present embodiment, as described above, since each participant is the first and second participants, the first window W1 includes the first participant image FP1 corresponding to the first participant, and A second participant image FP2 corresponding to the second participant is displayed.

The second window W2 is an area for selecting the display number of images handled by each terminal device 2 in the composite image to be displayed on the projector 3.
Specifically, as shown in FIG. 6, first to third selection images FC1 to FC3 are displayed in the second window W2.
The first selection image FC1 is an image for selecting that the images handled by any one of the terminal devices 2 are displayed on the full screen (the number of displays is 1) among the terminal devices 2 that have been established.
The second selection image FC2 is an image for selecting that the images handled by the two terminal devices 2 are displayed on the left and right in two screens (the number of displays is 2) among the terminal devices 2 that have been established.
The third selection image FC3 is an image for selecting that the images handled by the four terminal devices 2 are displayed on four screens on the top, bottom, left, and right among the terminal devices 2 that have already been connected (the number of displays is four).

The third window W <b> 3 is an area for selecting a display position and a display size of each image handled by each terminal device 2 that has already been connected in the composite image displayed on the projector 3.
For example, when the second selected image FC2 is selected by an operation (mouse operation) on the input means 24 by the first participant, as shown in FIG. 6, the projector 3 displays in the third window W3. Two areas Ar1 and Ar2 corresponding to the two screens to be displayed are displayed.
Although specific illustration is omitted, when the first selection image FC1 is selected, only one area corresponding to one screen displayed by the projector 3 is displayed in the third window W3. When the third selection image FC3 is selected, four areas corresponding to four screens displayed by the projector 3 are displayed in the third window W3.

Then, when the first participant selects the display position of each image handled by each terminal device 2 that has already been established, the first participant operates the input unit 24 to display each participant image displayed in the first window W1. One of the participant images is dragged and dropped onto any one of the regions displayed in the third window W3.
For example, in the example shown in FIG. 6A, the first participant image FP1 is dragged and dropped in the left area Ar1 in the third window W3, and the second participant image FP2 is dragged and dropped in the right area Ar2. Shows the state.
By selecting the display position in this way, an image handled by the first terminal device 2A (hereinafter also referred to as “first image”) is displayed on the left side, and an image handled by the second terminal device 2B (hereinafter referred to as “first image”). In this description, it is instructed to display “second image” on the right side.

In addition, when the first participant selects the display size of each image handled by each terminal device 2 that has already been established, the first participant operates the input unit 24 to select which of the areas displayed in the third window W3. Place the cursor (not shown) at the edge of the area, and drag and drop to change the size of the area.
For example, the example shown in FIG. 6B shows a state in which the left area Ar1 in the third window W3 is enlarged by the above operation, and the right area Ar2 is reduced in accordance with the operation.
By selecting the display size in this way, the image handled by the first terminal device 2A (such as the material of the first participant (main speaker)) is displayed in a large size, and the image handled by the second terminal device 2B (other participation) The user's material) is displayed in a small size.

After step S102, when the input means 24 is operated by the first participant as described above, the input receiving means 211 is handled by each terminal device 2 with established connection in the composite image displayed on the projector 3. Input of the display state (display number, display position, display size) of each image is received (step S103: input reception step).
Then, the input receiving unit 211 stores display data relating to the display state (display number, display position, display size) in the memory 22.
For example, when the display state shown in FIG. 6B is input, the input reception unit 211 sets the display number to “2”, sets the display position of the image handled by the first terminal device 2A to “left”, Display data indicating that the display position of the image handled by the second terminal device 2B is “right” and the display size (ratio) of each image handled by the first and second terminal devices 2A and 2B is “A: B”. It is stored in the memory 22.
In the present embodiment, “A” represents the ratio of the first image in the composite image displayed on the projector 3, and “B” represents the ratio of the second image. As this ratio, the ratio of the width and height of the first and second images, the length of the diagonal line, the ratio of the area, and the like can be adopted. The display data related to the display size may be information other than the ratio as long as it is information indicating the display size of the first and second images. For example, the display data relating to the display size may be information indicating the width, height, diagonal length, area, and the like of the first and second images, or information indicating the difference between them.

After step S103, when the first participant performs an input operation to “display the composite image on the projector 3 in the input display state” with the input unit 24 of the first terminal device 2A, the input receiving unit 211 Recognize input operations.
Then, the communication control unit 213 transmits the display data stored in the memory 22 to the projector 3 via the network LAN (step S104: transmission control step).
Further, the CPU 21 controls the operation of the image processing means 25, stops displaying the setting window W, and causes the display 27 to display a normal image.

After step S104, the communication control means 311 of the projector 3 receives display data from the first terminal device 2A via the network LAN (step S105: reception step).
Then, the CPU 31 stores the received display data in the memory 32.
After step S105, the communication control unit 311 makes a transmission request for image data and audio data to the first and second terminal apparatuses 2A and 2B via the network LAN (step S106).

After step S106, when the communication control means 213 of the first and second terminal devices 2A and 2B receives a data transmission request from the projector 3 (steps S107 and S108), an image is sent to the projector 3 via the network LAN. Data and audio data are transmitted (steps S109 and S110: transmission control step).
Here, the image data is image data related to an image (display screen) currently displayed on each display 27 in the first and second terminal devices 2A and 2B. Specifically, the image data is currently stored in each VRAM 26. Stored image data.
In addition, the voice data is collected at each microphone 28 by the first and second terminal devices 2A and 2B at the present time, and the voice signal from each microphone 28 is processed by each voice processing. The voice data converted by the means 29.

After steps S109 and S110, the communication control means 311 of the projector 3 receives image data and audio data from the first and second terminal devices 2A and 2B via the network LAN (step S111: reception step).
Then, the CPU 31 specifies the transmission source (terminal device 2) by identification information (IP address or the like) for identifying the terminal device 2, and associates the received image data and audio data with the transmission source terminal device 2 in the memory. 22 is stored.
In the following, among the image data and audio data stored in the memory 22, data whose transmission source is the first terminal device 2A will be referred to as first image data and first audio data, and the transmission source will be the second terminal. The data that is the device 2B is described as second image data and second audio data.

After step S111, the first control unit 312 controls the operation of the image processing unit 34 based on the display data stored in the memory 22, generates a composite image (step S112), and displays the composite image on the screen. (Step S113: first control step).
For example, when the display state shown in FIG. 6B is input in step S103, the first control unit 312 generates and displays a composite image as shown below based on the display data.
That is, the first control means 312 sets the display position of the image (first image) based on the first image data to “left” based on the display data (display number, display position), and the image based on the second image data. The display position of (second image) is “right”.
In addition, the first control means 312 has a composite image (a first image (material of the main speaker (first participant), etc.) having the display size of the first and second images as a ratio based on the display data. A second image (a composite image larger than other participants (second participants)) is generated (composite image data related to the composite image is generated on the VRAM 35), and the composite image is displayed on the screen.

Further, after step S111 (in FIG. 5, for convenience of explanation, the processing after step S112, S113 is performed), the second control unit 313 performs the audio processing unit 37 based on the display data stored in the memory 22. The operation is controlled to generate synthesized speech (step S114), and the synthesized speech is output from the speaker 38 (step S115: second control step).
For example, when the display state shown in FIG. 6B is input in step S103, the second control unit 313 generates synthesized speech as shown below based on the display data.
That is, the second control means 313 causes the first audio amplifying unit 371A to process the first audio data based on the display data (the display size ratio of each image), and the audio after the first audio data is converted. Amplify the signal. In the present embodiment, the first and second images are included in the composite image, and the first image is larger than the second image. For this reason, the second control unit 313 causes the first audio amplification unit 371A to amplify the audio signal with an amplification degree larger than the default value.
In addition, the second control unit 313 causes the second audio amplification unit 371B to process the second audio data based on the display data (the display size ratio of each image), and converts the second audio data into audio. Amplify the signal. In the present embodiment, the first and second images are included in the composite image, and the second image is smaller than the first image. For this reason, the second control unit 313 causes the second audio amplification unit 371B to amplify the audio signal with an amplification factor smaller than the default value. In other words, the second control means 313 makes the sound corresponding to the first image larger than the sound corresponding to the other image, as the first image is displayed larger than the other images in the composite image.

Then, the voice synthesizer 372 generates a synthesized voice signal by synthesizing the voice signals amplified by the first and second voice amplifiers 371A and 371B, and the synthesized voice based on the synthesized voice signal is converted into the speaker 38. Output via.
That is, when the display state shown in FIG. 6B is input in step S103, the second control means 313 sends the voice of the main speaker (first participant) to other participants (second participants). ) To generate and output a synthesized voice emphasized with respect to the voice.
Thereafter, steps S109 to S115 are sequentially and repeatedly executed, and the display screens of the first and second terminal devices 2A and 2B at the present time are displayed on the screen as composite images, and the first and second terminal devices 2A at the present time are displayed. , 2B, the voices of the participants collected by the microphones 28 are output as synthesized speech from the speaker 38.

The first embodiment described above has the following effects.
In the present embodiment, since the conference system 1 includes a plurality of terminal devices 2 and projectors 3, by appropriately inputting the display state using the terminal device 2, the material of the main speaker (use of the main speaker) Can be set to an arbitrary display position, an arbitrary display size, or the like in the composite image (divided display screen). For this reason, other participants can easily determine which image the main speaker is explaining.
In addition, the voice of each participant is collected by each microphone 28, and a synthesized speech in which the voice of the main speaker is emphasized with respect to the voices of other participants can be generated and output. Even when some of the other participants are talking, the voice of the main speaker can be easily heard.
Therefore, the convenience can be improved.
In addition, since the projector 3 itself generates the synthesized image and the synthesized voice, the configuration of the conference system 1 can be simplified without requiring a conventional server.

Furthermore, since the display state includes the display size described above, if the first and second control means 312 and 313 generate a synthesized image and synthesized speech based on the display size, the material of the main speaker (main speaker) Image and voice handled by the terminal device 2 provided for use of the other participants can be emphasized with respect to materials and voices of other participants.
Therefore, it becomes easy to view the material and voice of the main speaker, and the convenience can be improved.

[Second Embodiment]
Next, 2nd Embodiment of this invention is described based on drawing.
In the following description, the same configurations and the same members as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted or simplified.
FIG. 7 is a block diagram showing the conference system 1 in the second embodiment.
FIG. 8 is a block diagram illustrating a configuration of the projector 3 according to the second embodiment.
FIG. 9 is a block diagram illustrating a configuration of the server device 4 according to the second embodiment.
In the present embodiment, as shown in FIGS. 7 to 9, the server device 4 is added to the configuration of the conference system 1 in addition to the terminal device 2 and the projector 3 as compared with the first embodiment, and the projector 3 The only difference is that some of the functions of the CPU 31 are omitted and the partial functions are added to the CPU 41 of the server device 4.

The server device 4 is composed of a personal computer similar to the terminal device 2, and as shown in FIG. 9, a CPU 41, a memory 42, a network interface (network I / F) 43, an input means 44, an image processing means 45, A VRAM 46, a display 47, and sound processing means 48 are provided.
The CPU 41 executes the program stored in the memory 42, thereby, as shown in FIG. 9, the communication control unit 411 as the third control unit, and the first in the projector 3 (CPU 31) described in the first embodiment. , Function as first and second control means 412 and 413 similar to the second control means 312 and 313.
The sound processing means 48 has the same configuration as the sound processing means 37 of the projector 3 described in the first embodiment.

Next, a video / audio processing method according to the second embodiment will be described.
FIG. 10 is a flowchart for explaining a video / audio processing method according to the second embodiment.
In the following, for convenience of explanation, it is assumed that the connection between the terminal device 2 and the server device 4 via the network LAN and the connection between the projector 3 and the server device 4 via the network LAN have already been established.
In the following, similarly to the first embodiment, the terminal devices 2 that are connected to the server device 4 via the network LAN are referred to as first and second terminal devices 2A and 2B.

In the image / audio processing method of this embodiment, as described above, the projector 3 (CPU 31) has a part of the functions of the projector 3 (CPU 31), and the projector 3 (CPU 41) described in the first embodiment has been described. The server device 4 (CPU 41) executes a part of the processing executed by the CPU 31).
That is, as shown in FIG. 10, the server apparatus 4 (CPU 41) executes steps S105, S106, S111, S112, and S114 that were executed by the projector 3 (CPU 31) in the first embodiment.

In this embodiment, in step S112, the image processing unit 45 generates composite image data under the control of the first control unit 412. In step S114, the voice processing unit 48 generates a synthesized voice signal under the control of the second control unit 413, and converts the synthesized voice signal (analog) into digital data (synthesized voice data).
Further, in each terminal device 2, the transmission / reception partner at the time of executing steps S 104 and S 107 to S 110 becomes the server device 4.

Then, the communication control unit 411 of the server device 4 transmits the synthesized image data and synthesized voice data generated in steps S112 and S114 to the projector 3 via the network LAN (step S201).
On the other hand, the communication control means 311 of the projector 3 receives the synthesized image data and synthesized audio data from the server device 4 via the network LAN (step S202).
Then, the CPU 31 stores the synthesized image data on the VRAM 35, displays a synthesized image based on the synthesized image data on the screen (step S203), and analog-synthesizes the synthesized voice data by the voice processing unit 37A. The sound is converted into a sound signal, and the synthesized sound based on the synthesized sound signal is output from the speaker 38 (step S204).

According to the second embodiment described above, there are the following effects in addition to the same effects as in the first embodiment.
In the present embodiment, since the generation of the composite image and the synthetic voice is executed by the server device 4 instead of the projector 3, it is not necessary to provide the projector 3 with a function for generating the synthetic image and the synthetic voice. It is possible to construct a conference system 1 using

It should be noted that the present invention is not limited to the above-described embodiments, and modifications, improvements, and the like within the scope that can achieve the object of the present invention are included in the present invention.
In each said embodiment, 2A of 1st terminal devices performed step S101-S104, but if a participant other than a 1st participant operates the input means 24 of the other terminal device 2, another terminal device 2 executes steps S101 to S104.
In each of the above-described embodiments, the sound corresponding to the image whose display size is increased is emphasized with respect to the sound corresponding to the other image. However, the sound corresponding to the image positioned at a predetermined display position is not limited thereto. You may comprise so that it may emphasize with respect to the audio | voice corresponding to another image.

In the second embodiment, the terminal device 2 has the function of executing steps S101 to S103 (the input receiving unit 211 and the GUI control unit 212). However, the present invention is not limited to this, and the server device 4 has the function. It does not matter.
In each of the above embodiments, the projector 3 is employed as the image display device according to the present invention. However, the present invention is not limited thereto, and a liquid crystal display, a plasma television, an organic EL (Electro Luminescence), or the like may be employed.

In each of the embodiments described above, the amplification level of the first audio amplification unit 371A is larger than the default value, and the amplification level of the second audio amplification unit 371B is smaller than the default value. The volume may be differentiated by increasing (or decreasing) the other amplification degree.
In each of the embodiments described above, the volume adjustment is performed by the projector 3 or the server device 4. However, the present invention is not limited to this, and the volume adjustment is performed by each terminal device 2. A configuration in which the server device 4 synthesizes may be adopted.

  The present invention can be used for a conference system that performs a conference using an image display device such as a projector.

  DESCRIPTION OF SYMBOLS 1 ... Conference system, 2 ... Terminal device, 3 ... Projector, 4 ... Server device, 28 ... Microphone (voice collecting means), 36 ... Image projection means (display means), 38 ... speaker (audio output means), 211 ... input receiving means, 213 ... communication control means (transmission control means), 312, 412 ... first control means, 313,413 ... 2 control means, 411 ... communication control means (third control means), LAN ... network (communication path), S103 ... input acceptance step, S104, S109, S110 ... transmission control step, S105, S111: reception step, S113: first control step, S115: second control step.

Claims (6)

A conference system, comprising: a plurality of terminal devices; and an image display device connected to be communicable with the plurality of terminal devices, wherein a composite image obtained by combining the images handled by the plurality of terminal devices is displayed on the image display device. There,
The terminal device
Voice collecting means for collecting ambient sounds;
An input receiving means for receiving an input of a display state of an image handled by the terminal device and an image handled by another terminal device in the composite image;
Transmission control means for transmitting to the image display device audio information relating to the sound collected by the sound collecting means, display information relating to the display state, and image information relating to an image handled by the terminal device;
The image display device includes:
Display means for displaying an image;
Receiving means for receiving the audio information, the display information, and the image information;
A first control unit configured to generate the composite image by combining the images based on the image information from the plurality of terminal devices based on the display information, and to display the generated composite image on the display unit; ,
Audio output means for outputting audio;
A second control unit configured to generate a synthesized voice by synthesizing each voice based on the voice information from the plurality of terminal devices based on the display information, and to output the generated synthesized voice to the voice output unit; A conference system characterized by comprising: A composite image comprising a plurality of terminal devices, an image display device, and the information processing devices communicably connected to the plurality of terminal devices and the image display device, and a composite image obtained by combining the images handled by the plurality of terminal devices A conference system for displaying on the image display device,
The terminal device
Voice collecting means for collecting ambient sounds;
An input receiving means for receiving an input of a display state of an image handled by the terminal device and an image handled by another terminal device in the composite image;
Transmission control means for transmitting to the information processing apparatus audio information relating to the voice collected by the voice collecting means, display information relating to the display state, and image information relating to an image handled by the terminal device;
The information processing apparatus includes:
First control means for generating the composite image by combining the images based on the image information from the plurality of terminal devices based on the display information;
Second control means for generating synthesized speech by synthesizing each voice based on each of the voice information from the plurality of terminal devices based on the display information;
Comprising, via the communication path, synthetic image information relating to the synthetic image and synthetic voice information relating to the synthetic speech to the image display device;
The image display device includes:
Display means for displaying the composite image based on the composite image information from the information processing apparatus;
A conference system comprising: voice output means for outputting the synthesized voice based on the synthesized voice information from the information processing apparatus. In the conference system according to claim 1 or 2,
The display state is
Including the display size of each image handled by the plurality of terminal devices,
The second control means includes
Based on the display size of each image, the output level of each sound corresponding to each image is adjusted to generate the synthesized sound. An image display device that is communicably connected to a plurality of terminal devices and displays a composite image in which images handled by the plurality of terminal devices are combined,
The image display device
Display means for displaying an image;
Audio information relating to the sound collected by each of the plurality of terminal devices, display information relating to a display state of an image handled by each of the plurality of terminal devices in the synthesized image, and images handled by each of the plurality of terminal devices Receiving means for receiving information;
A first control unit configured to generate the composite image by combining the images based on the image information from the plurality of terminal devices based on the display information, and to display the generated composite image on the display unit; ,
Audio output means for outputting audio;
A second control unit configured to generate a synthesized voice by synthesizing each voice based on the voice information from the plurality of terminal devices based on the display information, and to output the generated synthesized voice to the voice output unit; An image display device comprising: A conference system comprising a plurality of terminal devices and an image display device connected to be communicable with the plurality of terminal devices, and causing the image display device to display a combined image obtained by combining the images handled by the plurality of terminal devices. A video / audio processing method used,
An input receiving step in which the terminal device receives an input of a display state of an image handled by the terminal device and an image handled by the other terminal device in the composite image;
A transmission control step in which the terminal device transmits to the image display device audio information relating to the sound collected by the sound collecting means, display information relating to the display state, and image information relating to an image handled by the terminal device;
A receiving step in which the image display device receives the audio information, the display information, and the image information;
The image display device generates a composite image by combining the images based on the image information from the plurality of terminal devices based on the display information, and displays the generated composite image Steps,
A second control step in which the image display device generates a synthesized speech by synthesizing each speech based on the speech information from the plurality of terminal devices based on the display information, and outputs the generated synthesized speech; When,
An audio / video processing method comprising: An image audio processing method of an image display device that is communicably connected to a plurality of terminal devices and displays a composite image in which images handled by the plurality of terminal devices are combined,
Audio information relating to the sound collected by each of the plurality of terminal devices, display information relating to a display state of an image handled by each of the plurality of terminal devices in the synthesized image, and images handled by each of the plurality of terminal devices A receiving step for receiving information;
A first control step of generating the composite image by combining the images based on the image information from the plurality of terminal devices based on the display information, and displaying the generated composite image;
A second control step of generating a synthesized voice by synthesizing voices based on the voice information from the plurality of terminal devices based on the display information, and outputting the generated synthesized voice;
An audio / video processing method comprising:

Images