JPH0746561A - Multmedia equipment - Google Patents

Abstract

PURPOSE:To easily identify the window of an opposite party receiving the voice from plural windows by a user. CONSTITUTION:When there is any opposite party receiving no voice higher than a prescribed level for prescribed time, a level detection circuit 22 instructs to display the opposite party with an icon on a display device 34 to a display control circuit 32 by detecting the level of the voice from the opposite party. When the voice higher than the prescribed level is received from the opposite party displayed with icon, the window display of the opposite party on the display device 34 is instructed to the display control circuit 32.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multimedia device for communicating multimedia (image, voice, data, etc.) with a plurality of parties via a communication line.

[0002]

2. Description of the Related Art Conventionally, a video conference system performs multimedia communication by multiplexing images, voice and data with a plurality of parties via a communication line, and mixes voices received from a plurality of parties into one. The image output from the speaker and received from a plurality of parties can be displayed on the multi-window.

[0003]

However, in the conventional example,
Currently, the images of all the opponents are displayed in the window regardless of whether or not audio is currently received. Further, the voices received from a plurality of parties are equally mixed and output regardless of the position of each window, so that the voices corresponding to any windows can be heard uniformly. For this reason, it is difficult for the user to understand which window the partner outputs as the sound output from the speaker. Also, displaying the images of all the opponents in the window puts a heavy burden on the device.

The present invention aims at presenting a multimedia device which solves such problems.

[0005]

A multimedia device according to a first aspect of the present invention performs multimedia communication with a plurality of partners via a communication line and displays a plurality of images received from the partner in a multi-window. A media device, which displays a voice level detecting means for detecting a voice level received from the other party, and an image of the other party who has not received a predetermined voice for a predetermined time based on the detection result of the voice level detecting means. R window means for displaying a window to be displayed as an icon.

A multimedia device according to the second invention is
A multimedia device for performing multimedia communication with a plurality of parties via a communication line and displaying received images in a multi-window, a means for reading the positional relationship of each window, and a received voice from the plurality of parties, And an output unit for outputting at a volume according to the positional relationship of the corresponding windows.

[0007]

By the above means, the person who has not made a voice for a predetermined time is excluded from the window display, so that the user can easily understand the window to be noticed, that is, the window of the person corresponding to the voice.

Also, the window of the other party corresponding to the voice can be easily understood by the user due to the strength relation of the voice output from each speaker.

[0009]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a schematic block diagram of the first embodiment of the present invention. 10 is a CPU that controls the entire apparatus,
Reference numeral 12 is an R storing a program executed by the CPU 10.
OM, 14 is a RAM for storing data used in the CPU 10, 16 is an operating device including a keyboard and a touch panel, and 18 is ITU-T (former CCITT) recommendation G. 722, a voice encoding / decoding circuit for performing encoding and decoding of a voice signal, 20 is a voice input microphone,
Reference numeral 22 is a level detection circuit that detects the level of the received voice, 24 is a voice mixing circuit that mixes the received voices of the other parties into one, and 26 is a speaker that outputs the voice.

28 is ITU-T Recommendation H.264. An image encoding / decoding circuit that executes encoding and decoding of an image signal in accordance with H.261, 30 is a camera for inputting an image, 32 is a multi-window display of images of a plurality of partners, a window display and an icon display. Is a display control circuit for switching between, and 34 is a display device including an LCD or a CRT for displaying an image.

Reference numeral 36 denotes ITU-T recommendation H.264. A demultiplexing circuit that multiplexes and separates images, voices, and data according to 221, 38 is a line control circuit that controls connection and communication with a communication line, and 40 is a clock that can read the current time.

FIG. 2 shows the structure of the audio frame supplied from the audio encoding / decoding circuit 18 to the audio mixing circuit 24. Voice signals from a plurality of partners are time-division multiplexed, and the voice signals of each partner are stored in time slots. In FIG. 2, 42 is a time slot for storing the voice signal of the partner # 1, 44 is a time slot for storing the voice signal of the partner # 2, 46 is a time slot for storing the voice signal of the partner # 3, and 48 is a partner #. 4 is a time slot for storing 4 audio signals. Figure 2 shows the case where there are four people,
The time slot 48 is followed by a time slot with the same repetition.

FIG. 3 shows an example of the screen of the display device 34 displayed during communication. In FIG. 3, 50 is a display device 34.
Screen, 52 is a window for displaying the image received from the other party # 1 on the screen 50, and 54 is the other party # on the screen 50
2 is a window for displaying an image received from # 2, 56 is a window for displaying an image received from partner # 3 on screen 50, and 58 is an icon for displaying an image received from partner # 4 on screen 50. FIG. 3 shows a case where there are four opponents, and a still image or a name indicating the opponent is displayed on the icon.

FIG. 4 shows the memory configuration of the RAM 14. 6
0 is a variable N that stores the number of the other party, 62 is a variable Tm that stores the maximum display time of the window of the other party who is not receiving voice, 64 is a variable T that stores the current time, 66 is the number of the other party Variable I, 68 is a variable V for storing the voice level of the other party I, 70 is a variable Ti (1), 7 for storing the time when the voice of the other party # 1 is received and the window is displayed.
2 is a variable Ti (2) for storing the time when the voice of the partner # 2 is received and the window is displayed, and 74 is a variable Ti for storing the time when the voice of the partner # 3 is received and the window is displayed.
(3) and 76 are variables Ti (N) for storing the time when the voice of the partner #N is received and the window is displayed.

FIG. 5 shows a flow chart of the program stored in the ROM 12. At the start of communication, the variable N is set to the number of the other party, the variable Tm is set to the maximum display time of the window of the other party who is not receiving voice, and the variable Ti (I) is set to the time to start communication with the other party I. To do.

The program is started each time the audio frame shown in FIG. 2 is received.

First, the current time is read from clock 40 into T (S1), and variable I is set to 1 (S2). If the variable I is less than or equal to the variable N, the process proceeds to S4, and if not, the process ends (S3).

In S4, the level detection circuit 22 detects the voice level from the opponent I and stores it in the variable V. If the variable V is zero, the process proceeds to S6, and if not, the process proceeds to S8 (S5). In S6, the variable T and the variable Ti are compared,
If the difference is larger than the variable Tm, proceed to S7, otherwise proceed to S10.

In step S7, the display control circuit 32 is instructed to display the image of the partner I on the display device 34 as an icon. In S8, the display control circuit 32 is instructed to display the image of the partner I on the display device 34 in a window.

The content of the variable T is set in the variable Ti (I) (S9), 1 is added to the variable I, and the process returns to S3 (S1).
0).

In step S7, even if the display control circuit 32 is instructed again to display the icon with respect to the other party who is currently displayed with the icon, only the icon is continuously displayed. , No change happens. Similarly, in S8, even if the display control circuit 32 is instructed again to display an image in the window for the other party who is currently displaying the image in the window, only the image is continuously displayed in the window. And no change happens.

According to the above-described embodiment, while monitoring the voice level of the communication partner at all times, the partner who is not uttering a voice is switched from the window display to the icon display for a predetermined time, so that the number of windows is reduced accordingly. The user has a window of interest in relation to the audio currently being received,
It can be easily distinguished. Further, since the number of windows displayed is reduced, the load of window display of the device is reduced.

In the above-mentioned embodiment, the ROM 12 and the RAM 14 are used as the storage device for storing the programs and data, but a floppy disk, a hard disk or a memory card may be used.

In the above embodiment, ITU-T Recommendation H.264 is used as the image encoding / decoding method. 261 is adopted, MP
EG (Moving Picture Coding)
Other image encoding / decoding methods such as Experts Group) may be used.

In the above embodiment, a moving image is used as the image, but it may be a still image or continuous still images.

In the above embodiment, ITU-T Recommendation H.264 is used as the multimedia multiplexing method. 221 is used,
A method of assigning a plurality of calls to images, voices, and data may be used.

In the above embodiment, the voices and images of a plurality of partners are multiplexed by the time division multiplexing method, but frequency multiplexing, phase difference multiplexing and packet multiplexing may be used.

In the above embodiment, the windows are displayed in a partially overlapped manner, but it is also possible to employ a display method in which there is no overlap such as screen division.

In the above embodiment, the icon displays a still image or name indicating the partner, but the icon may display a moving image of the partner.

In the above embodiment, it is judged that the voice is not received when the voice level of the other party is zero (0), but the voice is received when the voice level of the other party is smaller than the predetermined value. You may decide not to. Alternatively, among all the opponents, the predetermined number of people whose voice level is relatively low may be determined as the opponent who has not received the voice.

In the above embodiment, the variable Tm is set at the start of communication, but the user may change it via the operating device 16 during communication. Further, it is possible to set the variable Tm for each partner and give priority to display with an icon regardless of the voice level.

FIG. 6 shows a schematic block diagram of the second embodiment of the present invention. 110 is a CP that controls the entire apparatus
U, 112 are ROMs for storing programs executed by the CPU 110, 114 are RAMs for storing data used by the CPU 110, 116 is an operating device including a keyboard and a touch panel, 118 is ITU-T Recommendation G. 722, a voice encoding / decoding circuit that performs encoding and decoding of a voice signal, 120 is a voice input microphone, 122 is a voice amplifying circuit that amplifies the voice received from the voice encoding / decoding circuit 118, and 124L and 124R.
Is a voice mixing circuit that mixes the voices of a plurality of partners input from the voice amplification circuit 122 into one, and 126L is
The left speaker that outputs the sound input from the sound mixing circuit 124L and the right speaker 126R that outputs the sound input from the sound mixing circuit 124R.

Reference numeral 128 denotes ITU-T Recommendation H.264. An image encoding / decoding circuit that performs encoding and decoding of an image signal in accordance with H.261, 130 is a camera for inputting images, 132 is a display of images of a plurality of partners on a multi-window, and a display state of windows is read. Display control circuit,
Reference numeral 134 denotes a display device including an LCD or a CRT that displays an image.

Reference numeral 136 denotes ITU-T Recommendation H.264. A demultiplexing circuit 138 that multiplexes and separates images, voices, and data according to 221 is a line control circuit that controls connection and communication with a communication line.

The format of the audio frame sent from the audio encoding / decoding circuit 118 to the audio mixing circuits 124L and 124R is the same as that of the first embodiment shown in FIG.

FIG. 7 shows a display device 134 displayed during communication.
Of an example of the screen of the speaker 126
Shown with L and 126R. 7, 150 is a screen of the display device 134, 152 is a window for displaying an image received from the other party # 1 in the screen 150, 154 is a window for displaying an image received from the other party # 2 in the screen 150, 156 is a window for displaying an image received from the other party # 3 on the screen 150, and 158 is a screen 150
It is a window displaying an image received from the other party # 4.

D1 is the distance from the left end of the screen 150 to the center of the window 152 of the opponent # 1, D2 is the distance to the center of the window 154 of the opponent # 2, D3 is the distance to the center of the window 156 of the opponent # 3, D4 is the distance from the center of the window 158 of the opponent # 4, W is the distance from the left end to the right end of the screen 150, that is, the width of the screen 150. DL is the distance from the center of the left speaker 126L to the left end of the screen 150, and DR is the distance from the center of the right speaker 126R to the right end of the screen 150.

Note that FIG. 7 shows a case where there are four opponents, and each of the windows 152 to 158 can be moved within the screen 150 by operating the operating device 116.

FIG. 8 shows the memory configuration of the RAM 114. 160 is a variable N for storing the number of opponents, 162 is a variable w for storing the width W of the screen 150, 164 is a variable d (L) for storing the distance DL from the center of the left speaker 126L to the left end of the screen 150, 166. Is a variable d (R) that stores the distance DR from the center of the right speaker 126R to the right end of the screen 150, and 168 stores the amplification factor by which the audio amplification circuit 122 amplifies the audio input from the audio encoding / decoding circuit 118. Is a variable A to be executed. That is, the audio from the audio encoding / decoding circuit 118 multiplied by A is the audio mixing circuit 1
This corresponds to the total amount of sound output from 24L and 124R.

170 is a variable I for storing the other party's number, 1
Reference numeral 72 is a variable di that stores the distance (D1, D2, D3 or D4) from the left end of the screen 150 to the center of the window of the opponent I, and 174 is a voice output from the voice amplification circuit 122 to both voice mixing circuits 124L and 124R. A variable R (L) 176 for storing the ratio of the sound output to the sound mixing circuit 124L with respect to the total amount is a sound mixing circuit with respect to the total amount of sound output from the sound amplifying circuit 122 to both sound mixing circuits 124L and 124R. A variable R (R) that stores the ratio of the sound output to 124R.

FIG. 9 shows a flow chart of the program stored in the ROM 112. At the start of communication, the variable N is the number of opponents, the variable w is the width of the screen 150, the variable d (L) is the distance DL from the center of the left speaker 126L to the left end of the screen 150, and the variable d (R) is the right speaker 126R. It is assumed that the distance DR from the center to the right end of the screen 150 and the audio amplification factor of the audio amplification circuit 122 are set to the variable A, respectively.

Further, this program is activated each time the voice frame shown in FIG. 2 is received.

First, the variable I is set to 1 (S10).
1). If the variable I is less than or equal to the variable N, the process proceeds to S103,
If not, the process ends (S102).

In S103, the display control circuit 132
The distance (D1, D2, D3 or D4) from the left end of the screen 150 to the center of the window of the partner I is read and stored in the variable di. For the variable R (L), the voice mixing circuit 12
The ratio of the sound output to 4L is stored (S104).
This ratio is calculated from the center of the left speaker 126L with respect to the distance between the centers of the left and right speakers 126L and 126R.
The percentage of the distance to the center of the window, that is, (d
It can be calculated by (L) + di) / (d (L) + w + d (R)). 1-R (L) is stored in the variable R (R) (S105).

Based on the audio output ratios stored in S104 and S105, the audio amplification circuit 122 sets the amplification factor of the audio to be output to the audio mixing circuit 124L to A × R.
(L) is instructed (S106), and the amplification factor of the sound to be output to the sound mixing circuit 124R is instructed to be A × R (R) (S107).

1 is added to the variable I, and the process returns to S102 (S
108).

The position of each window can be changed by the user operating the operation device 116 during communication, and the value of the variable di changes accordingly.

As is clear from the above description, according to the second embodiment, the strength of the voice from the speaker enables the user to estimate the position of the window of the partner corresponding to the voice. In addition, the audio and video can be viewed without a feeling of strangeness.

In the second embodiment, as in the first embodiment, the ITU-T Recommendation H.264 is used. MPEG instead of H.261
Other image encoding / decoding methods such as the above may also be adopted, and still images or continuous still images may be used as images instead of moving images. 221 may be replaced by a multimedia multiplexing method for allocating a plurality of calls to images, voices, and data. For multiplexing voices and images of a plurality of parties, instead of the time division multiplexing method, a frequency division method is used. Any of multiplex, phase difference multiplex, and packet multiplex may be adopted, and it is also possible to change to a screen display in which windows do not overlap, such as screen division.

In the second embodiment, two left and right speakers 1 are used.
Although only 26L and 126R are used, three or more speakers can be used. For example, by providing upper and lower speakers and changing the sound output ratio of the upper and lower speakers, more accurate correspondence between the direction of sound and the position of the window can be achieved.

In the second embodiment, the variable A is not changed during communication, but it may be changed by an operation from the operation device 116.

[0053]

As can be understood from the above description, according to the present invention, the user can always easily identify and gaze at the window of the other party corresponding to the voice.

[Brief description of drawings]

FIG. 1 is a schematic block diagram of a first embodiment of the present invention.

FIG. 2 is a frame structure diagram of a voice frame during communication according to the first embodiment.

FIG. 3 is an example of a display screen during communication according to the first embodiment.

FIG. 4 is a memory configuration of a RAM 14 of the first embodiment.

FIG. 5 is a flowchart of a program stored in the ROM 12 of the first embodiment.

FIG. 6 is a schematic block diagram of a second embodiment of the present invention.

FIG. 7 is an example of a display screen during communication according to the second embodiment.

FIG. 8 is a memory configuration of a RAM 114 according to a second embodiment.

FIG. 9 is a flowchart of a program stored in the ROM 112 according to the second embodiment.

[Explanation of symbols]

10: CPU 12: ROM 14: RAM 16: Operation circuit 18: Voice encoding / decoding circuit 20: Microphone
22: Level detection circuit 24: Audio mixing circuit 26: Speaker 28: Image encoding / decoding circuit 30: Camera
32: display control circuit 34: display device 36: demultiplexing circuit 38: line control circuit 40: clock 42, 4
4,46,48: Time slot 50: Display screen 5
2, 54, 56: Window 58: Icon 60:
Variable N 62 for storing the number of opponents: Variable T for storing the maximum display time of the window of the opponent who is not receiving voice
m 64: Variable T that stores the current time T 66: Variable I that stores the number of the other party 68: Variable V that stores the voice level of the other party I 70: The time when the window of the other party # 1 is received and the window is displayed Variable Ti (1) 72 to store: The time when the voice of the partner # 2 is received and the window is displayed is stored. Variable Ti (2) 74: The voice of the partner # 3 is received and the time to display the window is stored. Variable Ti (3)
76: Variable Ti (N) 110 for storing the time when the window of the other party #N is received and the window is displayed 110: CPU 11
2: ROM 114: RAM 116: Operation circuit 11
8: Speech coding / decoding circuit 120: Microphone 122:
Audio amplification circuit 124L, 124R: audio mixing circuit 1
26L, 126R: left and right speakers 128: image encoding / decoding circuit 130: camera 132: display control circuit
134: Display device 136: Demultiplexing circuit 138: Line control circuit 150: Display screen 152, 154, 15
6, 158: Windows D1, D2, D3, D4: Distance from the left end of the screen 150 to the center of each window
W: width of screen 150 DL: distance from center of left speaker 126L to left end of screen 150 DR: right speaker 1
Distance from the center of 26 to the right edge of screen 150 160:
Variable N 162 for storing the number of opponents: Variable w 164 for storing the width W of the screen 150: Variable d (L) 166 for storing the distance DL from the center of the left speaker 126L to the left end of the screen 150L: For the right speaker 126R Variable d (R) 168 that stores the distance DL from the center to the right end of the screen 150: Variable A that stores the voice amplification factor 170: Variable I that stores the number of the other party 172: From the left end of the screen 150 to the window of the other party I Variable di 174 for storing distance to center: Variable R for storing ratio of voices output to voice mixing circuit 124L
(L) 176: Variable R (R) that stores the ratio of the sound output to the sound mixing circuit 124R

Claims (4)

[Claims] 1. A multimedia device which performs multimedia communication with a plurality of partners via a communication line and displays a plurality of images received from the partner in a multi-window, and detects a sound level received from the partner. And a means for displaying, as an icon, a window for displaying an image of a partner who has not received a predetermined voice for a predetermined time, based on the detection result of the voice level detecting means. Multimedia device to do. 2. A means for displaying, in a window, an image of the other party who has received the voice of the predetermined level or higher.
Multimedia device. 3. A multimedia device for performing multimedia communication with a plurality of parties via a communication line and displaying received images in a multi-window, comprising means for reading a positional relationship of each window, and means for reading the positional relationship between the windows. And an output unit for outputting the received voice of the above at a volume according to the positional relationship of the corresponding windows. 4. The multimedia device according to claim 3, wherein the output unit is a unit that outputs from a plurality of speakers at a volume of a ratio according to a positional relationship of the windows.