JP3573850B2 - Video conferencing systems - Google Patents

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a video conference system in which a plurality of narrow area networks are connected to a wide area network, and more particularly, to a technique for displaying a participant image using a multipoint control unit (hereinafter, referred to as “MCU”).
[0002]
[Technical background]
2. Description of the Related Art A video conference system (hereinafter simply referred to as a "conference system") is configured by connecting a plurality of LANs (narrow networks) such as Ethernet to one WAN (wide area network) such as an ATM (asynchronous transfer mode) line. It is often done.
Conventionally, in this type of system, an MCU is provided on any one LAN, or provided on each LAN.
Here, the MCU receives multiplexed signals of participant images and participant voices from respective video conference terminal devices (hereinafter, referred to as “terminal devices”) of each conference participant, and receives these multiplexed signals. Is separated into a video signal and an audio signal, and from these signals, an image to be displayed on an image output device of each terminal device (hereinafter referred to as a “conference image”) and output to an audio output device of each terminal device The audio to be performed (hereinafter referred to as “conference audio”) is created.
For example, the conference image may be one participant image selected based on a speaker or the like. In addition, the display area is a composite image divided and displayed on an appropriate number (for example, four) of participant images, and the speaking participant may be highlighted. The conference voice may be a synthesized voice of voices of all conference participants, or may be a synthesized voice of one voice or voices of a plurality of specific participants.
The MCU again multiplexes the conference image and the conference audio created as described above, and multicasts the multiplexed signal to the terminal device of each participant (in some cases, bystander).
[0003]
By the way, in a video conference system in which a plurality of LANs are connected to each other via a WAN, there are cases where it is desired to change a display image in LAN units. For example, when the head office LAN, the first office LAN, and the second office LAN are connected via a WAN, the participants of the terminal devices of the head office LAN are connected to the first office LAN. There is a case in which a participant image of the terminal device that the user wants to display on the image display device with a certain priority. Further, for example, a participant of each terminal device of each office LAN may want to display a participant image of each terminal device of the LAN to which the participant belongs on an image display device with a certain priority.
However, in any of the conventional video conference systems described above, since the conference image is basically the same for all terminal devices, there is a participant image from a terminal device belonging to a certain LAN as described above. There is a problem that it cannot be displayed by priority.
[0004]
[Object of the invention]
SUMMARY OF THE INVENTION It is an object of the present invention to provide a conference system which can display a conference image on an image display device of a terminal device with a priority determined based on a silence time for each LAN.
[0005]
Summary of the Invention
The video conferencing system according to the present invention is characterized in that a plurality of LANs are connected to a WAN and at least one video conference connected to the WAN via an MCU (hereinafter, these MCUs are referred to as “LMCUs”). (1) The LMCU monitors the duration of silence of a participant of each of the terminal apparatuses, and weights the duration of silence according to the LAN to which the terminal apparatus belongs. Display priority determining means for determining the display priority of each participant image, and
(2) There is provided a means for selecting a predetermined number of participant images from the respective participant images in descending order of display priority and multicasting these selected images to each terminal device in the LAN. And
[0006]
The display priority determining means (1) monitors the silence duration of each participant of the terminal device in the LAN to which the LMCU belongs, and (2) responds to the LAN to which the terminal device belongs in the silence continuation time. The display priority of each participant image is determined. The display priority determining means may be divided into two components, and the processes (1) and (2) may be performed separately.
The selection / output means selects a predetermined number of participant images from the respective participant images in descending order of display priority, and multicasts the selected images to the terminal conference device in the LAN.
[0007]
There are various modes for the above weighting. For example, a certain time can be added to or subtracted from the silence duration from another LMCU. Alternatively, the silence duration from another LMCU can be multiplied by a predetermined coefficient. The priority is higher as the silence duration is shorter.
[0008]
In the conference system of the present invention, by appropriately setting the weights (methods and degrees), it is possible to select a participant image according to the circumstances of each LAN. Moreover, as in the conventional case, when two or more types of composite images are created by a single MCU, WAN communication increases accordingly. However, in the conference system of the present invention, since this operation is localized, the amount of data communication is increased. Does not increase.
[0009]
In the conference system of the present invention, means for giving the highest display priority to a specific participant may be provided. As a result, the display priority of a specific participant such as the chair can be set higher than other participants.
[0010]
【Example】
FIG. 1 is a schematic diagram showing an example of a network to which the conference system of the present invention is applied. In FIG. 1, a plurality of LANs 21, 22, and 23 are connected to a WAN 1 via routers 31, 32, and 33, respectively. The terminal devices 211 to 215 are connected to the LAN 21, the terminal devices 221 to 223 are connected to the LAN 22, and the terminal devices 231 to 236 are connected to the LAN 23. The LAN 21 is also connected to terminal devices 216 and 217 of participants who can only participate in the conference as bystanders (that is, participants who only receive the participant image and the participant voice).
[0011]
The routers 31 to 33 have LMCUs 41 to 43, respectively, and the LMCUs 41 to 43 can communicate with each other via the WAN1.
[0012]
2 and 3 are diagrams showing the configuration of the LMCU 41 in more detail. FIG. 2 shows a LAN input processing unit, and FIG. 3 shows a WAN input processing unit.
In FIG. 2, multiplexed signals M _{11 to} M ₁₅ of participant images and participant voices from terminal devices 211 to 215 are received by LAN input unit 51.
As shown in FIG. 2, the LAN input unit 51 distributes and outputs these multiplexed signals M _{11 to} M ₁₅ to the classifiers 531 to 535 via the distributor 52.
Classifier 531 to 535 divides each multiplexed signal _M 11 _{~M 15} to the participant audio _A 11 _{to A 15} and participants image _V 11 _{~V 15,} the V 11 _{~V 15} first participant image The selection means 54 outputs A _{11 to} A ₁₅ to the first display priority determination means 55.
[0013]
The first display priority determining means 55 determines the display priority of the participant images V _{11 to} V _{15 based} on the participant voices A _{11 to} A _15, and determines the display priority of the four high-priority participant images. The display priorities P ₁₁ , P ₁₂ , P ₁₃ and P ₁₅ are selected. This display priority can be determined by the silence duration of each participant of the terminal devices 211 to 215 as described later.
Here, it is assumed that the display priorities of the participant images are V ₁₁ , V ₁₂ , V ₁₃ , and V ₁₅ higher than V ₁₄ .
The first display priority determining unit 55 outputs the display priorities P ₁₁ , P ₁₂ , P ₁₃ , and P ₁₅ to the first participant image selecting unit 54, the multiplexing unit 58, and the priority buffer 62. , And outputs the participant voices A _{11 to} A ₁₅ to the first voice synthesis means 56.
[0014]
The first participant image selection means 54 selects four participant images V ₁₁ , V ₁₂ , V ₁₃ , and V ₁₅ based on the display priorities P ₁₁ , P ₁₂ , P ₁₃ , and P ₁₅ , and selects these. Output to the first image combining means 57. The first image synthesizing unit 57 synthesizes these participant images into one image VC ₁ whose display area is divided into four, and outputs the synthesized image VC ₁ to the first multiplexing unit 58 and the image buffer 63. .
The first voice synthesis means 56 synthesizes the participant voices A _{11 to} A ₁₅ into one voice AC ₁ and outputs the synthesized voice AC ₁ to the voice buffer 61 and the first multiplexing means 58.
The multiplexing means 58 multiplexes the synthesized image VC ₁ , the priority information P ₁ (consisting of display priorities P ₁₁ , P ₁₂ , P ₁₃ , and P ₁₅ ) and the synthesized voice AC _1, and multiplexes the multiplexed signal M ₁ Is output to the WAN output unit 59. WAN output unit 59 the multiplexed signal _{M 1,} multicasts to LMCU42,43 of LAN22,23 through WAN1.
[0015]
The LMCUs 42 and 43 of the LANs 21 and 22 have the same LAN input processing unit as described above, and output multiplexed signals M ₂ and M ₃ onto the WAN 1. Here, _{M 2} is the composite image VC ₂ participants image _{V 21, V 22, V 23} of the terminal device 221, 222, 223, priority information _{P 2} of these participants image (display priority _P 21, P _22, composed of _{P 23),} and a multiplexed signal of the synthesized speech AC ₂ terminal devices 221 to 223, _{M 3} is participant image _V 32 of the terminal device _{222,223,224,225, V} 33, V _34, the composite image VC ₃ of _{V 35,} (consisting of display priority _{P 32, P 33, P 34} , P 35) the priority information _{P 3} of these participants images, and synthesized speech AC ₃ of the terminal apparatus 231 to 236 Multiplexed signal.
[0016]
As shown in FIG. 3, the multiplexed signals M ₂ and M ₃ from the LMCUs 42 and 43 are received by the WAN input unit 71.
The WAN input unit 71 outputs M ₂ and M ₃ to the distributor 72, and the distributor 72 outputs these multiplexed signals M ₂ and M ₃ to the classifiers 731 and 732, respectively.
The classifier 731 outputs the synthesized image VC ₂ to the image classifier 741, the priority information P ₂ to the priority classifier 742, and the synthesized speech AC ₂ to the speech synthesis unit 75. The classifier 732 outputs the synthesized image VC ₃ to the image classifier 743, the priority information P ₃ to the priority classifier 744, and the synthesized speech AC ₃ to the speech synthesis unit 75.
[0017]
Image classifier 741, a composite image VC ₂ is divided into _{V 21, V 22, V 23} , and outputs them to the second participant image selection unit 76. Similarly, the classifier 743, the composite image VC ₃ is divided into _{V 32, V 33, V 34} , V 35, and outputs them to the second participant image selection unit 76.
Priority classifier 742 divides the priority information _{P 2} to the display priority _{P 21, P 22, P 23} , and outputs them to the second display priority determining means 77. Similarly, priority classifier 744 divides the priority information _{P 3} to the display priority _{P 32, P 33, P 34} , P 35, and outputs them to the second display priority determining means 77 .
[0018]
The second display priority determining means 77 applies a predetermined weight to the display priority P ₂₁ , P ₂₂ , and P ₂₃ for the LAN ₂₁ and assigns the display priority P ₃₂ , P ₃₃ , P ₃₄ , and P ₃₅ to the display priority P ₂₁ , P ₂₂ , and P _23. On the other hand, a predetermined weight is applied to the LAN 23.
Here, the second display priority determining means 77 uniformly lowers the display priorities P ₂₁ , P ₂₂ , and P ₂₃ by a certain frequency, and sets new weighted display priorities P ′ ₂₁ , P ′ ₂₂ , P _'23 generates, low only uniform degree that the display priority _{P 32,} P _33, P _{34, P 35,} new display priority _P weighting' _32, P _{'33, P'} 34, P ' ₃₅ is generated. The second display priority determining unit 77 receives the display priorities P ₁₁ , P ₁₂ , P ₁₃ , and P ₁₅ from the priority buffer 62 described with reference to FIG. The above-mentioned weighted display priorities are input, and four of the display priorities (here, P ₁₂ , P ₁₅ , P ′ ₂₃ , P ′ ₃₂ ) having the highest display priority are specified, The selection signal S2 is output to the second participant image selection means 76.
[0019]
Participants image selecting unit 76 inputs the participant image _{V 21 ~V 23, V 31 ~V} 36, participant image _VC 1 from the image buffer 63 described in FIG. _{2 (V 11 ~V 13, V} 15 ) Is input, and among the participant images, the participant images V ₁₂ , V ₁₅ , V ₂₃ , and V ₃₂ having a high display priority are selected based on the selection signal S ₂ , and the second image synthesis is performed. Output to means 78. The second image synthesizing unit 78 synthesizes these participant images V ₁₂ , V ₁₅ , V ₂₃ , and V ₃₂ into one image V ₁ whose display area is divided into four, and multiplexes the synthesized image V ₁ . Output to means 79.
The voice synthesizing means 75 receives AC ₂ and AC ₃ as described above, and also receives a synthesized voice AC ₁ from the voice buffer 61 described in FIG. 2 and synthesizes them into one voice A _1. and it outputs the synthesized speech _{a 1} to the multiplexing unit 79.
The multiplexing unit 79 multiplexes the input synthesized image V ₁ and synthesized voice A ₁ and outputs the multiplexed signal _MLAN1 to the LAN output unit 80. The LAN output unit 80 multicasts the multiplexed signal MLAN1 to the terminal devices 211 to 215 and 216 and 217 via the _LAN 21.
[0020]
An embodiment in which the LMCU determines the display priority based on the duration of silence of each participant and outputs these participant images to the composite screen will be described below with reference to FIG.
The first display priority determining means 55 of each of the LMCUs 41 to 43 compares the voice level of each user with a certain threshold value, and determines from each voice level whether each participant is emitting voice. The display priority determination means 55 has a timer, monitors the duration of silence of the participant, and records the last time of speaking.
The evaluation value has a part determined by the silence duration and a part determined by weighting (see a bias value described later). The value that the evaluation value can take is usually 0 to 100. For example, when weighting is not performed, the evaluation value is 0 at the moment when the participant finishes speaking, becomes higher every time the silence time elapses, becomes 100 after the predetermined time elapses, and does not increase any more. The part determined by the weighting of the evaluation value is given independently of the silence time. For example, when the weight is 20, the initial value of the evaluation value is 20, and it increases from 20 every time the silence time elapses.
[0021]
The display priority is lower as the evaluation value is larger, and is higher as the evaluation value is smaller.
Display priority determining means 55, on the basis of the evaluation value, the display priority four high (i.e., a low evaluation value 4 participants) first participant image select signals S ₁ to identify the image Output to the means 54. The first participant image selecting means 54 selects four images based on the signal S ₁ and outputs these images to the first participant image synthesizing means 57. The first participant image synthesizing unit 57 synthesizes the four images into one image whose display area is divided into four, and outputs the synthesized image to the multiplexing unit 58 as described above.
[0022]
Figure 4 (a-1), ( a-2), (a-3) , the first participant image synthesizing means 57 image _VC 1 to Vc ₃ was synthesized by the LMCU41～43, and each divided image evaluation value (part determined by the silence _duration) shows e 1 to e _3. In each image VC ₁ to Vc _3, divided images on the display in descending order of priority (evaluation values in ascending order), the upper left, upper right, lower left, are displayed in the order at the lower right. For example, in _{LMCU41, V 11, V 12,} V 13, V 15 and is displayed, (a-1) as shown, the evaluation value _is "3" for _{V 12,} the _{V 15} is "21 _{"for V 11} is" 78 _", the _{V 13} is" 90 ".
[0023]
These synthesized images VC _{1 to} VC ₃ and the evaluation value are multiplexed together with the synthesized voice by the multiplexing means 58 of the LMCUs 41 to 43, and are exchanged between the LMCUs via the WAN 1. Each LMCU synthesizes the voice from the other LMCU directly with the voice of each terminal device of its own LAN.
The weighting by the second display priority determining means 77 described in FIG. 3 is performed by adding a bias value (a part of the evaluation value determined by the weighting) to the evaluation value from another LMCU.
[0024]
For example, the second display priority determining means 77 of LMCU41 is without adding Henchi the evaluation value _{e 1} self, adding other point polarization value "20" for the evaluation value from LMCU42,43. FIG 4 (b-1), a (b-2), (b -3) , the evaluation value _{e '2, e' 3} after the polarization value is added an evaluation value _{e 1} and Henchi not added Show. The display priority determining means 77 refers to the evaluation values e ₁ , e ′ ₂ , e ′ ₃ to generate a signal S ₂ for selecting the four images having the smallest evaluation values, and to generate a signal S ₂ for the participant image. Output to the selection means 76. As described above, the participant image selection means 76 selects four images (in this case, V ₁₂ , V ₁₅ , V ₂₃ , V ₃₂ ) based on the signal S ₂ , and sends the selected image to the second image synthesis means 78. Then, the combining means 78 creates a combined image as shown in FIG.
[0025]
The same processing as described above is performed in the LMCUs 42 and 43, and each second image combining unit 78 creates a combined image as shown in FIGS. 4 (c-2) and (c-3). As described above, the LAN output unit 80 of each of the LMCUs 41 to 43 transmits the composite images VC _{1 to} VC ₃ shown in FIGS. 4 (a-1), (c-2), and (c-3) to the terminal device of each LAN. , And multicast transmission together with the synthesized voices AC _{1 to} AC ₃ .
In FIG. 4 (c-1), (c-2), and (c-3), the participant images are arranged in descending order of priority, but are displayed but changed only in position. It is preferable that the display position of the participant image is maintained as it is (that is, the arrangement of the display areas of the participant image divided into four parts is changed).
[0026]
FIG. 5 shows the processing steps in the LMCUs 41 to 43. FIG. 2 shows a flow of processing of a participant image and a flow of processing of an audio signal. For example, the LMCU 41 receives a multiplexed signal M ₂ (synthesized image VC ₂ , synthesized voice AC ₂ ) and a multiplexed signal M ₃ (synthesized image VC ₃ , synthesized voice AC ₃ ) from the LMCUs 42 and 43, and receives them and their own. A state is shown in which a synthesized image V ₁ and a synthesized voice A ₁ are created from a synthesized image VC ₁ and a synthesized voice AC ₁ .
[0027]
The other point bias value is for maintaining locality in the LAN, and when this value is set to 0, the same image is displayed on all terminal devices belonging to the conference. That is, among all participants of the conference, four persons are selected and displayed in ascending order of silence duration.
When the local deviation value from another LMCU is increased, the display frequency of the participants in the own LAN increases.
[0028]
In addition, one chairperson can be selected according to the agreement between the LMCUs 41 to 43. The LMCU of the LAN to which the chair belongs belongs to the chair mode, and the evaluation value for the chair is "-120" (other-point bias value (20) and the maximum value (100) of the evaluation value portion determined by the silence duration time regardless of the voice level). (A value obtained by adding a negative sign to the sum of Thus, the chair's evaluation value is minimized in any LMCU, so that the chair image can always appear on the composite screen. It should be noted that not only the chair but also speakers who want to pay attention can be treated in the same manner as the chair. The number of participants to be treated in this way is not limited to one, but may be plural (for example, two) at the same time.
[0029]
As is clear from the above embodiments, the conference system of the present invention also includes the following embodiments.
That is, the MCU provided in each LAN is
As a LAN input processing unit,
[1] first display priority determining means (55) for determining a display priority of a participant image received from each terminal device of the LAN based on a short duration of silence of each participant;
[2] first participant image selection means (54) for selecting a predetermined number of participant images from the participant images in the order of the display priority,
[3] first image combining means (57) for combining these on one screen;
[4] first voice synthesizing means (56) for synthesizing voice received from each terminal device of the LAN,
[5] The images synthesized by the first participant image selecting means, the display priorities of these participant images, and the voice synthesized by the first voice synthesizing means are converted into the MCU of another LAN. First transmission means (58, 59) for transmitting the multiplexed data to
Respectively,
As a WAN input processing unit,
[6] means (71) for receiving the multiplexed data from MCUs of other plural LANs;
[7] The silence durations received from the MCUs of the other plurality of LANs are weighted respectively determined for the other plurality of LANs, and are selected by the first participant image selecting means of the output processing unit. The display priority of the predetermined number of participant images and the display number of the predetermined number of participant images respectively received from the MCUs of the other plurality of LANs is determined by the short duration of silence of each participant. Participant image selection means (77),
[8] A predetermined number of participant images received by the predetermined number of participant images selected by the first participant image selection means of the output processing unit and the plurality of other LAN MCUs A second image synthesizing means (76) for selecting, from the participant images for the number of participants, in descending order of display priority and synthesizing them on one screen;
[9] a second voice synthesizing unit (75) for synthesizing the voice synthesized by the first voice synthesizing unit and each voice received from the MCUs of the other plurality of LANs;
[10] A second transmission of transmitting the synthesized image synthesized by the second participant image selecting means and the voice synthesized by the second voice synthesizing means to the terminal device in the LAN as multiplexed data. Means (79, 80);
, Respectively.
[0030]
【The invention's effect】
According to the present invention, a conference image can be displayed on an image display device of a terminal device with a priority based on a silence time determined for each LAN.
In particular, according to the present invention, since the MCU function is distributedly managed, the load per LMCU can be reduced, and the effect of reducing the WAN traffic throughout the conference system can be achieved.
[Brief description of the drawings]
FIG. 1 is a schematic diagram showing an example of a system to which a video conference system of the present invention is applied.
FIG. 2 is an explanatory diagram showing an example of an input processing unit of an LMCU in the video conference system of the present invention.
FIG. 3 is an explanatory diagram showing an example of a WAN input processing unit in the video conference system of the present invention.
FIGS. 4 (a-1) to (a-3) are diagrams showing evaluation values of each image and each divided image synthesized by each LMCU, and FIGS. 4 (b-1) to (b-3) are diagrams showing the evaluation values. FIG. 3C is a diagram illustrating an evaluation value after a bias value is added to a value, and FIGS. 3C to 3C are diagrams illustrating images synthesized by each LMCU and transmitted to each terminal device.
FIG. 5 is an explanatory diagram showing how the LMCU exchanges multiplexed signals with each other via a WAN.
[Explanation of symbols]
1 WAN
21-23 LAN
211 to 217 Terminal devices 221 to 223 of LAN 21 Terminal devices 231 to 236 of LAN 22 Terminal devices 31 to 33 of LAN 23 Routers 41 to 43 LMCU
51 LAN input units 531 to 535 Classifier 54 First participant image selecting unit 55 First display priority determining unit 56 First voice synthesizing unit 57 First image synthesizing unit 58 Multiplexing unit 59 WAN output unit 61 Audio buffer 62 Priority buffer 63 Image buffer 71 WAN input unit 72 Distributors 731 and 732 Classifiers 741 and 743 Image classifiers 742 and 744 Priority classifier 76 Second participant image selection means 77 Second display priority Determining means 78 image synthesizing means 79 multiplexing means 80 LAN output unit

Claims (3)

A plurality of narrow area networks are connected to the wide area network, and each of the plurality of narrow area networks is provided with at least one video conference terminal connected to the wide area network via a multipoint control unit. Is a video conferencing system that communicates with a multipoint control unit provided in another narrow area network, and in which a plurality of participants are displayed on an image display device of each video conference terminal device, wherein the multipoint The control unit monitors the silence duration of the participant of each of the terminal devices, and performs a weighting on the silence duration according to the narrow-area network to which the terminal device belongs, to thereby obtain the image of each of the participant images. Display priority determining means for determining display priority, and Means for selecting a predetermined number of participant images from the participant images in descending order of display priority, and multicasting these selected images to each video conference terminal device in the narrow area network. A video conference system. The video conference system according to claim 1, further comprising means for assigning the highest display priority to a specific participant. The weighting is an addition of a value representing a weight to an evaluation value determined based on the silence duration , and the display priority order is determined in ascending order of the value of the result of the addition. The video conference system according to claim 1 or 2, wherein

Images