JP2005311916A - Stream data distribution system, stream data distribution method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To distribute stream data, which reflects a request of each terminal device. <P>SOLUTION: A stream data distribution unit 11 receives voice stream data and video image stream data of each participant from each terminal device through a network and selects video image stream data to be transmitted to each terminal device from the received video image data on the basis of the voice stream data. In the stream data distribution unit 11, the video image stream data is selected based on a rule table 92 showing a distribution rule of the video image stream data for each participant's terminal device. In addition the rule table 92 can be rewritten from the participant's terminal device through the network. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a stream data distribution device, a stream data distribution method, and a program thereof.

  2. Description of the Related Art Conventionally, there is a video conference system that analyzes audio information of participants (terminal devices), selects predetermined video stream data, and distributes them to the terminal devices of all participants.

For example, Patent Literature 1 discloses a technique for distributing video stream data of a terminal device having the highest volume (participant's voice) to each participant's terminal device.
JP 10-13803 A (Claim 1 etc.)

  However, in the technique disclosed in Patent Document 1, the stream data distributed to each terminal device by the video conference control device (stream data distribution device) is uniform. That is, for example, there is a problem that the stream data distribution device cannot distribute the stream data reflecting the request of each terminal device.

  Therefore, an object of the present invention is to provide a stream data distribution apparatus and the like that solve the above-described problems.

  In order to solve the above-described problem, the present invention receives audio stream data and video stream data of each participant from each terminal device via a network, and based on the audio stream data, the received video stream data Among them, a stream data distribution device that selects video stream data to be transmitted to each terminal device, an input interface that receives input of audio stream data and video stream data transmitted from each participant's terminal device, and each participation device For each of the identification information of each participant's terminal device, and information on the video stream data to be selected for the analysis result, for each identification information of each participant's terminal device Rule table showing the analysis result and the rule table Referring to the received video stream data, a video stream selection unit that selects video stream data to be transmitted to each participant's terminal device, the selected video stream data, and the participant's terminal device A stream data creation unit that creates delivery stream data based on the received audio stream data; and an output interface that outputs the created delivery stream data to the terminal device of each participant via the network. The configuration. Other configurations will be described in the embodiments described later.

Note that the audio characteristic analysis unit in the claims is realized by a CPU (Central Processing Unit) executing an audio characteristic analysis module of an embodiment described later, and the video stream selection unit is an embodiment described later by the CPU. This is realized by executing the video stream selection module.
The stream data creation unit in the claims is realized by the CPU executing a multiplexing module according to an embodiment described later and at least one of a formation module and an audio / video frame creation module.

  According to the present invention, video stream data can be selected and transmitted for each terminal device in accordance with audio stream data from each terminal device. In addition, each participant sets the selection method (setting rule) of video stream data requested by the participant in the rule table, so that the participant can receive distribution of the video stream data according to his request.

Therefore, for example, as a setting rule for the terminal device of the participant A in the rule table, “the video stream data to be transmitted to the user terminal device has the highest volume of the audio stream data among the terminal devices other than the user terminal device. It is assumed that the rule that “the video stream data of the terminal device should be selected” is set.
Here, if the voice of participant A is the loudest, the video stream data of the participant with the second loudest voice can be transmitted (displayed) to the terminal device of participant A.
That is, the participant A who has the highest voice can discuss the video while viewing the video of the participant with the second highest voice (for example, the other party who is discussing) instead of his own video.

Further, for example, as a setting rule for the terminal device of the participant A in the rule table, “The video stream data to be transmitted to the user's own terminal device has a predetermined volume of the audio stream data among the terminal devices other than the user's terminal device”. It is assumed that the rule “video stream data transmitted from a terminal device newly exceeding the threshold value should be selected” is set.
Here, when Participant A is speaking, when another participant holds an objection with a sound having a volume equal to or higher than a predetermined threshold, the video stream data of the participant who has held the objection is the Participant A's video stream data. It is transmitted (displayed) to the terminal device.
That is, the participant A can know who has held the objection.

  Further, since the stream data distribution device selects and transmits new video stream data after a predetermined time has elapsed since the previous selection time of the video stream data, for example, when each participant speaks one after another, It is possible to prevent the video stream data to be transmitted (displayed) from changing one after another in a short time. That is, it is possible to prevent the video stream data once displayed on the terminal device from disappearing in an instant.

<Basic configuration>
Hereinafter, the best mode for implementing a stream data distribution apparatus and the like of the present invention (hereinafter referred to as “embodiment”) will be described in detail with reference to the drawings.

First, the basic configuration of the stream data distribution apparatus according to the present embodiment will be described, and then the characteristic configuration will be described.
FIG. 1 is a diagram showing an overview of a stream data distribution system including a stream data distribution apparatus according to an embodiment of the present invention. The outline of this system will be described with reference to FIG.
Hereinafter, stream data is abbreviated as a stream as appropriate.

  The stream data distribution system according to the present embodiment receives the terminal devices 1 (1a to d) of the participants A to D such as a video conference and the stream data of the participants A to D from the terminal devices 1 (1a to d). The stream data distribution device 11 transmits (distributes) the stream data to the terminal device 1 (1a to 1d) and the network 5 that connects them.

The terminal devices 1 (1a to d) are means for transmitting and receiving video stream data and audio stream data of the participants A to D via the network 5.
The terminal device 1 (1a to d) is composed of, for example, a PC (Personal Computer) or the like, and includes input means such as a keyboard and a mouse, display means such as a liquid crystal monitor, audio output means such as a speaker, and a camera. The imaging means and audio input means such as a microphone are connected.

  For example, the terminal device 1 a uses the video of the participant A captured by a camera or the like as real-time video stream data, and the voice of the participant A input by a microphone or the like as the real-time audio stream data via the network 5. Transmit to the stream data distribution apparatus 11. When the terminal device 1a receives the video stream data and audio stream data of the participants A to D via the network 5, the terminal device 1a displays the video stream data on a display unit such as a monitor, and outputs the audio stream data as audio from a speaker or the like. Output to the means.

The network 5 is, for example, an IP network. Details of the stream data distribution apparatus 11 will be described later.
Incidentally, although only four terminal devices 1a to 1d are illustrated in FIG. 1, the terminal device 1 may be composed of five or more terminal devices 1 as a matter of course. In FIG. 2 described below, a case where N terminal devices 1 are connected to the network 5 in FIG. 1 will be described as an example.

(Stream data distribution device)
FIG. 2 is a block diagram showing a configuration of the stream data distribution apparatus of FIG. As shown in FIG. 2, the stream data distribution apparatus 11 includes a CPU 7, a memory 8, a storage unit 9, an input interface 20, and an output interface 28.

The input interface 20 receives input of video stream data and audio stream data transmitted from the terminal device 1 (1a to n) via the network 5 (see FIG. 1).
The output interface 28 is an interface for outputting distribution stream data (audio stream data and video stream data) to the terminal device 1 (1a to 1n) via the network 5.
The storage unit 9 stores a stream data distribution program 91 and a rule table 92, and specifically includes a rewritable storage medium such as a hard disk.
Note that the stream data delivery method and the like of the present embodiment are realized by the CPU 7 loading and executing the stream data delivery program 91 in the storage unit 9 on the memory 8. Details of the stream data distribution program 91 and the rule table 92 will be described later.

Next, the execution processing of the stream data distribution program 91 by the CPU 7 will be described in detail with reference to FIGS. 3 and 4 (see FIGS. 1 and 2 as appropriate).
FIG. 3 is a block diagram showing the configuration of the stream data distribution program of FIG. FIG. 4 is a diagram quoted for explaining the processing procedure of the stream data distribution program of FIG.

  As shown in FIG. 3, the stream data distribution program 91 includes an audio / video separation module 21, an audio characteristic analysis module 22, a video selection module 23, a video stream synthesis module 24, an audio stream synthesis module 25, and an audio. A video frame creation module 26 and a formation module 27 are included. The video selection module 23 and the video stream synthesis module 24 are collectively referred to as a video stream selection module 30 as appropriate.

(Audio / video separation module)
The audio / video separation module 21 is a module for causing the CPU 7 to divide the stream data into audio stream data and video stream data.

(Voice characteristic extraction module)
The audio characteristic analysis module 22 is a module for causing the CPU 7 to analyze the audio characteristics of the audio stream data.

(Video selection module)
The video selection module 23 sends the video stream to be transmitted to each terminal device 1 (1a to n) to the CPU 7 based on the analysis result of the audio characteristics of the terminal device 1 (1a to n) and the rule table 92 (see FIG. 2). This module selects data.
Here, the rule table 92 indicates information of video stream data to be selected by the video stream synthesis module 24 with respect to the analysis result of the audio characteristic analysis module 22. For example, when the rule “select the video stream data transmitted from the terminal device with the highest volume” is set in the rule table 92, the CPU 7 determines which terminal device 1 has the highest volume based on the analysis result. And a selection signal of video stream data including identification information of the terminal device 1 is output.

(Video stream composition module)
The video stream synthesis module 24 sends the CPU 7 to the terminal device 1 (1a to n) out of the video stream data received from the terminal device 1 (1a to n) based on the selection signal (selection result) of the video selection module 23. This is a module for selecting video stream data to be transmitted.

(Audio stream synthesis module)
The audio stream synthesis module 25 performs processing such as audio superposition on the audio stream data of each terminal device 1 (1a to n) separated by the audio / video separation module 21 to the CPU 7 to synthesize the audio stream data. It is a module that performs

(Audio / video frame creation module)
The audio / video frame creation module 26 is a module that causes the CPU 7 to create an audio / video frame that combines the audio stream data synthesized by the audio stream synthesis module 25 and the video stream data selected by the video stream synthesis module 24. is there.

(Forming module)
The forming module 27 creates a video stream data and audio stream data (distribution stream data) to be transmitted from the audio / video frame to the packet or ATM (Asynchronous Transfer Mode) cell, that is, the terminal device 1 (1a to n). This is a module to be formed.

Next, a processing procedure of each component module of the stream data distribution program 91, specifically, an execution process of each module by the CPU 7 will be described with reference to FIG.
It is assumed that the stream data distribution device 11 has established a communication session with the terminal device 1 (1a to n) in advance for transmission and reception of stream data with the terminal device 1 (1a to n). That is, it is assumed that the stream data distribution device 11 receives the IP address of the terminal device 1 (1a to n) from the terminal device 1 (1a to n) and stores it in the storage unit 9 or the like.
Also, the audio stream data and video stream data transmitted from the terminal device 1 (1a to n) include the data type (audio stream data or video stream data) and the transmission source terminal device 1 (1a) in the header information of the data. ˜n) have identification information such as an IP address.

  First, the input interface 20 in FIG. 4 receives input of audio stream data and video stream data (audio / video stream 50) transmitted from the terminal device 1 (1a to n). Then, the audio / video stream data 50 is converted into audio / video stream data 51 and output to the audio / video separation module 21 (step S301).

  When the audio / video separation module 21 receives the audio / video stream data 51 from the input interface 20, the audio / video stream data 51 is referred to the header information of the data, and the audio / video stream data 51 is converted into the audio streams 52a to 52n and the video streams 55a to 55a. n (step S302). The audio streams 52a to 52n are output to the audio characteristic analysis module 22 and the audio stream synthesis module 25, and the video streams 55a to 55n are output to the video stream synthesis module 24.

When the audio characteristic analysis module 22 receives the audio streams 52a to 52n from the audio / video separation module 21, the characteristics of these audio stream data are analyzed (step S311). Then, the analysis result is output as analysis information 53 to the video selection module 23.
Here, the analysis of the characteristics of the audio streams 52a to 52n is, for example, the analysis of the characteristics of the volume (sound pressure level) of each of the audio streams 52a to 52n, and the analysis information 53 includes the audio stream data 52 to n. For each piece of identification information of the terminal device 1 (1a to n) of the transmission source, a device that indicates the volume of the terminal device 1 (1a to n) can be considered.

  When the video selection module 23 receives the analysis information 53 from the audio characteristic analysis module 22, it outputs a selection signal for video stream data. That is, referring to the analysis information 53 and the rule table 92, it is selected which terminal device 1 (1a to n) to transmit the video stream data. Then, the selection result is output as the video selection signal 54 to the video stream synthesis module 24 (step S312).

  For example, when the rule for selecting the video of the participant with the highest volume is set in the rule table 92 in step S312, the audio stream data with the highest volume has been transmitted based on the analysis information 53. The terminal device 1 (1a to n) is specified, and a video selection signal 54 including identification information (IP address or the like) of the terminal device 1 (1a to n) is created. Then, this signal is output to the video stream synthesis module 24.

  When the video stream synthesis module 24 receives the video selection signal 54 from the video selection module 23 and receives the video streams 55a to 55n from the audio / video separation module 21, the video stream data is selected. Based on the selection result, a video stream 56 is created (step S313). That is, based on the identification information of the terminal device 1 (1a to n) included in the video selection signal 54, the video stream data to be transmitted to the terminal device 1 (1a to n) is selected from the video streams 55a to 55n. A video stream 56 is created.

  For example, when the IP address of the terminal device 1a is included in the video selection signal 54 in step S313 described above, the video stream synthesis module 24 selects the video stream 55a from the terminal device 1a. Then, a video stream 56 is created based on the video stream 55 a, and this data is output to the audio / video frame creation module 26.

  On the other hand, when the audio stream synthesizing module 25 receives the audio streams 52a to 52n output from the audio / video separating module 21 in step S302, the audio stream 52a to n is synthesized to generate an audio stream 57 (step S302). S321). The audio stream 57 is output to the audio / video frame creation module 26.

  When the audio / video frame creation module 26 receives the video stream 56 from the video stream synthesis module 24 and receives the audio stream 57 from the audio stream synthesis module 25, the audio / video frame 58 is obtained by synthesizing the video stream 56 and the audio stream 57. Is created (step S331). The audio / video frame 58 is output to the forming module 27.

  When the forming module 27 receives the audio / video frame 58, the audio / video stream 62 (for example, stream data for distribution such as a packet or ATM cell) is created from the frame (step S332). The audio / video stream 62 is output to the output interface 28.

  When the audio / video stream 62 is received by the output interface 28, the audio / video stream 62 is created from the audio / video stream 62 into the audio / video stream 60 (step S341). And this data is transmitted to the terminal device 1 (1a-n).

In this way, the stream data distribution program 91 transmits video stream data to be transmitted to each terminal device 1 (1a to n) according to the volume of the audio stream data from each terminal device 1 (1a to n). Select and send.
That is, for example, when the volume of the audio stream data from the terminal device 1a (participant A) is the maximum, the stream data distribution device 11 sends the video stream of the terminal device 1a to each of the terminal devices 1 (1a to n). Data (participant A's video) is transmitted.

The configuration of the stream data distribution program 91 is not limited to the configuration described above.
FIG. 5 is a block diagram showing the configuration of the stream data distribution program of FIG. FIG. 6 is a diagram cited for explaining the processing procedure of the stream data distribution program of FIG. Constituent elements similar to those in the above-described configuration are denoted by the same reference numerals, and description thereof is omitted.
For example, as shown in FIG. 5, the stream data distribution program 91 performs multiplexing processing of audio stream data and video stream data instead of the audio stream synthesis module 25 and the audio / video frame creation module 26 of FIG. A multiplexing module 29 may be provided.

  That is, in the configuration of the stream data distribution program 91 in FIG. 3, the audio / video frame creation module 26 combines the video stream 56 and the audio stream 57 as shown in FIG. However, as shown in FIG. 6, the video stream 56 output from the video stream synthesizing module 24 is set as a video stream 59 by the forming module 271, and the audio stream 57 is set as an audio stream 61 by the forming module 272. Then, the video stream 59 and the audio stream 61 may be multiplexed by the multiplexing module 29 and output to the output interface 28.

<Characteristic composition>
Next, a characteristic configuration of the stream data distribution apparatus 11 according to the present embodiment will be described. The same components as those described above are denoted by the same reference numerals, and the description thereof is omitted.

  FIG. 7 is a block diagram showing a configuration of the stream data distribution apparatus according to the present embodiment. FIG. 8 is a block diagram showing the configuration of the stream data distribution program of FIG.

  As illustrated in FIG. 7, the stream data distribution device 11 further includes a selection time storage unit 93 that stores a selection time when the video stream data is selected in the storage unit 9. Details of the selection time storage unit 93 will be described later.

  Further, the rule table 92 in FIG. 7 includes information on video stream data to be selected with respect to the analysis result of the audio characteristic analysis module 22 for each identification information (IP address or the like) of each terminal device 1 (1a to n). I remember it.

Further, as shown in FIG. 8, the stream data distribution program 91 includes a video selection module 230.
The video selection module 230 is a module that selects video stream data for each terminal device 1 (1a to 1n) with reference to the analysis results of the rule table 92 and the audio characteristic analysis module 22.

With the above configuration, the stream data distribution device 11 can select video stream data according to the rules set for each terminal device 1 (1a to n) in the rule table 92.
Next, the characteristic configuration of the present embodiment will be described in detail.

(Rule table)
Table 1 below illustrates the rule table 92. As illustrated in Table 1, the rule table 92 includes identification information (such as an IP address) of the terminal device 1 (1a to n) and rules (setting rules) set for each terminal device 1 (1a to n). Is memorized.
As illustrated in Table 1, the rule table 92 includes information on the name of the terminal device 1 (1a to n) and the user name (participant name) of the terminal device 1 (1a to n). Also good.

  For example, in the information of No. 1 in the rule table 92 of Table 1, the user of the terminal device 1a with the IP address “aaa.aaa.aaa.aaa” is the participant “A” and is set in this terminal device 1a. The rule is to select “participant of the terminal device with the maximum volume of the audio stream data other than the terminal device (the video stream data to be transmitted to the terminal device is the terminal device other than the terminal device) , The video stream data of the terminal device with the maximum volume of the audio stream data should be selected). The rule table 92 is set with a rule “fixed to participants A, B, C, and D (the video stream data transmitted to the terminal device is only from the terminal devices 1a to d)”. May be.

The contents of the rule table 92 can be rewritten via the terminal device 1 (1a to n) and the network 5.
That is, the participant may input the setting information of the setting rule from the input unit or the like of the terminal device 1 (1a to n), and the stream data distribution device 11 may accept this and rewrite the rule table 92.
The rule table 92 may be rewritten before or after the stream data distribution apparatus 11 starts transmitting stream data to the terminal apparatus 1 (1a to n). By doing in this way, for example, participants A to D set rules before starting a video conference or the like using the terminal device 1 (1a to d), and in the middle of watching the state of the conference It is also possible to change the setting rule.
Details of the video stream data selection processing of the video selection module 230 with reference to the rule table 92 will be described later.

(Selected time memory)
The selection time storage unit 93 stores the selection time at which video stream data was selected by the video selection module 230. Then, based on the video selection module 230, the CPU 7 selects new video stream data after a predetermined time (for example, 1 minute) has elapsed since the previous video stream data selection time. Here, whether or not a predetermined time has passed refers to a timer (not shown) in the stream data distribution apparatus 11 or the like.
In this way, for example, when the participants A to D of this system speak one after another in a short time, they are selected (transmitted) to the terminal device 1 (1a to d) accordingly. The video stream data can be prevented from changing one after another. That is, it is possible to prevent the images of the participants A to D from disappearing in an instant from the monitor or the like of the terminal device 1 (1a to d).

Next, a processing procedure of the stream data distribution program 91 based on the above-described configuration, specifically, an execution process of each module by the CPU 7 will be described with reference to FIG. 9 (see FIGS. 1 to 8 as appropriate). FIG. 9 is a diagram cited for explaining the processing procedure of the stream data distribution program of FIG. The same components as those described above are denoted by the same reference numerals, and the description thereof is omitted.
The video selection modules 230a to n, video stream synthesis modules 24a to n, video stream selection modules 30a to n, audio / video frame creation modules 26a to n, and formation modules 27a to n in FIG. The processing of the video stream synthesis module 24, the video stream selection module 30, the audio / video frame creation module 26, and the formation module 27 with respect to the terminal devices 1a to n will be described.

Here, a process in which the stream data distribution device 11 selects video stream data for the terminal device 1a (participant A) will be described as an example.
Note that the processing from step S901 to step S911 in FIG. 9 is the same as the processing from step S301 to step S311 in FIG. 4, and thus the description thereof will be omitted, and the processing from step S912a (video selection module 230a) in FIG. explain.

  First, when the video selection module 230a receives the analysis information 53 output from the audio characteristic analysis module 22 in step S911, video stream data to be transmitted to the terminal device 1a is selected (step S912a). Specifically, the video selection module 230a selects video stream data to be transmitted to the terminal device 1a based on the analysis information 53 and the setting rules of the terminal device 1a in the rule table 92. Then, this information is output to the video stream synthesis module 24a as a video selection signal 54a.

Note that the video stream data selection process is performed when a predetermined time elapses from the previous video stream data selection time. That is, when the video selection module 230a refers to the selection time of the previous video stream data recorded in the selection time storage unit 93 and determines that a predetermined time has elapsed from the previous selection time, When it is determined that the predetermined time has not yet elapsed, the video stream data is not selected. When the video stream data is selected by the video selection module 230 a, the selection time is recorded in the selection time storage unit 93.
In this way, for example, when the participants A to D speak one after another in a short time, a video stream that is selected (transmitted) to the terminal device 1 (1a to d) according to this Data can be prevented from changing one after another.

  Next, the video stream synthesis module 24a selects video stream data to be transmitted to the terminal device 1a (step S913a). That is, in the video stream synthesis module 24a, the video stream 56a to be transmitted to the terminal device 1a from the video streams 55a to 55n received from the audio / video separation module 21 based on the video selection signal 54a from the video selection module 230a. Select. Then, the selection result (video stream 56a) is output to the audio / video frame creation module 26a.

  The audio / video frame creation module 26a synthesizes the video stream 56a received from the video stream synthesis module 24a and the audio stream 57 received from the audio stream synthesis module 25 to create an audio / video frame 58a (step S931a). . Then, this frame is output to the forming module 27a.

  Next, when the forming module 27a receives the audio / video frame 58a, it converts it into an ATM cell or the like to create an audio / video stream 63a (step S932a). The audio / video stream 63a is output to the multiplexing module 29.

When the multiplexing module 29 receives the audio / video streams 63a to 63n, these data are multiplexed (step S933). That is, an audio / video stream 62 obtained by multiplexing the audio / video streams 63a-n of the terminal devices 1a-n is created and output to the output interface 28. When the audio / video stream 62 is received by the output interface 28, the audio / video stream 62 is converted into the audio / video stream 60 (step S 941) and transmitted to the terminal device 1 (1 a to n) via the network 5.
Note that step S921 in FIG. 9 is the same as the process in step S321 in FIG.
Thereafter, the terminal device 1 (1a to n) receives the video stream data and the audio stream data from the stream data distribution device 11 via the network 5. Then, the video of the participant is displayed on the display unit of the terminal device 1 (1a to n), and the voice of the participant is output to the audio output unit to perform a video conference or the like.

  Incidentally, for the terminal devices 1a to d of the participants A to D, a rule as exemplified in the above-described Table 1 is set, and the volume of the terminal device 1a (participant A) is the largest, and the terminal device 1b (participant B) ) Is the second largest volume, the video stream data selected by the stream data distribution device 11 is as shown in Table 2 below.

  As shown in Table 2, the stream data distribution device 11 is transmitted from the terminal device 1b (participant B, volume second highest) to the terminal device 1a (participant A, volume is maximum). The video stream data is selected, and the video stream data transmitted from the terminal device 1a (participant A) is selected for the terminal devices 1b to d (participant C, volume is not maximum).

That is, in the basic configuration described above, since the stream data distribution device 11 does not individually select the video stream data for the terminal device 1 (1a to n), the same video stream data is uniformly transmitted to each participant. However, according to the characteristic configuration described above, each of the participants A to D sets the selection method (setting rule) of the video stream data requested by the participant A to D in the rule table 92. The video stream data corresponding to the request is distributed.
Therefore, for example, when the participant A wants to discuss while watching the video of the other party discussing with himself / herself in this system, the volume of the terminal device 1a in the rule table 92 is set as the volume at the terminal device other than his / her terminal device 1a. May be set to select the video stream data with the largest.
By doing so, if the voice of the participant A (self) is the loudest, the video of the second loudest participant (discussion partner) is displayed on the terminal device 1a, and his voice is second. If it is larger, the video of the participant with the loudest voice will be displayed. That is, the participant A can discuss while viewing the video of the other party.

Further, the setting rules described in the rule table 92 are not limited to the above example.
For example, as a setting rule, “video stream data to be transmitted to one's own terminal device is transmitted from a terminal device whose volume of the audio stream data newly exceeds a predetermined threshold among terminal devices other than one's own terminal device. A rule of “selecting selected video stream data” may be set.
That is, for example, when participant A is speaking, when participant B holds an objection (when speaking at a volume equal to or higher than a predetermined threshold), a rule for transmitting the video of participant B to terminal device 1 May be set.
Here, the above-described rule is set as the rule of the terminal device 1 (1a to d) in the rule table 92, and the volume of the participant B is equal to or higher than the predetermined threshold while the participant A is speaking at a volume higher than the predetermined threshold. How the video stream data transmitted to the terminal device 1 (1a to 1d) changes when the objection is caught will be described with reference to Table 3 below.

That is, as shown in Table 3, the stream data distribution device 11 transmits video stream data other than the terminal device 1a (participant A) to the terminal device 1a (participant A), and the terminal device 1 (1b to 1b). d) When the video stream of the terminal device 1a (participant A) was transmitted, the participant B got an objection, so that the terminal device 1a (participant A) and the terminal device 1c (participant) C) The video stream data of the terminal device 1b (participant B) is selected (transmitted) to the terminal device 1d (participant D). Accordingly, the participant A, the participant C, and the participant D can see that the participant B is in the objection by watching the video of the participant B.
Here, the video stream data transmitted to the terminal device 1b remains the data of the terminal device 1a (participant A).

  Further, when a plurality of rules are set as the setting rules of the terminal device 1 (1a to n) in the rule table 92, the rule table 92 may include information on the priority of application of each rule.

  For example, the video stream data transmitted to the terminal device 1a to the terminal device 1a is the video stream data of the terminal device having the maximum volume of the audio stream data among the terminal devices other than the terminal device. The rule to be selected and (2) the video stream data to be transmitted to the own terminal device, the volume of the audio stream data among terminal devices other than the own terminal device newly exceeds a predetermined threshold (for example, 60 dB) When setting two rules for selecting video stream data transmitted from a terminal device, setting information for applying the rule (2) with priority over the rule (1) may be included. .

In this way, for example, while participant A (terminal device 1a, volume maximum) and participant B (terminal device 1b, volume second) are discussing, participant C (terminal device 1c, Since the rule (2) is applied when the object (3rd volume, volume 60 dB or more) is in dispute, the video stream data of the terminal device 1c (participant C) is transmitted to the terminal device 1a (participant A). Will be.
Further, when the setting of applying the rule (1) with priority over the rule (2) is performed, the video stream data of the terminal device 1b (participant B) is transmitted to the terminal device 1a (participant A). Will be.

  In the configuration of the stream data distribution program 91 in FIG. 8, the multiplexing module 29 performs the multiplexing process of the audio / video streams 63a to 63n as shown in FIG. 9, but as shown in FIG. The multiplexing module 29 may perform the multiplexing process of the video streams 59a to 59n and the audio stream 61. Specifically, as shown in FIG. 10, the stream data distribution program 91 may be configured not to include the audio stream synthesis module 25 and the audio / video frame creation module 26 of FIG.

  FIG. 10 is a block diagram showing the configuration of the stream data distribution program of FIG. FIG. 11 is a diagram cited for explaining the processing procedure of the stream data distribution program of FIG. The same components as those described above are denoted by the same reference numerals and description thereof is omitted.

  As shown in FIG. 11, the processing procedure of the stream data distribution program 91 of FIG. 10 is to create video streams 59a-n to be transmitted to the terminal devices 1 (1a-n) by the forming modules 271a-n and to multiplex modules To 29. The forming module 272 outputs the audio stream 61 to the multiplexing module 29. The multiplexing module 29 multiplexes the video streams 59 a to 59 n and the audio stream 61 to create an audio / video stream 62.

The present invention is not limited to the embodiment described above, and can be modified without departing from the spirit of the invention.
For example, in the present embodiment, the stream data distribution device 11 selects the video stream data based on the volume of the audio stream data of each terminal device 1 (1a to 1n), but the frequency of the audio stream data, etc. Alternatively, the selection may be made based on the combination of the characteristics of the video stream data, the characteristics of the video stream data, and the characteristics of the audio stream data. Furthermore, the target to be selected may be audio stream data in addition to video stream data.
In the above-described embodiment, the case where the present system is applied to a video conference or the like has been described as an example. However, the present invention may be applied to communication using a messenger or the like.

The stream data distribution device 11 and the terminal device 1 (1a to 1n) according to the embodiment of the present invention can be realized by a computer and a stream data distribution program, and the program can be read by the computer. It is also possible to store in a medium (CD-ROM or the like) and provide by the storage medium. It is also possible to provide the program through the network 5.
The computer system here includes software such as an OS (Operating System) and hardware such as peripheral devices.

It is the figure which showed the outline | summary of the stream data delivery system containing the stream data delivery apparatus of embodiment of this invention. It is a block diagram which shows the structure of the stream data delivery apparatus of FIG. FIG. 3 is a block diagram showing a configuration of a stream data distribution program of FIG. 2. It is the figure quoted in order to demonstrate the process sequence of the program for stream data delivery of FIG. FIG. 3 is a block diagram showing a configuration of a stream data distribution program of FIG. 2. It is the figure quoted in order to demonstrate the process sequence of the program for stream data delivery of FIG. It is a block diagram which shows the structure of the stream data delivery apparatus of this Embodiment. It is a block diagram which shows the structure of the stream data delivery program of FIG. It is the figure quoted in order to demonstrate the process sequence of the program for stream data delivery of FIG. It is a block diagram which shows the structure of the stream data delivery program of FIG. It is the figure quoted in order to demonstrate the process sequence of the program for stream data delivery of FIG.

Explanation of symbols

1 (1a to n) terminal device 5 network 7 CPU
8 Memory 9 Storage Unit 11 Stream Data Distribution Device 20 Input Interface 21 Audio / Video Separation Module 22 Audio Characteristic Analysis Module (Audio Characteristic Analysis Unit)
23 Video selection module 24 (24a to n) Video stream synthesis module 25 Audio stream synthesis module 26 (26a to n) Audio / video frame creation module (stream data creation unit)
27 (27a to n) formation module (stream data creation unit)
28 Output Interface 29 Multiplexing Module (Stream Data Creation Unit)
30 (30a to n) video stream selection module (video stream selection unit)
91 Stream Data Distribution Program 92 Rule Table 93 Selection Time Storage Unit 230 (230a to n) Video Selection Module 271 (271a to n) Formation Module (Stream Data Creation Unit)
272 Formation module (stream data creation unit)

Claims (11)

Video stream data to be transmitted to each terminal device out of the received video stream data based on the audio stream data, receiving audio stream data and video stream data of each participant from each terminal device via a network A stream data distribution device for selecting
An input interface for receiving input of audio stream data and video stream data transmitted from each participant's terminal device;
An audio characteristic analysis unit that analyzes the characteristics of the audio stream data transmitted from each participant's terminal device;
For each identification information of the terminal device of each participant, a rule table indicating information of video stream data to be selected for the analysis result;
With reference to the analysis result and the rule table, a video stream selection unit that selects video stream data to be transmitted to each participant's terminal device from among the received video stream data;
A stream data creation unit that creates stream data for distribution based on the selected video stream data and audio stream data received from the terminal device of each participant;
An output interface for outputting the created stream data for distribution to the terminal device of each participant via the network;
A stream data distribution apparatus comprising: The rule table is stored in a rewritable storage medium,
The stream data distribution device includes:
When the setting information of the rule table is received from the terminal device of each participant via the network, the setting rule of the rule table is rewritten based on the setting information. The stream data distribution apparatus according to claim 1. The audio characteristic analysis unit analyzes characteristics related to the volume of audio stream data transmitted from the terminal device of each participant,
The rule table is
As information of the video stream data to be transmitted to the terminal device of each participant, the video stream data to be transmitted to the terminal device is the terminal device with the maximum volume of the audio stream data among the terminal devices other than the terminal device. The stream data distribution device according to claim 1 or 2, further comprising a rule that video stream data should be selected. The audio characteristic analysis unit analyzes characteristics related to the volume of audio stream data transmitted from the terminal device of each participant,
The rule table is
As information of the video stream data to be transmitted to each participant's terminal device, the video stream data to be transmitted to the terminal device has a predetermined threshold value for the volume of the audio stream data among the terminal devices other than the terminal device. The stream data distribution device according to claim 1 or 2, further comprising a rule that video stream data transmitted from a terminal device exceeding the above should be selected.   The video stream selection unit stores the selection time at which the video stream data was selected in the storage unit of the stream data distribution device, and a new video after a predetermined time has elapsed from the time at which the previous video stream data was selected. The stream data distribution device according to claim 3 or 4, wherein the stream data distribution device is configured to select stream data. Video stream data to be transmitted to each terminal device out of the received video stream data based on the audio stream data, receiving audio stream data and video stream data of each participant from each terminal device via a network A stream data delivery method using a stream data delivery device for selecting
The stream data distribution device is
Receiving audio stream data and video stream data input transmitted from each participant's terminal device;
Analyzing the characteristics of the audio stream data transmitted from the terminal device of each participant;
For each piece of identification information of the terminal device of each participant, with respect to the analysis result, refer to the rule table indicating the information of the video stream data to be selected by the stream data distribution device and the analysis result, and the reception Selecting video stream data to be transmitted from each video stream data to the terminal device of each participant;
Creating distribution stream data based on the selected video stream data and audio stream data received from each participant's terminal device;
Outputting the created stream data for distribution to the terminal device of each participant via the network;
The stream data delivery method characterized by performing. The rule table is stored in a rewritable storage medium,
The stream data distribution device includes:
Receiving input of setting information of the rule table from the terminal device of each participant via the network;
Rewriting the setting rule of the rule table based on the input setting information;
The stream data delivery method according to claim 6, further executed. In the step of analyzing the characteristics of the audio stream data transmitted from each participant's terminal device, the characteristics relating to the volume of the audio stream data transmitted from each participant's terminal device are analyzed,
The rule table is
As information of the video stream data to be transmitted to the terminal device of each participant, the video stream data to be transmitted to the terminal device is the terminal device with the maximum volume of the audio stream data among the terminal devices other than the terminal device. 8. The stream data distribution method according to claim 6, further comprising a rule that video stream data should be selected. In the step of analyzing the characteristics of the audio stream data transmitted from each participant's terminal device, the characteristics relating to the volume of the audio stream data transmitted from each participant's terminal device are analyzed,
The rule table is
As information of the video stream data to be transmitted to each participant's terminal device, the video stream data to be transmitted to the terminal device has a predetermined threshold value for the volume of the audio stream data among the terminal devices other than the terminal device. The stream data distribution method according to claim 6 or 7, further comprising a rule that video stream data transmitted from a terminal device exceeding the above should be selected. The step of selecting video stream data to be transmitted to each participant's terminal device further includes the step of storing the time at which the video stream data was selected in the storage unit of the stream data distribution device,
10. The stream data distribution method according to claim 8, wherein new video stream data is selected after a predetermined time has elapsed from the time when the previous video stream data was selected. A stream data distribution program for causing a computer to execute the stream data distribution method according to any one of claims 6 to 10.

Images