JP4329358B2 - Stream delivery method and stream delivery system - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a stream distribution method, a stream distribution device, a transmission terminal, and a reception terminal, and in particular, receives stream data such as audio and moving images transmitted from one or more transmission terminals simultaneously to one or more reception terminals. The present invention relates to a stream distribution method, a stream distribution apparatus, a transmission terminal, and a reception terminal that convert to stream data suitable for the display capability of the terminal, distribute, display and output.
[0002]
In recent years, broadcasting and multi-point video conferencing have been carried out by distributing audio and video stream data using the Internet.
Various types of Internet terminals are prevalent, and there are a wide variety of terminals that use such broadcasting and multipoint conferences. Stream data that conforms to the display capabilities of users' terminals can be efficiently sent to many terminals. Realization of a delivery method is desired.
[0003]
[Prior art]
In a conventional stream distribution method, in a stream distribution apparatus that relays and distributes stream data transmitted from a transmission terminal, terminal capability information such as resolution, bit rate, and frame rate indicating the degree of display output capability of the reception terminal Is received from the receiving terminal in advance, and the received stream data is decoded and encoded again in a form suitable for the capability of each receiving terminal, and then distributed to each receiving terminal.
[0004]
FIG. 17 shows a stream distribution method according to the prior art.
The transmission terminals 10a and 10b and the reception terminals 20a and 20b are connected to the stream distribution device 30 via the network 40.
Each transmission terminal is provided with a stream transmission unit that transmits stream data. In the embodiment of FIG. 17, the transmission terminals 10a and 10b are provided with stream transmission units 11a and 11b that transmit stream data, respectively.
[0005]
The stream distribution device 30 is provided with a buffer and a decoding unit for temporarily storing stream data corresponding to the connected transmitting terminal, and a combining unit, an encoding unit, A terminal capability information reception storage unit is provided. In the embodiment of FIG. 17, a buffer 34a and a decoding unit 31a are provided corresponding to the transmitting terminal 10a, a buffer 34b and a decoding unit 31b are provided corresponding to the transmitting terminal 10b, and a combining unit 32a is provided corresponding to the receiving terminal 20a. The encoding unit 33a and the terminal capability information reception storage unit 35a are respectively provided with a synthesis unit 32b, an encoding unit 33b, and a terminal capability information reception storage unit 35b corresponding to the reception terminal 20b.
[0006]
The receiving terminal is provided with a terminal capability information notifying unit, a decoding unit, and a display output unit. In the embodiment of FIG. 17, the receiving terminal 20a includes a terminal capability information notifying unit 21a, a decoding unit 22a, and a display output unit 23a. The terminal 20b is provided with a terminal capability information notification unit 21b, a decryption unit 22b, and a display output unit 23b.
FIG. 17 shows an example in which the stream data Sa and Sb transmitted from the transmission terminals 10a and 10b are distributed to the reception terminals 20a and 20b by the stream distribution device 30.
[0007]
The stream distribution device 30 is notified in advance of the capability information Na and Nb from the receiving terminals 20a and 20b, and stores and holds them in the terminal capability information reception storage units 35a and 35b provided corresponding to the respective receiving terminals.
Next, the stream data Sa and Sb transmitted from the transmission terminals 10a and 10b are temporarily stored in the buffers 34a and 34b, and then decoded into stream data Da and Db by the decoding units 31a and 31b.
[0008]
The combining block 32a provided corresponding to the receiving terminal 20a combines the decoded stream sequences Da and Db into a stream sequence Ea for the receiving terminal 20a, and then the encoding unit 33a receives and stores the terminal capability information Referring to unit 35a, it is encoded into code Ta in a form suitable for the display output capability of receiving terminal 20a, and distributed to receiving terminal 20a.
The same applies to the delivery of stream data to the receiving terminal 20b, and the synthesis block 32a provided corresponding to the receiving terminal 20a synthesizes the decoded stream sequences Da and Db to generate a stream sequence Eb for the receiving terminal 20b. After that, the encoding unit 33b refers to the terminal capability information reception storage unit 35b, encodes it into a code Tb suitable for the display output capability of the receiving terminal 20b, and distributes it to the receiving terminal 20b.
[0009]
In the configuration of FIG. 17, the number of transmitting terminals is 2 and the number of receiving terminals is 2, but the same applies to cases other than 2, and the stream distribution device 30 decodes the stream data from each transmission terminal and distributes it. After synthesizing the decoded stream data from all transmitting terminals for each receiving terminal, it is encoded into a form suitable for the terminal capability of the receiving terminal and distributed to the receiving terminal.
[0010]
In the embodiment shown in FIG. 17, the transmitting terminal and the receiving terminal are expressed as separate devices, but the functions of both the transmitting terminal and the receiving terminal can be included in the same terminal as in the case of a multi-ground video conference. In that case, stream data from a terminal that is a delivery destination of the stream data after the synthesis is not included in the processing target of the synthesis unit. For example, when the transmitting terminal 10a and the receiving terminal 20a are configured as the same terminal, the combining unit 32a sets the stream data from all the transmitting terminals other than the transmitting terminal 10a as a target for combining.
[0011]
In the embodiment shown in FIG. 17, a plurality of transmission terminals are assumed in the case of a multi-ground video conference. However, in the case of stream broadcasting, a single transmission terminal can be configured. At this time, the function of the stream distribution device 30 may be incorporated in the transmission terminal.
One of the stream data encoding methods suitable for the display output capability of the receiving terminal is a hierarchical encoding method.
[0012]
FIG. 16 is an outline of hierarchical encoding.
Hierarchical coding takes the case of image stream data as an example, and decomposes a single image stream data into several streams with different image quality to form a hierarchical stream sequence leveled according to the degree of image quality. Is.
A case will be described as an example where image stream data is divided into levels according to image resolution levels to form a hierarchical stream sequence.
[0013]
In the transmission side processing for hierarchically encoding and transmitting image stream data, first, in encoding E1, the original stream data is encoded at the lowest resolution, and the result is set as a hierarchical stream U1 of hierarchical level 1. Next, in encoding E2, the difference data between the result of encoding the original image at medium resolution and the low-resolution stream data created in encoding E1 is encoded, and the result is the hierarchical level 2 hierarchical stream U2. Further, in encoding E3, the difference data between the result of encoding the original image at a high resolution and the medium resolution stream data generated by the encoding E2 is encoded, and the result is stored in the hierarchical stream U3 of the hierarchical level 3 And
[0014]
The hierarchical stream sequence composed of the hierarchical streams U1, U2, and U3 generated above is transmitted to the receiving side.
In the reception-side process, first, the low-resolution hierarchical stream U1 is decoded by the decoding D1, and the low-resolution decoded data 1 is obtained. If the display output capability on the receiving side is at a low resolution level, the decoded data 1 is displayed and output, and the process ends.
[0015]
Next, the result of decoding the hierarchical stream U2 in the decoding E2 is additionally added to the decoded data 1 decoded in the decoding D1, thereby obtaining the medium-resolution decoded data 2. Here, when the display output capability on the receiving side is the medium resolution level, the decoded data 2 is displayed and output, and the process is terminated.
Further, the result of decoding the hierarchical stream U3 in the decoding E3 is additionally added to the decoded data 2 decoded in the decoding D2, thereby obtaining the high-resolution decoded data 3, which is displayed and output, and the processing is terminated. To do.
[0016]
As described above, by encoding the stream data into a hierarchical stream sequence, the receiving terminal can display and output using the hierarchical stream from the hierarchical stream sequence to a hierarchical level corresponding to the display capability of the receiving terminal. Thus, the transmitting side can efficiently cope with receiving terminals having various display output capabilities by only transmitting one layer stream sequence.
[0017]
In the above description, the method of selecting a layer stream suitable for its own terminal from the layer stream sequence received on the reception side has been described, but when the transmission side knows the display capability of the reception terminal, A hierarchical stream up to a resolution corresponding to the display capability can be selected and transmitted to the receiving terminal.
In the outline of hierarchical encoding in FIG. 16, an example of hierarchical encoding of three layers is shown, but the number of layers is not limited to this.
[0018]
In the stream delivery method of the prior art shown in FIG. 17 above, stream data once received by the stream delivery device is decoded and then encoded into a form suitable for the capability of the receiving terminal, but this is partly improved Using hierarchical encoding method and IP multicast, layer encoded stream data transmitted from one transmitting terminal is adjusted to a plurality of receiving terminals into a hierarchical stream sequence suitable for the receiving terminal capability. Thus, a method for distributing stream data efficiently without performing decoding and re-encoding is disclosed (for example, see Patent Document 1).
[0019]
[Patent Document 1]
JP 2001-352533 A (3rd to 4th pages, FIG. 1)
[0020]
[Problems to be solved by the invention]
In the conventional stream distribution technique, stream data transmitted from a transmission terminal is once decoded, re-encoded into a form suitable for the capability of the reception terminal, and distributed to the reception terminal. In addition, in order to efficiently distribute stream data that conforms to the display capability of the receiving terminal while suppressing the load on the stream distributing device, a layer transmitted from one transmitting terminal is used by using a layer encoding technique. There has been proposed a method of distributing encoded stream data in a form that efficiently matches the display capabilities of a plurality of receiving terminals without performing decoding and encoding processes.
[0021]
However, when performing a multipoint video conference or the like, it is indispensable to efficiently distribute stream data from a plurality of transmitting terminals to a plurality of receiving terminals, and stream data that is not hierarchically encoded is transmitted from the transmitting terminal. In some cases, conventional methods cannot cope with them.
The present invention provides a stream data distribution method in which stream data transmitted from one or more transmission terminals or stream data that is not hierarchically encoded is displayed to one or more reception terminals. An object of the present invention is to provide a stream distribution method that efficiently distributes stream-encoded stream data suitable for the capability.
[0022]
[Means for Solving the Problems]
In a first aspect of the present invention, in the stream distribution method for relaying a stream distribution device and distributing and displaying stream data from a transmission terminal to a reception terminal, the reception terminal has one or more types of display output capability values. The terminal capability information included is notified in advance to the stream distribution device, the stream distribution device holds the terminal capability information for all receiving terminals, and the transmitting terminal hierarchically encodes the stream data to a hierarchical level. Necessary for disassembling into a stream stream corresponding to stream data and displaying and outputting to the receiving terminal for each layer stream corresponding to one or more kinds of capability values included in the terminal capability information The capability value is set and the layered information and the layered stream sequence are transmitted to the stream distribution device as layered information, Based on the terminal capability information and the layered information, the stream distribution apparatus selects a layer stream from the layer stream sequence so that the receiving terminal can simultaneously display and output stream data from each transmitting terminal. A terminal layer stream sequence, generating terminal composite information designating a display output area of stream data from each transmitting terminal, transmitting the terminal composite information and the terminal layer stream sequence to the receiving terminal, and receiving terminal Is configured to display and output the terminal layer stream sequence according to the terminal composite information.
[0023]
According to the first invention, the stream-distributed stream data from one or more transmitting terminals is distributed to each receiving terminal without performing decoding and re-encoding processing, and each receiving terminal Hierarchized stream data from all transmitting terminals can be simultaneously displayed and output at a quality level adapted to the capabilities of the receiving terminal.
A second invention is a stream delivery method for relaying a stream delivery device, delivering stream data from a transmission terminal to a reception terminal, and displaying the stream data.
The receiving terminal notifies the stream distribution device in advance of terminal capability information including one or more types of display output capability values, and the stream distribution device holds the terminal capability information for all receiving terminals, The stream distribution apparatus determines, based on the terminal capability information, a target number of layers for hierarchically encoding received stream data and a required capability value necessary for displaying and outputting each layer stream to the receiving terminal. Layered information, and after decoding the received stream data, layered coding is performed based on the layered information to form a layered stream sequence, and the layered stream is based on the terminal capability information and the layered information. From the column, the receiving terminal selects a hierarchical stream so that the stream data from each transmitting terminal can be displayed and output at the same time. Generating a terminal composite information indicating a display output area in a receiving terminal for displaying and outputting the terminal layer stream string, and transmitting the terminal composite information and the terminal layer stream string to the receiving terminal; The receiving terminal is configured to display and output the terminal layer stream sequence according to the terminal composite information.
[0024]
According to the second invention, stream data that has not been hierarchically encoded from one or more transmitting terminals is once decoded by the stream distribution apparatus, and then encoded into a hierarchical stream sequence that conforms to the capabilities of all receiving terminals. The subsequent processing can select and distribute only the hierarchical stream suitable for the capability of each receiving terminal in the same manner as in the first invention, and encoding for each receiving terminal becomes unnecessary. Further, by combining with the first invention, it becomes possible to cope with terminals having various capabilities including a transmitting terminal that does not support hierarchical coding, and a more flexible stream distribution system can be constructed.
[0025]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 shows a first embodiment of the present invention in the stream distribution method (1) of the present invention.
One or more transmission terminals and one or more reception terminals are connected to the stream distribution device 70 via the network 40, and the stream data transmitted from the transmission terminal is distributed to the one or more reception terminals via the stream distribution device 70. Is done.
[0026]
In the embodiment of FIG. 1, the configuration of the transmission terminals 50a and 50b and the reception terminals 60a and 60b is illustrated, but both the transmission terminal and the reception terminal can be configured by one or more arbitrary numbers.
The transmitting terminal 50a hierarchically encodes the stream data to be displayed and output at the receiving terminal by the hierarchical stream generation / transmission unit 51a to generate the hierarchical stream sequence Ua, and at the same time, the hierarchical information notification unit 52a performs the hierarchical stream sequence Ua Then, the hierarchical information Xa including the attribute information of the hierarchical stream included in is generated, and then the hierarchical information Xa and the hierarchical stream sequence Ua are transmitted to the stream distribution device 70.
[0027]
For the transmission terminal 50b, similarly to the transmission terminal 50a, stream data is hierarchically encoded by the hierarchical stream generation / transmission unit 51b to generate a hierarchical stream sequence Ub, and at the same time, the hierarchical information notification unit 52b Is generated, and the hierarchical information Xb and the hierarchical stream string Ub are transmitted to the stream distribution device 70.
[0028]
The same applies when three or more transmission terminals are provided.
The stream distribution device 70 is provided with a hierarchical information reception storage unit, a buffer, and an output suppression section detection storage unit corresponding to each transmission terminal, and a terminal capability information reception storage unit, distribution selection information corresponding to each reception terminal A reception storage unit, a composite information generation storage unit, and a hierarchy selection unit are provided.
[0029]
In the embodiment shown in FIG. 1, a hierarchical information reception storage unit 71a, a buffer 77a, and an output suppression section detection storage unit 76a are provided corresponding to the transmission terminal 50a, and a terminal capability information reception storage unit corresponding to the reception terminal 60a. 75a, a distribution selection information reception storage unit 74a, a composite information generation storage unit 72a, and a hierarchy selection unit 73a are provided.
Similarly, a hierarchical information reception storage unit 71b, a buffer 77b, and an output suppression section detection storage unit 76b are provided corresponding to the transmission terminal 50b, and a terminal capability information reception storage unit 75b, distribution selection information corresponding to the reception terminal 60b. A reception storage unit 74b, a composite information generation storage unit 72b, and a hierarchy selection unit 73b are provided.
[0030]
The same applies when three or more transmitting terminals and receiving terminals are provided.
When the stream distribution device 70 receives the hierarchical stream sequence Ua from the transmission terminal 50a, the stream distribution device 70 temporarily stores it in the buffer 77a, and then the output suppression section detection storage unit 76a means a time zone in which the display output can be suppressed from the hierarchical stream sequence Ua The output suppression section is detected and stored.
[0031]
The same applies to the case where the hierarchical stream sequence Ub is received from the transmitting terminal 50b, and after storing it once in the buffer 77b, the output suppression interval detection storage unit 76a detects and stores the output suppression interval from the hierarchical stream sequence Ub.
On the other hand, the hierarchical information Xa transmitted from the transmission terminal 50a is stored in the hierarchical information reception storage unit 71a. The same applies to the hierarchized information Xb transmitted from the transmitting terminal 50b, and is stored in the hierarchized information reception storage unit 71b.
[0032]
The terminal capability information reception storage unit 75a receives and stores terminal capability information Na indicating the display output capability of the reception terminal 60a notified from the reception terminal 60a in advance, and the distribution selection information reception storage unit 74a is previously received by the reception terminal 60a. The distribution selection information Za of the receiving terminal 60a notified from is received and stored. This distribution selection information Za is used when the stream data from all transmitting terminals cannot be displayed and output at the same time by the display output capability of the receiving terminal 60a indicated by the terminal capability information Na. Used to select and limit stream data to be distributed to 60a.
[0033]
The same applies to the terminal capability information reception storage unit 75b and the delivery selection information reception storage unit 74b, and the terminal capability information Na and the delivery selection information Za are previously received from the receiving terminal 60b and stored in advance.
The composite information generation storage unit 72a refers to the hierarchical information of each transmission terminal stored in the hierarchical information reception storage unit and the capability information of the reception terminal 60a stored in the terminal capability information storage unit 75a. A terminal allowable layer level value common to all transmitting terminals that can be received by the receiving terminal 60a for simultaneously displaying and outputting a layer stream sequence from all transmitting terminals is selected. Then, it generates composite information for each transmitting terminal necessary for the receiving terminal 60a to display and output the stream data distributed from each transmitting terminal, and the selected terminal allowable hierarchy level value and composite for all transmitting terminals A collection of information is set as composite information Ca and stored in the composite information generation storage unit 72a. The composite information for each transmission terminal includes transmission terminal identification information, a screen display start position when stream data is displayed on the screen, and output suppression section information indicating whether display output may be suppressed. ing.
[0034]
When the above-mentioned terminal allowable layer level value cannot be selected, that is, even if it is limited to the lowest layer layer stream, the stream data from all transmitting terminals cannot be displayed and output at the same time with the terminal capability of the receiving terminal 60a. Restricts the transmission terminal that is the target of stream data distribution processing in accordance with the distribution selection information Za held in the distribution selection information reception storage unit 74a, and the above-mentioned composite for the limitedly selected transmission terminal The information generated and collected is held in the composite information generation storage unit 72a as composite information Ca.
[0035]
In the embodiment shown in FIG. 1, the composite information generation storage unit 72a includes the hierarchization information stored in the hierarchization information reception storage units 71a and 71b and the reception terminal 60a stored in the terminal capability information storage unit 75a. With reference to the terminal capability information, the receiving terminal 60a selects a terminal allowable layer level value that is allowed to simultaneously display and output the stream data from the transmitting terminals 50a and 50b, and at the same time, composite information about the transmitting terminals 50a and 50b is obtained. Collect and hold as composite information Ca.
[0036]
Next, for the transmission terminals registered in the composite information Ca, the hierarchy selection unit 73a is a hierarchy from the lowest hierarchy in the hierarchical stream sequence from each transmission terminal to the hierarchy indicated by the terminal allowable hierarchy level value. A stream is selected, and a hierarchical stream sequence Ta composed of the selected hierarchical streams for each transmitting terminal is transmitted to the receiving terminal 60a. At the same time, from the composite information Ca, the receiving terminal 60a extracts the screen display start position, which is information for arranging the stream data from each transmitting terminal on the screen, and sends it to the receiving terminal 60a as terminal composite information Ta To do.
[0037]
The same applies to the composite information generation storage unit 72b and the hierarchy selection unit 73b.The composite information generation storage unit 72b includes the hierarchical information of each transmission terminal stored in the hierarchical information reception storage unit corresponding to each transmission terminal, and , By referring to the capability information of the receiving terminal 60b stored in the terminal capability information storage unit 75b, it is acceptable for the receiving terminal 60b to simultaneously display and output the stream data from all the transmitting terminals. Select the terminal allowed hierarchy level value. Then, it generates composite information for each transmitting terminal necessary to display and output the stream data distributed from each transmitting terminal to the receiving terminal 60b, and the selected terminal allowable hierarchy level value and all the transmitting terminals The composite information is collected into composite information Cb, which is held in the composite information generation storage unit 72b.
[0038]
When the above-mentioned terminal allowable layer level value cannot be selected, that is, even if it is limited to the lowest layer layer stream, the stream data from all transmitting terminals cannot be displayed and output at the same time with the terminal capability of the receiving terminal 60b. Restricts the transmission terminal that is the target of the stream data distribution process according to the distribution selection information Zb held in the distribution selection information reception storage unit 74b, and the above-mentioned composite for the limitedly selected transmission terminal The information generated and collected is held in the composite information generation storage unit 72b as composite information Cb. In this case, the terminal allowable hierarchy level value is set to the lowest hierarchy level.
[0039]
In the embodiment shown in FIG. 1, the composite information generation storage unit 72b includes the hierarchization information stored in the hierarchization information reception storage units 71a and 71b and the reception terminal 60b stored in the terminal capability information storage unit 75b. Referring to the terminal capability information, the receiving terminal 60b selects a terminal allowable layer level value that is allowed to simultaneously display and output the stream data from the transmitting terminals 50a and 50b, and generates composite information of the transmitting terminals 50a and 50b. And collect and hold as composite information Cb.
[0040]
Next, for the transmission terminal registered in the composite information Cb, the hierarchy selection unit 73b has a hierarchy from the lowest hierarchy in the hierarchical stream sequence from each transmission terminal to the hierarchy indicated by the terminal allowable hierarchy level value. A stream is selected, and a layer stream sequence Tb composed of the selected layer stream for each transmitting terminal is transmitted to the receiving terminal 60b. At the same time, from the composite information Cb, the receiving terminal 60b extracts the screen display start position, which is information for arranging the stream data from each transmitting terminal on the screen, and sends it to the receiving terminal 60b as terminal composite information Tb. To do.
[0041]
Although the above description has been made assuming that the number of transmitting terminals is 2 and the number of receiving terminals is 2, the same applies to the case where the number is 3 or more.
In the receiving terminal 60a, the terminal capability information notification unit 61a transmits terminal capability information Na indicating the display output capability of the terminal itself to the stream distribution device 70 in advance. Also, the delivery selection information notifying unit 65a selects a transmitting terminal to be displayed and output when stream data from all transmitting terminals cannot be displayed at the same time with the terminal capability of the own terminal even if only the lowest layer stream is received. The distribution selection information Za for this is transmitted to the stream distribution device 70 in advance.
[0042]
The receiving terminal 60a holds the terminal composite information Va transmitted from the stream distribution device 70 in the terminal composite information reception storage unit 62a. Here, the terminal composite information Va includes a pair of identification information of each transmitting terminal and a screen display start position for identifying an area on the screen for displaying and outputting stream data from the transmitting terminal. By searching for information identifying the transmitting terminal included in the stream data as a key, an area on the screen where the stream data should be displayed and output can be specified.
[0043]
On the other hand, the layer stream sequence Ta selected and transmitted by the stream distribution device 70 is decoded by the decoding unit 63a, and then the display output unit 64a stores the terminal composite stored in the terminal composite information reception storage unit 62a. Display output according to information Va.
The same applies to the receiving terminal 60b. The terminal capability information notification 61b transmits the terminal capability information Nb to the stream distribution device 70 in advance, and the distribution selection information notification unit 65b transmits the distribution selection information Zb to the stream distribution device 70 in advance. The terminal composite information reception storage unit 62b holds the terminal composite information Vb transmitted from the stream distribution device 70. On the other hand, the layer stream sequence Tb selected and transmitted by the stream distribution device 70 is decoded by the decoding unit 63b, and then the display output unit 64b stores the terminal composite stored in the terminal composite information reception storage unit 62b. Display output according to information Vb.
[0044]
The same applies when the number of receiving terminals is 3 or more.
In the embodiment of FIG. 1, the transmitting terminal and the receiving terminal are expressed as separate devices, but the functions of both the transmitting terminal and the receiving terminal can be included in the same terminal as in the case of a multi-ground video conference. In this case, the stream distribution processing target of the stream distribution device 70 does not include stream data from a terminal that is a stream data distribution destination. For example, when the transmission terminal 50a and the reception terminal 60a are configured as the same terminal, the composite information generation storage unit 72a and the hierarchy selection unit 73a perform the above stream stream data from all transmission terminals other than the transmission terminal 10a. Process the delivery.
[0045]
In the embodiment shown in FIG. 1, a plurality of transmission terminals are assumed in the case of a multi-ground video conference, but the number of transmission terminals can be set to 1 as in the case of stream broadcasting. The function of the distribution device 70 can also be integrated into the transmission terminal.
The composite information held in the composite information generation storage unit, the terminal capability information stored in the terminal capability information reception storage unit, and the distribution selection information stored in the distribution selection information reception storage unit have a storage area for each receiving terminal. Information about all receiving terminals may be stored in a lump without providing them.
[0046]
Similarly, the hierarchization information for each transmission terminal stored in the hierarchization information reception storage unit and the output suppression interval information stored in the output suppression interval detection storage unit may be stored collectively for all transmission terminals.
FIG. 2 is a transfer sequence (1) according to the stream distribution method of the present invention, and will be described for each processing step according to the configuration of the embodiment of FIG.
[0047]
S01. The receiving terminal 60a transmits the terminal capability information Na and the distribution selection information Za to the stream distribution device 70 in advance.
Here, the terminal capability information Na is information indicating the display output capability of the receiving terminal 60a. For example, in the configuration of the terminal capability information of the present invention shown in FIG. 3 described later, the maximum capability, the maximum bit rate, and the maximum frame rate are used. Become.
[0048]
When the stream distribution device 70 determines that the distribution selection information Za cannot be displayed and output to the receiving terminal 60a simultaneously with the display output capability indicated by the terminal capability information Na, the stream data from all the transmitting terminals cannot be displayed simultaneously. This is information used when selecting and limiting stream data to be distributed to the receiving terminal 60a. For example, in the configuration of the delivery selection information shown in FIG. 8 to be described later, the display output priority indicating the priority of the display output of the stream data from each transmission terminal or the information indicating whether or not the stream data is output from each transmission terminal. Information.
[0049]
For the terminal capability information Na and the distribution selection information Za, refer to the configuration of the terminal capability information of the present invention shown in FIG. 3 and the configuration of the distribution selection information of the present invention shown in FIG.
S02. The stream distribution device 70 stores the terminal capability information Na and the distribution selection information Za received from the reception terminal 60a in the terminal capability reception storage unit 75a and the distribution selection information reception storage unit 74a, respectively.
[0050]
S03. Similarly to step S01, the receiving terminal 60b transmits the terminal capability information Nb and the delivery selection information Zb to the stream delivery device 70 in advance.
S04. Similarly to step S02, the stream distribution device 70 stores the terminal capability information Na and the distribution selection information Za received from the reception terminal 60a in the terminal capability reception storage unit 75a and the distribution selection information reception storage unit 74a, respectively. To do.
[0051]
In the description of steps S01 to S04 above, the processing of the receiving terminal 60b is described after the processing of the receiving terminal 60a. The processing order of steps S01 to S02 and steps S03 to S04 may be reversed depending on the transmission timing T01 of the receiving terminal, but the processing contents are the same.
S05. The transmission terminal 50a receives the layered information Xa including the required capability value of display output necessary for displaying and outputting each layer stream in the layer stream sequence Ua to the reception terminal, and the layer stream sequence Ua. Transmit to the stream distribution device 70.
[0052]
For an example of the hierarchized information Xa, refer to the configuration of the hierarchized information of the present invention in FIG.
S06. The stream distribution apparatus 70 stores the received hierarchical information Xa in the hierarchical information reception storage unit 71a.
In addition, the received hierarchical stream sequence Ua is stored in the buffer 77a, and at the same time, an output suppression section that can suppress display output to the receiving terminal is detected, and the result is stored in the output suppression section detection storage unit 76a.
[0053]
S07. Similar to step S05 above, the transmitting terminal 50b includes the layered information Xb including the required output capability value for displaying and outputting each layer stream of the layer stream sequence Ub to the receiving terminal, and The hierarchical stream sequence Ub is transmitted to the stream distribution device 70.
S08. Similarly to step S06, the stream distribution device 70 stores the received hierarchical information Xb in the hierarchical information reception storage unit 71b.
[0054]
In addition, the received hierarchical stream sequence Ub is stored in the buffer 77b, and at the same time, an output suppression section that can suppress display output to the receiving terminal is detected, and the result is stored in the output suppression section detection storage unit 76a.
In the description of steps S05 to S08 above, the process of the transmission terminal 50b is described after the process of the transmission terminal 50a. The processing order of steps S05 to S06 and steps S07 to S08 may be reversed depending on the transmission timing T02 of the transmitting terminal, but the processing contents are the same.
[0055]
S09. The stream distribution device 70 receives the terminal capability information Na and the distribution selection information Za of the receiving terminal 60a stored in step S01, the hierarchical information Xa and Xb stored in steps S06 and S08, and output suppression Composite information Ca corresponding to the receiving terminal 60a is generated from the output suppression section information Pa and Pb stored in the section detection storage unit, and stored in the composite information generation storage unit 72a. The details of the composite information Ca are the composition of the composite information of the present invention shown in FIG. 5 to be described later, and the details of the composite information generation storage section 72a are the operation flowchart of the composite information generation and storage section of the present invention of FIG. See
[0056]
Here, in the terminal capability of the receiving terminal 60a, even if it is limited to the lowest layer stream, if it is determined that the stream data from all the transmitting terminals cannot be displayed and output at the same time, the distribution selection information Za is not targeted for distribution. The composite information related to the transmitting terminal selected for is not registered in the composite information Ca.
S10. From the composite information Ca generated in step S09, screen arrangement information when the receiving terminal 60a displays and outputs the stream data from each transmitting terminal is extracted and transmitted to the receiving terminal 60a as terminal composite information Va.
[0057]
For details of the terminal composite information Va, refer to the configuration of the terminal composite information of the present invention in FIG.
In addition, a hierarchical stream up to a hierarchical level corresponding to the terminal allowable hierarchical level value La of the receiving terminal 60a set in the composite information Ca is selected from the hierarchical stream sequence from the transmitting terminal registered in the composite information Ca. Then, the hierarchical stream sequence Ta is transmitted to the receiving terminal 60a.
[0058]
Therefore, since the transmission selection information Za held in the distribution selection information reception storage unit 74a in the process of step S09 is valid and is excluded from the distribution target, it is not registered in the composite information Ca. The hierarchical stream sequence from the transmitting terminal is not distributed to the receiving terminal 60a.
In the embodiment shown in FIG. 2, the transmission terminals 50a and 50b are described on the assumption that they are not excluded from the distribution target, and the hierarchical stream sequence Ua stored in the buffer 77a for the transmission terminal 50a and the transmission terminal 50b are described. The hierarchical stream string Ta is selected from the hierarchical stream string Ub stored in the buffer 77b to the hierarchical level corresponding to the terminal allowable hierarchical level value La of the receiving terminal 60a set in the composite information Ca. And transmitted to the receiving terminal 60a.
[0059]
The above describes the case where the number of transmitting terminals is 2, but the same applies to the case where the number of transmitting terminals is other than 2, and the sequence of hierarchical streams below the terminal allowable hierarchical level value La from all transmitting terminals is the hierarchical stream sequence Ta. Is transmitted to the receiving terminal 60a.
S11. The receiving terminal 60a, after decoding the hierarchical stream sequence Ta transmitted from the stream distribution device 70, from each transmission terminal according to the information included in the terminal composite information Va similarly notified from the stream distribution device 70 Simultaneously display and output the stream data.
[0060]
The terminal composite information Va includes transmission terminal identification information and a screen display start position for each transmission terminal, and the reception terminal 60a simultaneously receives stream data from each transmission terminal based on the terminal composite information Va. Display output is possible.
For the display output to the receiving terminal 60a, see the display output example of the receiving terminal of the present invention in FIG.
[0061]
S12. Similarly to step S09, the stream distribution device 70, the terminal capability information Nb and distribution selection information Zb of the reception terminal 60b stored in step S01, the transmission terminal 50a stored in steps S06 and S08, Composite information Cb corresponding to the receiving terminal 60b is generated from the hierarchical information Xa, Xb of 50b and the output suppression section information Pa, Pb1 stored in the output suppression section detection storage units 76a, 76b, and composite information The data is stored in the generation storage unit 72b.
[0062]
S13. From the composite information Cb generated in step S12, as in step S10, the receiving terminal 60b extracts terminal composite information Vb that indicates an area for displaying and outputting stream data from each transmitting terminal. Send to. On the other hand, from the hierarchical stream sequence Ua stored in the buffer 74a for the transmitting terminal 50a and the hierarchical stream sequence Ub stored in the buffer 74b for the transmitting terminal 50b, the receiving terminal 60b included in the composite information Cb The hierarchical stream up to the hierarchical level corresponding to the terminal allowable hierarchical level value Lb is extracted, and the extracted hierarchical stream sequence Tb is transmitted to the receiving terminal 60b.
[0063]
S14. The receiving terminal 60b decodes the hierarchical stream sequence Tb transmitted from the stream distribution device 70, and then transmits information from each transmission terminal according to the information included in the terminal composite information Vb notified from the stream distribution device 70. Simultaneously display and output the stream data.
In the description of steps S09 to S14 above, the processing of the receiving terminal 60b is described after the processing of the receiving terminal 60a. The processing order of steps S09 to S11 and steps S12 to S14 may be reversed depending on the transmission timing G03 of the stream distribution device, but the processing contents are the same.
[0064]
S15. Repeat the steps S05 to S14.
As a method of repeating steps S05 to S14, the stream data transmitted from all the transmitting terminals is temporarily stored in the buffer, and the stream distribution apparatus is sufficient to prevent the stream data received from all the transmitting terminals from being lost. This can be realized by starting the processing of steps S06 to S11 in a short cycle.
[0065]
In the above description, the number of transmitting terminals is 2 and the number of receiving terminals is 2. However, the same applies to a number other than 2.
FIG. 3 shows the configuration of terminal capability information according to the present invention.
The terminal capability information Na for the receiving terminal 60a is shown, and the maximum total resolution Na (MWH) represents the amount of image information that can be displayed on the terminal screen, and is the number of pixels that can be displayed on the terminal screen. The maximum total bit rate Na (MFR) is the number of bits that the terminal can receive per second. The maximum total frame rate Na (MFR) is an index representing how many times the screen can be rewritten per second in the screen display of the terminal.
[0066]
Here, a description will be given based on an example using the above three types of terminal capability information, but the type of terminal capability information is not limited to this. 3 describes the receiving terminal 60a, the same applies to receiving terminals other than the receiving terminal 60a.
FIG. 4 shows the structure of the hierarchized information of the present invention.
[0067]
The hierarchical information Xa corresponding to the transmission terminal 50a is shown, and the terminal information necessary for displaying and outputting the hierarchical stream corresponding to each hierarchical level in the hierarchical stream sequence that is the hierarchically encoded stream data to the terminal Includes required capacity values. The types of required capability values included in the hierarchized information are configured to correspond to the types of terminal capability information shown in FIG.
[0068]
Hierarchical information Xa (1) of hierarchical level 1 which is the lowest hierarchical level is stream identification information Xa (1, S) for identifying a hierarchical stream corresponding to hierarchical level 1, and required resolution Xa (1, WH). The required bit rate Xa (1, BR) and the required frame rate Xa (1, FR). The same applies to other hierarchical levels. For example, hierarchical information Xa (n) at hierarchical level n includes stream identification information Xa (n, S) for identifying a hierarchical stream corresponding to hierarchical level n, and required resolution Xa ( n, WH), required bit rate Xa (n, BR) and required frame rate Xa (n, FR).
[0069]
In the embodiment of FIG. 4, the transmission terminal 50a is described, but the same applies to transmission terminals other than the transmission terminal 50a.
FIG. 5 shows the composition of the composite information of the present invention.
The composite information is generated and held in a composite information generation / storage unit provided corresponding to each receiving terminal connected to the stream distribution device 70.
[0070]
The embodiment of FIG. 5 shows composite information Ca corresponding to the receiving terminal 60a.
The terminal allowable hierarchy level La is transmitted from each transmitting terminal in order to simultaneously display and output stream data from all transmitting terminals connected to the stream distribution device 70 with the display output capability possessed by the receiving terminal 60a. This indicates the value of the hierarchical level to be selected from the hierarchical stream sequence, and is a hierarchical level value common to all transmitting terminals.
[0071]
That is, a layer stream from the lowest layer level to the layer indicated by the terminal allowable layer level value La is selected from the layer stream sequence transmitted from each transmitting terminal and transmitted to the receiving terminal 60a. The receiving terminal 60a can display and output the stream data from all the transmitting terminals at the same time within the display output capability of the receiving terminal 60a by displaying and outputting the hierarchical stream sequence selected here.
[0072]
Composite information Ca (50a) is composite information that summarizes necessary information to display and output the stream data transmitted from the transmitting terminal 50a to the receiving terminal 60a. The transmitting terminal identification information Ca (50a, ID) and screen display start Position Ca (50a, DP) and output suppression section information Ca (50a, P) are included.
The transmission terminal identification information Ca (50a, ID) is identification information of the transmission terminal 50a. The screen display start position Ca (50a, DP) is information indicating the display start position on the screen of the reception terminal 60a that displays and outputs the stream data from the transmission terminal 50a. The output suppression section information Ca (50a, P) is the stream data of the stream data when the distribution of the stream data transmitted from the transmitting terminal is temporarily stopped even if the display output to the receiving terminal is small. Information for temporarily interrupting distribution. For example, there is a silent section indicating that the audio information is interrupted, and a non-operating section indicating that the motion of the image has stopped. In the case of the silent section, the audio stream data is used. Suppress delivery of stream data.
[0073]
The same applies to the composite information required to display and output the stream data transmitted from other transmission terminals other than the transmission terminal 50a to the reception terminal 60a. For example, composite information Ca (50z) corresponding to the transmission terminal 50z The transmission terminal identification information Ca (50z, ID) is identification information of the transmission terminal to be displayed and output to the reception terminal 60a, and the screen display start position Ca (50z, DP) is stream data from the transmission terminal 50z. Is the information indicating the display start position on the screen of the receiving terminal 60a that displays and outputs the output suppression section information Ca (50z, P) is temporarily stopped distribution of the stream data transmitted from the transmitting terminal 50z This is information for temporarily interrupting the delivery of the stream data when the influence on the display output to the receiving terminal is small.
[0074]
The composite information Ca is configured by collecting composite information for each transmission terminal, but only the transmission terminal registered in the composite information Ca is a target of stream data distribution to the reception terminal 60a.
In the display output capability indicated in the terminal capability information Na of the receiving terminal 60a, when it is determined that the stream data from all the transmitting terminals cannot be displayed and output at the same time, the composite information Ca is set according to the content of the distribution selection information Za. The transmitting terminal to be registered is limited, and composite information of the transmitting terminal that deviates from the condition of the delivery selection information Za is not included in the composite information Ca.
[0075]
The embodiment of FIG. 5 describes the receiving terminal 60a, but the same applies to receiving terminals other than the receiving terminal 60a.
FIG. 6 shows the structure of the terminal composite information according to the present invention.
The terminal composite information is generated by the composite information generation storage unit by extracting information necessary for display output processing of the receiving terminal from the composite information configuration of the present invention shown in FIG. 5 and transmitted to the corresponding receiving terminal. The
[0076]
FIG. 6 shows an embodiment of the terminal composite information Va corresponding to the receiving terminal 60a, and consists of terminal composite information for each transmitting terminal.
The terminal composite information Va (50a) is terminal composite information corresponding to the transmission terminal 50a, and includes transmission terminal identification information Va (50a, ID) and a screen display start position Va (50a, DP). The contents are the same as the transmission terminal identification information Ca (50a, ID) and the screen display start position Ca (50a, DP) in the composite information configuration of the present invention.
[0077]
The same applies to the terminal composite information corresponding to other transmitting terminals other than the transmitting terminal 50a. For example, in the case of the terminal composite information Va (50z) corresponding to the transmitting terminal 50z, the transmitting terminal identification information Va (50z, ID) and the screen It consists of a display start position Va (50z, DP), and has the same contents as the transmission terminal identification information Ca (50z, ID) and screen display start position Ca (50z, DP) in the composite information configuration of FIG. .
[0078]
The receiving terminal 60a performs display output according to the information included in the terminal composite information Va.
The embodiment of FIG. 6 describes the receiving terminal 60a, but the same applies to receiving terminals other than the receiving terminal 60a.
FIG. 7 is a display output example of the receiving terminal of the present invention.
[0079]
When the transmission terminal connected to the stream distribution device 70 is the transmission terminal 50a, the transmission terminal 50b,..., The transmission terminal 50z, the stream data from each transmission terminal is simultaneously displayed on the screen of the reception terminal 60a. This is an example.
An example in which the screen is divided and allocated so that the size of the display area of the stream data from each transmitting terminal is maximized within the range of the maximum resolution that is one of the terminal capability information of the receiving terminal 60a. It is.
[0080]
The embodiment of FIG. 7 describes the receiving terminal 60a, but the same applies to receiving terminals other than the receiving terminal 60a.
FIG. 8 shows the configuration of distribution selection information according to the present invention.
Examples of the distribution selection information Za of the receiving terminal 60a are shown, and (b) two examples of distribution selection information based on whether display output is possible and (b) distribution selection information based on display output priority are shown.
[0081]
When the stream distribution device 70 cannot select the terminal allowable layer level value La for the receiving terminal 60a, that is, even if it is limited to only the lowest layer layer stream, all the transmitting terminals Stream distribution device 70 restricts the stream data to be delivered to receiving terminal 60a according to this delivery selection information Za, and delivers the stream data within that restriction. Display output.
[0082]
In the delivery selection information based on whether or not display output is possible in (A), the display output enable / disable information indicating whether or not the receiving terminal 60a displays and outputs stream data from the transmitting terminal is set for each transmitting terminal. In the delivery selection information based on the output priority, priority is given to each transmitting terminal, and the stream data from the transmitting terminal having a higher priority is preferentially distributed to the receiving terminal 60a for display output. In FIG. 8 (B), the smaller the number, the higher the priority.
[0083]
The distribution selection information Za is transmitted to the stream distribution device 70 in advance.
The embodiment of FIG. 8 describes the receiving terminal 60a, but the same applies to receiving terminals other than the receiving terminal 60a.
FIG. 9 is an operation flowchart of the composite information generation and storage unit of the present invention, showing an embodiment of the receiving terminal 60a.
[0084]
S7201. Hierarchy that enables receiving terminal 60a to simultaneously display and output stream data from all transmitting terminals based on layering information of all transmitting terminals for maximum resolution Na (MWH), which is one of the terminal capability information of receiving terminal 60a Select the hierarchical level value La (MWH) by capability in the stream column. Here, the permissible tier level value La (MWH) for each capability is assumed to be “1” as the lowest tier level of the tier stream sequence, and counted up as the tier level increases. Further, if the display output capability of the receiving terminal 60a is low and even the hierarchical stream corresponding to the lowest hierarchical level cannot be allowed, the permissible hierarchical level value La (MWH) is set to “0”.
[0085]
For details of the selection process, refer to the operation flowchart for selecting an allowable hierarchy level value relating to the resolution of the present invention shown in FIG.
S7202. A hierarchy in which the receiving terminal 60a can simultaneously display and output stream data from all receiving terminals with the same processing procedure as in step S7201 above for the maximum bit rate Na (MBR) which is one of the terminal capability information of the receiving terminal 60a. Select the hierarchy level value La (MBR) in the stream sequence. Here, it is assumed that the permissible hierarchical level value La (MBR) for each capability is set to “1” as the lowest hierarchical level in the hierarchical stream sequence, and is counted up as the hierarchical level increases. Also, if the display output capability of the receiving terminal 60a is low and even the hierarchical stream corresponding to the lowest hierarchical level cannot be allowed, the permissible hierarchical level value La (MBR) is set to “0”.
[0086]
For details of the selection process, refer to the operation flowchart for selecting an allowable hierarchy level value relating to the bit rate of the present invention shown in FIG.
S7203. Hierarchy in which the receiving terminal 60a can simultaneously display and output stream data from all receiving terminals with the same processing procedure as in step S7202 for the maximum frame rate Na (MFR), which is one of the terminal capability information of the receiving terminal 60a. Select the hierarchy level value La (MFR) in the stream sequence. Here, it is assumed that the permissible hierarchical level value La (MFR) for each capability is incremented as the hierarchical level increases, with the lowest hierarchical level in the hierarchical stream sequence being “1”. If the display output capability of the receiving terminal 60a is low and even the hierarchical stream corresponding to the lowest hierarchical level is not allowable, the capability-specific allowable hierarchical level value La (MFR) is set to “0”.
[0087]
For details of the selection process, refer to the operation flowchart for selecting an allowable hierarchy level value relating to the frame rate of the present invention shown in FIG.
S7204. Determine whether the minimum value of the layer level values La (MWH), La (MBR), La (MFR) selected in steps S7201, S7202, and S7203 above is greater than "0". If it is larger, that is, “1” or more (YES), it means that an appropriate hierarchical level that can be allowed by the display output capability of the receiving terminal 60a has been selected, and the process proceeds to the next Step S7205.
[0088]
If it is not greater than "0", that is, if it is "0" (NO), an appropriate hierarchical level that is acceptable for the display output capability of the receiving terminal 60a cannot be selected, and only the hierarchical stream of the lowest hierarchical level is limited. Even so, it means that the stream data from all the transmitting terminals cannot be displayed and output at the same time, and the process proceeds to step S7209.
S7205. Composite information is generated and stored by regarding the minimum value of the permissible hierarchical level values La (MWH), La (MBR), and La (MFR) selected as the permissible hierarchical level value La that the receiving terminal 60a can accept. This is set in the composite information Ca held by the unit 72a.
[0089]
S7206. For all transmitting terminals, the screen display start position of the receiving terminal 60a that displays and outputs the stream data from the transmitting terminal is generated, and is set in the composite information Ca held by the composite information generation storage unit 72a. For all transmission terminals, reference is made to the output suppression section information held in the output suppression section detection storage unit, and the composite information Ca stored in the composite information generation storage unit 72a is set.
[0090]
S7207. Based on the composite information Ca held by the composite information generation storage unit 72a, the terminal composite information Va for the receiving terminal 60a is created and transmitted to the receiving terminal 60a.
S7208. In order to distribute the stream data to the receiving terminal 60a, the hierarchy selecting unit 73a is activated and the process is terminated.
S7209. The terminal allowable hierarchy level value La of the receiving terminal 60a is set to “1” as the lowest level in the composite information Ca held by the composite information generation storage unit 72a.
[0091]
S7210. Based on the distribution selection information Za held in the distribution selection information reception storage unit 74a, the transmission terminal to be displayed on the receiving terminal 60a is selected. For the selected transmitting terminal, the screen display start position of the receiving terminal 60a that displays and outputs the stream data from the transmitting terminal based on the corresponding hierarchical information, and the output suppression section information are created, and the composite information Ca After storing in the composite information generation storage unit 72a, the process proceeds to step S7207.
[0092]
In the operation flowchart of FIG. 9 described above, the composite information generation storage unit 72a corresponding to the reception terminal 60a has been described, but the same applies to the composite information generation storage unit corresponding to reception terminals other than the other reception terminals 60a.
FIG. 10 is an operation flowchart of the La (MWH) selection process of the present invention, showing details of step S7201 of the operation flowchart of FIG. 9, and the contents of each process step are as follows.
[0093]
S7201-1. A variable for temporarily storing a hierarchy level value is represented by i, and “1” is set as an initial value.
S7201-2. It is determined whether or not the value of the variable i is greater than the maximum hierarchical level value of the hierarchical stream sequence held in all buffers. Here, the maximum hierarchical level value of the hierarchical stream sequence is “2” or more in view of the meaning of the hierarchical stream sequence.
[0094]
If the value of the variable i is greater than the maximum hierarchy level value (YES), the process proceeds to step S7201-5 to determine the allowable hierarchy level value La (MWH) of the receiving terminal 60a, and the variable i If the value is not larger than the maximum hierarchy level value, that is, if it is equal to or less than the maximum hierarchy level value (NO), the process proceeds to the next step S7201-3 to continue the processing.
S7201-3. Based on the hierarchization information received from all the transmitting terminals, the total value WH (i) of the resolution of the hierarchy level i is calculated.
[0095]
Using the notation described in the configuration of the hierarchical information of the present invention shown in FIG.
WH (i) = Xa (i, WH) + Xb (i, WH) + ... + Xz (i, WH)
It becomes.
S7201-4. The total resolution value WH (i) of the hierarchy level i calculated in step S7201-3 is larger than the capability value Na (MWH) set in the capability information Na of the receiving terminal 60a shown in FIG. If it is large (YES), the process proceeds to the next step S7201-5 in order to determine the acceptable hierarchy level value La (MWH) of the receiving terminal 60a.
[0096]
If the total value WH (i) is not greater than Na (MWH) (NO), the process proceeds to step S7201-6 to perform the next layer process.
S7201-5. In step S7201-4, whether the hierarchy level i exceeds the maximum hierarchy level value, or the total resolution value WH (i) of the hierarchy level i is the capability value Na (MWH) of the receiving terminal 60a. Therefore, the value of the hierarchy level “i-1” that is one level lower is regarded as the hierarchy level value La (MWH) that the receiving terminal 60a can accept, and “i-1” is set to La (MWH) The process ends.
[0097]
S7201-6. For the processing of the next layer, the variable i is counted up and the process proceeds to step S7201-2.
FIG. 11 is an operation flowchart of the La (MBR) selection process of the present invention, which shows details of step S7202 of the operation flowchart of FIG. 9 and includes the following processing steps.
[0098]
S7202-1. A variable for temporarily storing a hierarchy level value is represented by i, and “1” is set as an initial value.
S7202-2. It is determined whether or not the value of the variable i is greater than the maximum hierarchical level value of the hierarchical stream sequence held in all the buffers. Here, the maximum hierarchical level value of the hierarchical stream sequence is “2” or more in view of the meaning of the hierarchical stream sequence.
[0099]
If the value of the variable i is larger than the maximum hierarchy level value (YES), the process proceeds to step S7202-5 to determine the allowable hierarchy level value La (MBR) of the receiving terminal 60a, and the variable i If the value is not greater than the maximum hierarchy level value, that is, not more than the maximum hierarchy level value (NO), the process proceeds to the next step S7202-3 and the process is continued.
S7202-3. Based on the hierarchization information received from all transmitting terminals, the total resolution BR (i) of the hierarchy level i is calculated.
[0100]
Using the notation described in the configuration of the hierarchical information of the present invention shown in FIG.
BR (i) = Xa (i, BR) + Xb (i, BR) + ... + Xz (i, BR)
It becomes.
S7202-4. Is the total value BR (i) of the resolution of the hierarchical level i calculated in the above step S7202-3 larger than the value Na (MBR) set in the capability information Na of the receiving terminal 60a shown in FIG. If not (YES), the process proceeds to the next step S7202-5 in order to determine the acceptable hierarchy level value La (MBR) of the receiving terminal 60a.
[0101]
If the total value BR (i) is not greater than Na (MBR) (NO), the process proceeds to step S7202-6 to perform the next layer process.
S7202-5. In step S7202-4 above, whether the hierarchy level i exceeds the maximum hierarchy level value, or the total resolution BR (i) of the hierarchy level i is the capability value Na (MBR) of the receiving terminal 60a. Therefore, the value of the hierarchy level “i-1” that is one level lower is regarded as the hierarchy level value La (MBR) that the receiving terminal 60a can accept, and “i-1” is set to La (MBR), The process ends.
[0102]
S7202-6. For the next level of processing, the variable i is counted up, and the flow proceeds to step S7202-2.
FIG. 12 is an operation flowchart of the La (MFR) selection process of the present invention, showing details of step S7203 of the operation flowchart of FIG. 9, and includes the following processing steps.
[0103]
S7203-1. A variable for temporarily storing a hierarchy level value is represented by i, and “1” is set as an initial value.
S7203-2. It is determined whether or not the value of the variable i is greater than the maximum hierarchical level value of the hierarchical stream sequence held in all buffers. Here, the maximum hierarchical level value of the hierarchical stream sequence is “2” or more in view of the meaning of the hierarchical stream sequence.
[0104]
When the value of the variable i is larger than the maximum hierarchy level value (YES), the process proceeds to step S7203-5 to determine the allowable hierarchy level value La (MFR) of the receiving terminal 60a, and the variable i If the value is not greater than the maximum hierarchy level value, that is, if it is equal to or less than the maximum hierarchy level value (NO), the process proceeds to the next step S7203-3 and the process is continued.
S7203-3. Based on the hierarchization information received from all transmission terminals, the total value FR (i) of the resolution of the hierarchy level i is calculated.
[0105]
Using the notation described in the configuration of the hierarchical information of the present invention shown in FIG.
FR (i) = Xa (i, FR) + Xb (i, FR) + ... + Xz (i, FR)
It becomes.
S7203-4. Is the total value FR (i) of the resolution of the hierarchy level i calculated in step S7203-3 above larger than the value Na (MFR) set in the capability information Na of the receiving terminal 60a shown in FIG. If not (YES), the process proceeds to the next step S7203-5 in order to determine the acceptable hierarchy level value La (MFR) of the receiving terminal 60a.
[0106]
If the total value FR (i) is not greater than Na (MFR) (NO), the process proceeds to step S7203-6 to perform the next layer process.
S7203-5. In step S7203-4, whether or not the hierarchical level i exceeds the maximum hierarchical level value, or the total resolution FR (i) of the hierarchical level i is the capability value Na (MFR) of the receiving terminal 60a. The value of the hierarchy level “i-1” that is one level lower is regarded as the permissible hierarchy level value La (MFR) that can be accepted by the receiving terminal 60a, and “i-1” is set to La (MFR). Then, the process ends.
[0107]
S7203-6. For the next level of processing, the variable i is counted up, and the flow proceeds to step S7203-2.
FIG. 13 is an operation flowchart of the hierarchy selection unit of the present invention and shows an embodiment of the hierarchy selection unit 73a provided corresponding to the receiving terminal 60a.
S7301. From the composite information Ca held in the composite information generation storage unit 72a, an entry provided corresponding to each transmission terminal is extracted.
[0108]
As shown in the composite information configuration of FIG. 5, the extracted entry includes transmission terminal identification information, screen display start position, and output suppression section information.
S7302. It is determined from the output suppression section information in the entry whether or not it is an output suppression section. If it is an output suppression section (YES), the process proceeds to step S7303 to process the next entry. If not (NO), the process proceeds to the next step S7303 in order to select and transmit the hierarchical stream.
[0109]
S7303. The hierarchical stream up to the hierarchy indicated by the allowable hierarchy level La set in the composite information Ca is extracted from the buffer storing the hierarchical stream sequence from the transmission terminal indicated by the transmission terminal identification information set in the entry. To the receiving terminal 60a.
S7304. It is determined whether or not there is an unprocessed entry in the composite information Ca. If there is (YES), the process proceeds to step S7301, and the next transmission terminal is processed. NO), the process is terminated.
[0110]
The processing contents of the above hierarchy selection unit are the same for other receiving terminals other than the receiving terminal 60a.
The above is the description of the first embodiment of the present invention.
FIG. 14 shows the second embodiment of the present invention in the stream distribution method (2) of the present invention.
[0111]
In the first embodiment, the stream data is hierarchically encoded at the transmission terminal, and then the hierarchical stream sequence and the hierarchical information are transmitted to the stream distribution device, whereas the stream distribution method (2) of the present invention shown in FIG. Then, the transmitting terminal transmits stream data that is not hierarchically encoded to the stream distribution apparatus.
One or more transmission terminals and one or more reception terminals are connected to the stream distribution device 70A via the network 40, and the stream data transmitted from the transmission terminal is distributed to the reception terminals via the stream distribution device 70A.
[0112]
In the embodiment of FIG. 14, the configurations of the transmission terminals 50a and 50b and the reception terminals 60a and 60b are shown, but both the transmission terminal and the reception terminal can be configured by one or more arbitrary numbers.
The stream distribution apparatus 70A is provided with a hierarchical information reception storage unit, two buffers, a transcoding unit, and an output suppression section detection storage unit corresponding to each transmission terminal, and receives terminal capability information corresponding to each reception terminal. A storage unit, a distribution selection information reception storage unit, a composite information generation storage unit, and a hierarchy selection unit are provided.
[0113]
In the embodiment shown in FIG. 14, a hierarchical information reception storage unit 71a, a buffer 77a, a buffer 79a, a transcoding unit 78a, and an output suppression section detection storage unit 76a are provided corresponding to the transmission terminal 50a, and correspond to the reception terminal 60a. Thus, a terminal capability information reception storage unit 75a, a distribution selection information reception storage unit 74a, a composite information generation storage unit 72a, and a hierarchy selection unit 73a are provided.
[0114]
Similarly, a hierarchical information reception storage unit 71b, a buffer 77b, a buffer 79b, a transcoding unit 78b, and an output suppression section detection storage unit 76b are provided corresponding to the transmission terminal 50b, and terminal capability information corresponding to the reception terminal 60b. A reception storage unit 75b, a distribution selection information reception storage unit 74b, a composite information generation storage unit 72b, and a hierarchy selection unit 73b are provided.
The same applies when three or more transmitting terminals and receiving terminals are provided.
[0115]
The transmission terminal 50a transmits the stream data Sa that is not hierarchically encoded to the stream distribution device 70A.
Similarly to the transmission terminal 50a, the transmission terminal 50b transmits the stream data Sb that is not hierarchically encoded to the stream distribution device 70A.
After the stream distribution device 70A stores the stream data Sa transmitted from the transmission terminal 50a in the buffer 77a, the transcoding unit 78a once decodes the stream data Sa, and then performs hierarchical encoding according to a predetermined target number of layers. Then, it is stored in the buffer 79a as a hierarchical stream sequence Ua. At the same time, the hierarchized information Xa including the required capability value for each hierarchical level necessary for the receiving terminal to display and output is generated and delivered to the hierarchized information reception storage unit 71a. Further, the output suppression section detection storage unit 76a detects and stores output suppression section information Pa that means a time zone in which display output can be suppressed.
[0116]
The same applies when the stream data Sb transmitted from the transmission terminal 50b is received. After the stream data Sb is stored in the buffer 77b, the transcoding unit 78b once decodes the stream data Sa, and then executes the hierarchical code according to the predetermined target layer number. And stored in the buffer 79b as a hierarchical stream sequence Ub. At the same time, the hierarchized information Xb including the required capability value for each hierarchical level necessary for the receiving terminal to display and output is generated and delivered to the hierarchized information reception storage unit 71b. Further, the output suppression section detection storage unit 76b detects and stores output suppression section information Pb which means a time zone in which display output can be suppressed.
[0117]
Here, the determination of the target number of layers and the required capacity value for each layer level can be realized by the following two methods.
In the first method, among the same type of capability values set as the terminal capability information of the receiving terminal, the total number over all receiving terminals of different capability values is set as the target hierarchy number, and the same type of value is Different capability values are used as required capability values for each hierarchical level.
[0118]
In the second method, an arbitrary number is set as the target layer number, and the highest capability value and the lowest capability value are selected from the capability values of the same type set in the terminal capability information of all receiving terminals. The maximum ability value and the minimum ability value are equally divided by the arbitrarily set target number of hierarchies, and the equally divided ability values are used as the required ability values of the respective hierarchy levels. In the second method, it may be assumed that the target number of hierarchies cannot be equally divided. In that case, the required capacity value is adjusted to a value closest to the equally divided value.
[0119]
The output suppression section detection storage unit 76b detects and stores output suppression section information Pb that means a time zone in which display output can be suppressed.
As the output suppression section, in the case of audio stream data, there is a silent section that means a section that does not contain valid audio information, and in this case, the display output of the receiving terminal does not have to distribute the audio stream data. Does not have a substantial impact. In addition, in the case of video stream data, there is a non-operation section that means a section in which no valid image motion is observed. In this case, even if the video stream data is not distributed, the display output of the receiving terminal is substantially Does not have a significant impact.
[0120]
The terminal capability information reception storage unit 75a receives and stores terminal capability information Na indicating the display output capability of the reception terminal 60a notified in advance from the reception terminal 60a, and the distribution selection information reception storage unit 74a stores in advance the reception terminal 60a. The distribution selection information Za of the receiving terminal 60a notified from is received and stored. The distribution selection information Za is used when the display output capability of the receiving terminal 60a indicated by the terminal capability information Na cannot simultaneously display and output the stream data distributed from each transmitting terminal. Used to select and limit stream data to be distributed to the receiving terminal 60a.
[0121]
The same applies to the terminal capability information reception storage unit 75b and the delivery selection information reception storage unit 74b, and the terminal capability information Na and the delivery selection information Za are previously received from the receiving terminal 60b and stored in advance.
The composite information generation storage unit 72a refers to the hierarchical information of each transmission terminal stored in the hierarchical information reception storage unit and the capability information of the reception terminal 60a stored in the terminal capability information storage unit 75a. Then, a terminal allowable layer level value common to all transmitting terminals, which is acceptable for the receiving terminal 60a to display and output stream data from all transmitting terminals simultaneously, is selected. Then, it generates composite information for each transmitting terminal necessary for the receiving terminal 60a to display and output the stream data distributed from each transmitting terminal, and the selected terminal allowable hierarchy level value and composite for all transmitting terminals A collection of information is set as composite information Ca and stored in the composite information generation storage unit 72a. The composite information for each transmission terminal includes transmission terminal identification information, a screen display start position when stream data is displayed on the screen, and output suppression section information indicating whether display output may be suppressed. ing.
[0122]
When the above-mentioned terminal allowable layer level value cannot be selected, that is, even if it is limited to the lowest layer layer stream, the stream data from all transmitting terminals cannot be displayed and output at the same time with the terminal capability of the receiving terminal 60a. Restricts the transmission terminal that is the target of stream data distribution processing in accordance with the distribution selection information Za held in the distribution selection information reception storage unit 74a, and the above-mentioned composite for the limitedly selected transmission terminal The information generated and collected is held in the composite information generation storage unit 72a as composite information Ca.
[0123]
In the embodiment shown in FIG. 14, the composite information generation storage unit 72a includes the hierarchization information stored in the hierarchization information reception storage units 71a and 71b and the reception terminal 60a stored in the terminal capability information storage unit 75a. With reference to the terminal capability information, the receiving terminal 60a selects a terminal allowable layer level value that is allowed to simultaneously display and output the stream data from the transmitting terminals 50a and 50b, and at the same time, composite information about the transmitting terminals 50a and 50b is obtained. Collect and hold as composite information Ca.
[0124]
Next, for the transmission terminals registered in the composite information Ca, the hierarchy selection unit 73a is a hierarchy from the lowest hierarchy in the hierarchical stream sequence from each transmission terminal to the hierarchy indicated by the terminal allowable hierarchy level value. A stream is selected, and a hierarchical stream sequence Ta composed of the selected hierarchical streams for each transmitting terminal is transmitted to the receiving terminal 60a. At the same time, from the composite information Ca, the receiving terminal 60a extracts the screen display start position, which is information for arranging the stream data from each transmitting terminal on the screen, and sends it to the receiving terminal 60a as terminal composite information Ta To do.
[0125]
The same applies to the composite information generation storage unit 72b and the hierarchy selection unit 73b.The composite information generation storage unit 72b includes the hierarchical information of each transmission terminal stored in the hierarchical information reception storage unit corresponding to each transmission terminal, and Common to all transmitting terminals, which is acceptable for the receiving terminal 60b to display and output stream data from all transmitting terminals simultaneously with reference to the capability information of the receiving terminal 60b stored in the terminal capability information storage unit 75b. Select the terminal allowed hierarchy level value. Then, it generates composite information for each transmitting terminal necessary to display and output the stream data distributed from each transmitting terminal to the receiving terminal 60b, and the selected terminal allowable hierarchy level value and all the transmitting terminals The composite information is collected into composite information Cb, which is held in the composite information generation storage unit 72b.
[0126]
When the above-mentioned terminal allowable layer level value cannot be selected, that is, even if it is limited to the lowest layer layer stream, the stream data from all transmitting terminals cannot be displayed and output at the same time with the terminal capability of the receiving terminal 60b. Restricts the transmission terminal that is the target of the stream data distribution process according to the distribution selection information Zb held in the distribution selection information reception storage unit 74b, and the above-mentioned composite for the limitedly selected transmission terminal The information generated and collected is held in the composite information generation storage unit 72b as composite information Cb. In this case, the terminal allowable hierarchy level value is set to the lowest hierarchy level.
[0127]
In the embodiment shown in FIG. 14, the composite information generation storage unit 72b includes the hierarchization information stored in the hierarchization information reception storage units 71a and 71b and the reception terminal 60b stored in the terminal capability information storage unit 75b. Referring to the terminal capability information, the receiving terminal 60b selects a terminal allowable layer level value that is allowed to simultaneously display and output the stream data from the transmitting terminals 50a and 50b, and generates composite information of the transmitting terminals 50a and 50b. And collect and hold as composite information Cb.
[0128]
Next, for the transmission terminal registered in the composite information Cb, the hierarchy selection unit 73b has a hierarchy from the lowest hierarchy in the hierarchical stream sequence from each transmission terminal to the hierarchy indicated by the terminal allowable hierarchy level value. A stream is selected, and a layer stream sequence Tb composed of the selected layer stream for each transmitting terminal is transmitted to the receiving terminal 60b. At the same time, from the composite information Cb, the receiving terminal 60b extracts the screen display start position, which is information for arranging the stream data from each transmitting terminal on the screen, and sends it to the receiving terminal 60b as terminal composite information Tb. To do.
[0129]
Although the above description has been made assuming that the number of transmitting terminals is 2 and the number of receiving terminals is 2, the same applies to the case where the number is 3 or more.
In the receiving terminal 60a, the terminal capability information notification unit 61a transmits terminal capability information Na indicating the display output capability of the terminal itself to the stream distribution device 70A in advance. Also, the delivery selection information notifying unit 65a selects a transmitting terminal to be displayed and output when stream data from all transmitting terminals cannot be displayed at the same time with the terminal capability of the own terminal even if only the lowest layer stream is received. The distribution selection information Za for this is transmitted to the stream distribution device 70A in advance.
[0130]
The receiving terminal 60a holds the terminal composite information Va transmitted from the stream distribution device 70A in the terminal composite information reception storage unit 62a. Here, the terminal composite information Va includes a pair of identification information of each transmitting terminal and a screen display start position for identifying an area on the screen for displaying and outputting stream data from the transmitting terminal. By searching for information identifying the transmitting terminal included in the stream data as a key, an area on the screen where the stream data should be displayed and output can be specified.
[0131]
On the other hand, the layer stream sequence Ta selected and transmitted by the stream distribution device 70A is decoded by the decoding unit 63a, and then the display output unit 64a stores the terminal composite stored in the terminal composite information reception storage unit 62a. Display output according to information Va.
The same applies to the receiving terminal 60b. The terminal capability information notification 61b transmits the terminal capability information Nb to the stream distribution device 70A in advance, and the distribution selection information notification unit 65b transmits the distribution selection information Zb to the stream distribution device 70A in advance. The terminal composite information Vb transmitted from the stream distribution device 70A is held in the terminal composite information reception storage unit 62b. On the other hand, the layer stream sequence Tb selected and transmitted by the stream distribution device 70A is decoded by the decoding unit 63b, and then the display output unit 64b stores the terminal composite stored in the terminal composite information reception storage unit 62b. Display output according to information Vb.
[0132]
In the embodiment of FIG. 14, the transmitting terminal and the receiving terminal are expressed as separate devices, but the functions of both the transmitting terminal and the receiving terminal can be included in the same terminal as in the case of a multi-ground video conference. In that case, the stream distribution processing target of the stream distribution device 70A does not include stream data from a terminal that is a stream data distribution destination. For example, when the transmission terminal 50a and the reception terminal 60a are configured as the same terminal, the composite information generation storage unit 72a and the hierarchy selection unit 73a perform the above stream stream data from all transmission terminals other than the transmission terminal 50a. Process the delivery.
[0133]
In the embodiment shown in FIG. 14, a plurality of transmission terminals are assumed in the case of a multi-ground video conference. However, the number of transmission terminals can be set to 1 as in the case of stream broadcasting. The function of the device 70A can be integrated into the transmission terminal.
The composite information held in the composite information generation storage unit, the terminal capability information stored in the terminal capability information reception storage unit, and the distribution selection information stored in the distribution selection information reception storage unit have a storage area for each receiving terminal. Information about all receiving terminals may be stored in a lump without providing them.
[0134]
Similarly, the hierarchization information for each transmission terminal stored in the hierarchization information reception storage unit and the output suppression interval information stored in the output suppression interval detection storage unit may be stored collectively for all transmission terminals.
FIG. 15 is a transfer sequence (2) according to the stream distribution method of the present invention, and will be described for each processing step according to the embodiment shown in FIG.
[0135]
S01. The receiving terminal 60a transmits terminal capability information Na and distribution selection information Za to the stream distribution apparatus 70A in advance.
Here, the terminal capability information Na is information indicating the display output capability of the receiving terminal 60a. For example, in the configuration of the terminal capability information of the present invention shown in FIG. 3, the terminal capability information Na includes a maximum resolution, a maximum bit rate, and a maximum frame rate. .
[0136]
When the stream distribution device 70A determines that the distribution selection information Za cannot be displayed and output to the reception terminal 60a simultaneously with the display output capability indicated in the terminal capability information Na, the stream data from all the transmission terminals cannot be displayed at the same time. In addition, it is information used when selecting and limiting stream data to be distributed to the receiving terminal 60a. For example, as in the configuration of the delivery selection information shown in FIG. 8, information on whether or not stream data can be received from each transmission terminal, or information on reception priority indicating the priority order for accepting stream data from each transmission terminal, Can be configured.
[0137]
For the terminal capability information Na and the distribution selection information Za, refer to the configuration of the terminal capability information of the present invention in FIG. 3 and the configuration of the distribution selection information of the present invention in FIG.
S02. The stream distribution device 70A stores the terminal capability information Na and the distribution selection information Za received from the reception terminal 60a in the terminal capability reception storage unit 75a and the distribution selection information reception storage unit 74a, respectively.
[0138]
S03. Similarly to step S01, the receiving terminal 60b transmits the terminal capability information Nb and the distribution selection information Zb to the stream distribution device 70A in advance.
S04. Similarly to step S02, the stream distribution device 70A stores the terminal capability information Na and the distribution selection information Za received from the reception terminal 60a in the terminal capability reception storage unit 75a and the distribution selection information reception storage unit 74a, respectively. To do.
[0139]
In the description of steps S01 to S04 above, the processing of the receiving terminal 60b is described after the processing of the receiving terminal 60a. The processing order of steps S01 to S02 and steps S03 to S04 may be reversed depending on the transmission timing T01 of the receiving terminal, but the processing contents are the same.
S05. The transmission terminal 50a transmits the stream data Sa that is not hierarchically encoded to the distribution device 70A.
[0140]
S06. The stream distribution device 70A stores the received stream data Sa in the buffer 77a, then once decodes it, hierarchically encodes it according to a predetermined target hierarchical level number, and stores it in the buffer 79a as a hierarchical stream sequence Ua. At the same time, the hierarchized information Xa including the display output required capability value necessary for displaying and outputting each hierarchical stream in the hierarchical stream sequence Ua to the receiving terminal is generated and stored. Moreover, the output suppression area information Pa which can suppress the display output to a receiving terminal is detected and memorize | stored.
[0141]
S07. Similarly to step S05, the transmission terminal 50b transmits the stream data Sb that is not hierarchically encoded to the distribution device 70A.
S08. Similar to step S06 above, the stream distribution device 70A stores the received stream data Sb in the buffer 77b, and then once decodes and hierarchically encodes the stream data in accordance with a predetermined target hierarchical level number and buffers it as a hierarchical stream sequence Ub. Store in 79b. At the same time, it generates and stores hierarchical information Xb that includes a display output required capability value necessary for displaying and outputting each hierarchical stream in the hierarchical stream sequence Ub to the receiving terminal. Moreover, the output suppression area information Pb which can suppress the display output to a receiving terminal is detected and memorize | stored.
[0142]
In the description of steps S05 to S08 above, the process of the transmission terminal 50b is described after the process of the transmission terminal 50a. The processing order of steps S05 to S06 and steps S07 to S08 may be reversed depending on the transmission timing T02 of the transmitting terminal, but the processing contents are the same.
S09. The stream distribution device 70A receives the terminal capability information Na and distribution selection information Za of the receiving terminal 60a stored in step S01, the hierarchical information Xa and Xb stored in steps S06 and S08, and output suppression Composite information Ca corresponding to the receiving terminal 60a is generated from the section information Pa and Pb, and stored in the composite information generation storage unit 72a. For details of the composite information Ca, refer to the composition of the composite information of the present invention in FIG. 5, and for details of the composite information generation and storage section 72a, refer to the operation flowchart of the composite information generation and storage section of the present invention in FIG.
[0143]
Here, in the terminal capability of the receiving terminal 60a, even if it is limited to the lowest layer stream, if it is determined that the stream data from all the transmitting terminals cannot be displayed and output at the same time, the distribution selection information Za is not targeted for distribution. The composite information related to the transmitting terminal selected for is not registered in the composite information Ca.
S10. From the composite information Ca generated in step S09, the terminal composite information Va indicating how the receiving terminal 60a displays and outputs the stream data from each transmitting terminal is extracted and transmitted to the receiving terminal 60a.
[0144]
For details of the terminal composite information Va, refer to the configuration of the terminal composite information of the present invention in FIG.
Further, the receiving terminal 60a set in the composite information Ca from the hierarchical stream sequence Ua stored in the buffer 79a for the transmitting terminal 50a and the hierarchical stream sequence Ub stored in the buffer 79b for the transmitting terminal 50b. The hierarchical stream up to the hierarchical level corresponding to the terminal allowable hierarchical level value La is extracted, and the extracted hierarchical stream sequence Ta is transmitted to the receiving terminal 60a.
[0145]
Therefore, since the transmission selection information Za held in the distribution selection information reception storage unit 74a in the process of step S09 is valid and is excluded from the distribution target, it is not registered in the composite information Ca. The hierarchical stream sequence from the transmitting terminal is not distributed to the receiving terminal 60a.
In the embodiment shown in FIG. 15, the transmission terminals 50a and 50b are described on the assumption that they are not excluded from the distribution target, and the hierarchical stream sequence Ua stored in the buffer 79a for the transmission terminal 50a and the transmission terminal 50b The hierarchical stream column Ta is selected from the hierarchical stream column Ub stored in the buffer 79b to the hierarchical level corresponding to the terminal allowable hierarchical level value La of the receiving terminal 60a set in the composite information Ca. And transmitted to the receiving terminal 60a.
[0146]
The above describes the case where the number of transmitting terminals is 2, but the same applies to the case where the number of transmitting terminals is 1 and 3 or more, and the sequence of hierarchical streams below the terminal allowable hierarchical level value La from all transmitting terminals is as follows: It is transmitted as a hierarchical stream sequence Ta to the receiving terminal 60a.
S11. The receiving terminal 60a, after decoding the hierarchical stream sequence Ta transmitted from the stream distribution device 70A, from each transmission terminal according to the information included in the terminal composite information Va similarly notified from the stream distribution device 70A Simultaneously display and output the stream data.
[0147]
The terminal composite information Va includes transmission terminal identification information and a screen display start position for each transmission terminal as shown in the configuration of the terminal composite information of the present invention in FIG. Based on this terminal composite information Va, stream data from each transmitting terminal can be simultaneously displayed and output.
For the display output to the receiving terminal 60a, see the display output example of the receiving terminal of the present invention in FIG.
[0148]
S12. Similarly to step S09, the stream distribution device 70A has the terminal capability information Nb and distribution selection information Zb of the reception terminal 60b stored in step S01, the transmission terminal 50a stored in steps S06 and S08, Composite information Cb corresponding to the receiving terminal 60b is generated from the hierarchized information Xa and Xb of 50b and the output suppression section information Pa and Pb1, and stored in the composite information generation storage unit 72b.
[0149]
S13. From the composite information Cb generated in step S12, the receiving terminal 60b extracts terminal composite information Vb indicating an area for displaying and outputting stream data from each transmitting terminal, and transmits the extracted terminal composite information Vb to the receiving terminal 60b.
On the other hand, from the hierarchical stream sequence Ua stored in the buffer 74a for the transmitting terminal 50a and the hierarchical stream sequence Ub stored in the buffer 74b for the transmitting terminal 50b, the receiving terminal 60b included in the composite information Cb The hierarchical stream up to the hierarchical level corresponding to the terminal allowable hierarchical level value Lb is extracted, and the extracted hierarchical stream sequence Tb is transmitted to the receiving terminal 60b.
[0150]
S14. The receiving terminal 60b decodes the hierarchical stream sequence Tb transmitted from the stream distribution device 70A, and then receives information from each transmission terminal according to the information included in the terminal composite information Vb notified from the stream distribution device 70A. Simultaneously display and output the stream data.
In the description of steps S09 to S14 above, the processing of the receiving terminal 60b is described after the processing of the receiving terminal 60a. The processing order of steps S09 to S11 and steps S12 to S14 may be reversed depending on the transmission timing T03 of the stream distribution device, but the processing contents are the same.
[0151]
S15. Repeat the steps S05 to S14.
As a method of repeating steps S05 to S14, the stream data transmitted from all the transmitting terminals is temporarily stored in the buffer, and the stream distribution apparatus is sufficient to prevent the stream data received from all the transmitting terminals from being lost. This can be realized by performing the processing of steps S06 to S11 in a short cycle.
[0152]
In the above description, the number of transmitting terminals is 2 and the number of receiving terminals is 2. However, the same applies to a number other than 2.
The examples shown in FIGS. 3 to 13 are the same in the second embodiment of the present invention shown in the stream delivery method (2) of the present invention in FIG. 14, and will not be described.
In the detailed description of the present invention described above, the case where stream data is distributed and displayed and output to the receiving terminal 60a has been described with emphasis. However, it may be distributed and displayed and output to other receiving terminals other than the receiving terminal 60a. It is the same.
[0153]
In the above-described embodiment, the number of transmitting terminals and the number of receiving terminals have been described as two. However, the present invention can be similarly applied to any number of one or more transmitting terminals and receiving terminals.
The first implementation of the present invention also applies to a case where a transmission terminal that transmits stream data that is hierarchically encoded and a transmission terminal that transmits stream data that is not hierarchically encoded are mixed and connected to the stream distribution apparatus. It can be easily realized by combining the form and the second embodiment.
[0154]
Although various modifications can be made to the realization method of FIGS. 3 to 13 shown in the embodiment of the present invention, the present invention can be similarly applied.
(Supplementary Note 1) In a stream distribution method for relaying a stream distribution device, distributing stream data from a transmission terminal to a reception terminal, and displaying the stream data,
The receiving terminal notifies the stream distribution device in advance of terminal capability information including one or more types of display output capability values, and the stream distribution device holds the terminal capability information for all receiving terminals,
The transmitting terminal hierarchically encodes the stream data, and decomposes the stream data into a stream of hierarchical streams that are stream data corresponding to a hierarchical level,
Corresponding to one or more types of capability values included in the terminal capability information, a required capability value necessary for display output to the receiving terminal is set for each hierarchical stream, and the hierarchical information is set as hierarchical information. Sending information and the hierarchical stream sequence to the stream delivery device;
The stream distribution device selects a hierarchical stream from the hierarchical stream sequence based on the terminal capability information and the hierarchical information so that the receiving terminal can simultaneously display and output stream data from each transmitting terminal. Terminal stream stream sequence,
Generate terminal composite information designating a display output area of stream data from each transmitting terminal, and transmit the terminal composite information and the terminal layer stream sequence to the receiving terminal,
The receiving terminal displays and outputs the terminal layer stream sequence according to the terminal composite information.
A stream delivery method characterized by the above.
[0155]
(Supplementary Note 2) In a stream distribution method for relaying a stream distribution device, distributing stream data from a transmission terminal to a reception terminal, and displaying the stream data,
The receiving terminal notifies the stream distribution device in advance of terminal capability information including one or more types of display output capability values, and the stream distribution device holds the terminal capability information for all receiving terminals,
The stream distribution apparatus determines, based on the terminal capability information, a target number of layers for hierarchically encoding received stream data and a required capability value necessary for displaying and outputting each layer stream to the receiving terminal. As hierarchical information,
After decoding the received stream data, hierarchical encoding based on the hierarchical information to form a hierarchical stream sequence,
Based on the terminal capability information and the layered information, a terminal layer stream sequence is selected by selecting a layer stream from the layer stream sequence so that the receiving terminal can simultaneously display and output stream data from each transmitting terminal. age,
Generating terminal composite information indicating a display output area in a receiving terminal for displaying and outputting the terminal layer stream sequence, and transmitting the terminal composite information and the terminal layer stream sequence to the receiving terminal;
The receiving terminal displays and outputs the terminal layer stream sequence according to the terminal composite information.
A stream delivery method characterized by the above.
[0156]
(Appendix 3) In the stream delivery method described in Appendix 2,
The target number of layers is the total number of different values for the same type of capability value in the terminal capability information for all receiving terminals,
Different values of the same type of capability value in the terminal capability information for all receiving terminals are set as required capability values of each layer stream.
A stream delivery method characterized by the above.
[0157]
(Appendix 4) In the stream delivery method described in Appendix 2,
The target hierarchy number is an arbitrary number set in advance,
Among the capability values of the same type in the terminal capability information for all receiving terminals, a value obtained by equally dividing the highest value and the lowest value by the target number of layers is a required capability value of each layer stream. To
A stream delivery method characterized by the above.
[0158]
(Supplementary note 5) In the stream delivery method according to any one of supplementary notes 1 to 4,
For the selection of the terminal layer stream sequence, for each type of required capability value included in the layered information, the highest layer level that does not exceed the corresponding type of capability value in the terminal capability information of the receiving terminal is selected. To the permissible hierarchy level by capability,
The lowest hierarchy level is selected from the one or more selected permissible hierarchy levels according to capability to be a terminal allowable hierarchy level,
Selecting a hierarchical stream between the lowest hierarchical level and the terminal allowed hierarchical level;
A stream delivery method characterized by the above.
[0159]
(Supplementary note 6) In the stream delivery method according to any one of supplementary notes 1 to 5,
The receiving terminal notifies the stream distribution device in advance of distribution selection information for selecting stream data to be distributed to the receiving terminal, the stream distribution device holds the previous period distribution selection information for all receiving terminals,
In selecting the terminal allowable hierarchy level, the stream distribution device calculates a value obtained by summing up the required capability values of the corresponding type in the same hierarchy level of the hierarchy information for all transmission terminals. Even if the total value exceeds the corresponding type of capability value in the terminal capability information of the receiving terminal,
The terminal allowable hierarchy level is the lowest hierarchy level, and the terminal hierarchy stream sequence is selected based on the delivery selection information.
A stream delivery method characterized by the above.
[0160]
(Supplementary note 7) In the stream delivery method according to any one of supplementary notes 1 to 6,
The stream distribution device detects an output suppression section indicating a time zone in which display output to the receiving terminal can be suppressed for stream data of a type selected in advance from stream data transmitted from the transmitting terminal,
During the output suppression interval, do not include the type of stream data from the transmission terminal in the terminal layer stream sequence,
A stream delivery method characterized by the above.
[0161]
(Supplementary Note 8) In a stream distribution device that distributes stream data transmitted from a transmission terminal to a reception terminal and outputs it for display,
Means for holding and storing terminal capability information including one or more types of display output capability values notified in advance from the receiving terminal for all receiving terminals;
A layer including a layer stream sequence which is layer-encoded stream data transmitted from the transmission terminal, and a necessary capability value necessary for displaying and outputting the layer stream in the layer stream sequence to the reception terminal Means for receiving activation information;
Based on the terminal capability information and the layered information, a terminal layer stream sequence is selected by selecting a layer stream from the layer stream sequence so that the receiving terminal can simultaneously display and output stream data from each transmitting terminal. And means to
Means for generating terminal composite information indicating a display output area of stream data from each transmitting terminal;
Means for transmitting the terminal composite information and the terminal layer stream sequence to the receiving terminal;
The stream delivery apparatus characterized by the above-mentioned.
[0162]
(Supplementary note 9) In a stream distribution device that distributes stream data transmitted from a transmission terminal to a reception terminal and displays the stream data,
Means for holding and storing terminal capability information including one or more types of display output capability values notified in advance from the receiving terminal for all receiving terminals;
Based on the terminal capability information, a target layer number for hierarchically encoding the received stream data and a required capability value necessary for displaying and outputting each layer stream to the receiving terminal are determined and held as hierarchical information Means to
Based on the terminal capability information and the layered information, a terminal layer stream sequence is selected by selecting a layer stream from the layer stream sequence so that the receiving terminal can simultaneously display and output stream data from each transmitting terminal. And means to
Means for generating terminal composite information indicating a display output area of stream data from each transmitting terminal;
Means for transmitting the terminal composite information and the terminal layer stream sequence to the receiving terminal;
The stream delivery apparatus characterized by the above-mentioned.
[0163]
(Supplementary note 10) In the stream delivery device according to any one of supplementary notes 8 to 9,
Means for receiving and holding in advance distribution selection information for selecting stream data to be distributed to the receiving terminal from the receiving terminal;
In the selection of the terminal allowable layer level, if the receiving terminal cannot simultaneously display and output the stream data from all the transmitting terminals even if it is selected as the lowest layer level, the terminal layer stream sequence is set based on the distribution selection information. With means to choose
The stream delivery apparatus characterized by the above-mentioned.
[0164]
(Supplementary Note 11) In the stream delivery device according to any one of Supplementary Notes 8 to 10,
Means for detecting an output suppression section indicating a time zone in which display output to the receiving terminal can be suppressed for stream data of a preselected type from the stream data transmitted from the transmitting terminal;
Means for selecting the terminal layer stream sequence by excluding the type of stream data from the transmitting terminal during the output suppression period;
The stream delivery apparatus characterized by the above-mentioned.
[0165]
(Supplementary Note 12) In a transmission terminal that distributes stream data to a reception terminal via a stream distribution device and displays the stream data,
Means for hierarchically encoding the stream data and decomposing the stream data into a stream of hierarchical streams that are stream data corresponding to the hierarchical level;
Corresponding to one or more kinds of capability values included in the terminal capability information held by the receiving terminal, hierarchical information is set by setting required capability values necessary for display output to the receiving terminal for each hierarchical stream. And means to
Means for transmitting the layered information and the layer stream sequence to the stream distribution device;
A transmitting terminal characterized by that.
[0166]
(Supplementary Note 13) In a receiving terminal that receives and displays stream data distributed from a transmitting terminal via a stream distributing device,
Means for notifying the stream distribution device in advance of terminal capability information including one or more display output capability values;
Means for receiving and holding terminal composite information including information for specifying a display output area of stream data from each transmission terminal transmitted from the stream distribution device;
Means for decoding and displaying the hierarchically encoded stream data transmitted from the stream distribution apparatus according to the terminal composite information
A receiving terminal.
[0167]
【The invention's effect】
In the distribution of stream data, the present invention selects and distributes a hierarchically encoded stream data transmitted from one or more transmitting terminals by selecting a range of the hierarchical stream in a form suitable for the capability of the receiving terminal. The stream data from the terminal can be displayed and output to the receiving terminal at the same time.
[0168]
As a result, even for multi-ground video conferences that simultaneously display and output stream-encoded stream data from multiple transmitting terminals to each receiving terminal, the load of the stream distribution device is suppressed and the capability of the receiving terminal is met. Efficient stream distribution can be performed.
Further, in the present invention, even when the transmitting terminal transmits stream data that is not hierarchically encoded, the stream distribution apparatus performs hierarchical encoding to an appropriate number of hierarchical levels, and layering suitable for the capabilities of each receiving terminal Configured to distribute the stream data.
[0169]
As a result, stream data from terminals that do not support hierarchical coding can be distributed in a form that conforms to the capabilities of the receiving terminal, and a stream distribution system that can flexibly support various types of terminals can be constructed. Is possible.
[Brief description of the drawings]
FIG. 1 shows a stream delivery method (1) of the present invention.
FIG. 2 shows a transfer sequence (1) according to the stream distribution method of the present invention.
FIG. 3 shows the configuration of terminal capability information according to the present invention.
FIG. 4 shows the structure of hierarchical information according to the present invention.
FIG. 5 shows composition of composite information according to the present invention.
FIG. 6 shows the structure of terminal composite information according to the present invention.
FIG. 7 is a display output example of the receiving terminal of the present invention.
FIG. 8 shows a configuration of distribution selection information according to the present invention.
FIG. 9 is an operation flowchart of the composite information generation storage unit of the present invention.
FIG. 10 is an operation flowchart (1) for selecting an allowable hierarchy level value according to the present invention.
FIG. 11 is an operation flowchart (2) for selecting an allowable hierarchy level value according to the present invention.
FIG. 12 is an operation flowchart (3) for selecting an allowable hierarchy level value according to the present invention.
FIG. 13 is an operation flowchart of the hierarchy selection unit of the present invention.
FIG. 14 is a stream distribution method (2) of the present invention.
FIG. 15 shows a transfer sequence (2) according to the stream delivery method of the present invention.
FIG. 16: Overview of hierarchical coding
FIG. 17 is a conventional stream distribution method.
[Explanation of symbols]
10a, 10b Conventional transmission terminal
11a, 11b Conventional stream transmitter
20a, 20b Conventional receiving terminal
21a, 21b Conventional capability information notification unit
22a, 22b Conventional receiving terminal decoding unit
23a, 23b Conventional display output unit
30 Prior art stream distribution device
31a, 31b Conventional decoding unit
32a, 32b Combining part of conventional technology
33a, 33b Conventional encoder
34a, 34b Prior art buffer
35a, 35b Conventional terminal capability information reception storage unit
40 network
50a, 50b, 50z Sending terminal
51a, 51b Hierarchical stream generation and transmission unit
52a, 52b Layered information notification section
53a, 53b Stream transmitter
60a, 60b, 60z receiving terminal
61a, 61b Terminal capability information notification part
62a, 62b Terminal composite information reception storage unit
63a, 63b Decoding part of receiving terminal
64a, 64b Display output section
65a, 65b Distribution selection information notification section
70,70A stream distribution device
71a, 71b Hierarchical information reception storage unit
72a, 72b Composite information generator
73a, 73b Hierarchy selector
74a, 74b Distribution selection information reception storage unit
75a, 75b Terminal capability information reception storage unit
76a, 76b Output suppression section detection memory
77a, 77b buffer
78a, 78b transcoding section
79a, 79b buffer
Na, Nb terminal capability information
Sa, Sb Non-hierarchical stream data
Ta, Tb terminal hierarchy stream sequence
Ua, Ub hierarchical stream sequence
Va, Vb terminal composite information
Xa, Xb Layered information
Za, Zb delivery selection information

Claims (5)

In a stream distribution method for relaying a stream distribution device, distributing stream data from a transmission terminal to a reception terminal, and displaying the stream data,
The receiving terminal notifies the stream distribution device in advance of terminal capability information including one or more types of display output capability values, and the stream distribution device holds the terminal capability information for all receiving terminals,
The transmitting terminal hierarchically encodes the stream data, and decomposes the stream data into a stream of hierarchical streams that are stream data corresponding to a hierarchical level,
Corresponding to one or more types of capability values included in the terminal capability information, a required capability value necessary for display output to the receiving terminal is set for each hierarchical stream, and the hierarchical information is set as hierarchical information. Sending information and the hierarchical stream sequence to the stream delivery device;
For each type of required capability value included in the layered information, the stream distribution device sets the highest hierarchical level that does not exceed the corresponding type of capability value in the terminal capability information of the receiving terminal. Selected based on the information, and set as the permissible hierarchical level by capability, and the lowest hierarchical level is selected from the one or more selected permissible hierarchical levels by capability as the terminal allowable hierarchical level, and the lowest hierarchical level To a terminal layer stream sequence by selecting a layer stream between the terminal allowable layer level and
Generate terminal composite information designating a display output area of stream data from each transmitting terminal, and transmit the terminal composite information and the terminal layer stream sequence to the receiving terminal,
The receiving terminal displays and outputs the terminal layer stream sequence according to the terminal composite information.
A stream delivery method characterized by the above. In a stream distribution method for relaying a stream distribution device, distributing stream data from a transmission terminal to a reception terminal, and displaying the stream data,
The receiving terminal notifies the stream distribution device in advance of terminal capability information including one or more types of display output capability values, and the stream distribution device holds the terminal capability information for all receiving terminals,
The stream distribution apparatus determines, based on the terminal capability information, a target number of layers for hierarchically encoding received stream data and a required capability value necessary for displaying and outputting each layer stream to the receiving terminal. As hierarchical information,
After decoding the received stream data, hierarchical encoding based on the hierarchical information to form a hierarchical stream sequence ,
For each type of required capability value included in the hierarchical information, the highest hierarchical level that does not exceed the corresponding type of capability value in the terminal capability information of the receiving terminal is selected based on the hierarchical information of all transmitting terminals And select the lowest permitted hierarchical level from among the one or more selected allowable hierarchical levels by capability, and set the terminal allowable hierarchical level from the lowest hierarchical level. Select the hierarchical stream between and the terminal hierarchical stream sequence,
Generating terminal composite information indicating a display output area in a receiving terminal for displaying and outputting the terminal layer stream sequence, and transmitting the terminal composite information and the terminal layer stream sequence to the receiving terminal;
The receiving terminal displays and outputs the terminal layer stream sequence according to the terminal composite information.
A stream delivery method characterized by the above. In streamed system for displaying output to deliver the stream data transmitted from the transmitting terminal to the receiving terminal,
Means for holding and storing terminal capability information including one or more types of display output capability values notified in advance from the receiving terminal for all receiving terminals;
A layer including a layer stream sequence which is layer-encoded stream data transmitted from the transmission terminal, and a necessary capability value necessary for displaying and outputting the layer stream in the layer stream sequence to the reception terminal Means for receiving activation information ;
For each type of required capability value included in the hierarchical information, the highest hierarchical level that does not exceed the corresponding type of capability value in the terminal capability information of the receiving terminal is selected based on the hierarchical information of all transmitting terminals And select the lowest permitted hierarchical level from among the one or more selected allowable hierarchical levels by capability, and set the terminal allowable hierarchical level from the lowest hierarchical level. Means for selecting a layer stream between and a terminal layer stream sequence;
Means for generating terminal composite information indicating a display output area of stream data from each transmitting terminal;
A stream distribution device comprising means for transmitting the terminal composite information and the terminal layer stream sequence to the receiving terminal;
Streamed system characterized by. The stream distribution system according to claim 3, wherein
Means for hierarchically encoding the stream data and decomposing the stream data into a stream of hierarchical streams that are stream data corresponding to the hierarchical level;
Corresponding to one or more kinds of capability values included in the terminal capability information held by the receiving terminal, hierarchical information is set by setting required capability values necessary for display output to the receiving terminal for each hierarchical stream. And means to
Streaming systems that comprising: a transmission terminal that the hierarchical information and the layer stream column equipped with a means for transmitting to the stream delivery apparatus. The stream distribution system according to claim 4, wherein
Means for notifying the stream distribution device in advance of terminal capability information including one or more display output capability values;
Means for receiving and holding terminal composite information including information for specifying a display output area of stream data from each transmission terminal transmitted from the stream distribution device;
The terminal according to the composite information, stream distribution system that further comprising a receiving terminal provided with the means for displaying output by decoding the hierarchically coded stream data transmitted from the stream distribution device.

Images