JP5339290B2 - Video multiplexing apparatus, video multiplexing method, and program - Google Patents

Description

  The present invention relates to a video multiplexing apparatus, a video multiplexing method, and a program for transmitting video having a plurality of synchronized stream configurations within a certain transmission band.

  As an encoding method for compressing a plurality of camera images together, the international standard encoding method H.264 is used. Multi-view video coding (MVC) is standardized as H.264 Annex H (see Non-Patent Document 1, for example). In MVC, a plurality of encoded video data is compressed and encoded in a multiplexed form. Encoded data of frames having the same time stamp are continuously multiplexed. That is, the encoded data is a stream in which a plurality of video data is multiplexed. Since frames with the same time stamp are continuously multiplexed, the decoder can decode and output MVC encoded data, thereby successively decoding and outputting a plurality of camera images with the same time stamp. it can.

  FIG. 26 is a block diagram showing a configuration of a video transmission system according to the prior art. In the figure, the conventional video transmission system includes a multiplexing device 1, a video transmission server 2, and a video reproduction client 3. In the multiplexing apparatus 1, the encoding unit 101 encodes and multiplexes M input streams at the same fixed encoding rate, and generates multiplexed data. The data storage unit 102 stores multiplexed data including the M streams.

  In the video transmission server 2, the data transmission unit 103 extracts N (≦ M) streams from the data storage unit 102 and sends them to the video reproduction client 3. Here, in order to transmit N streams, a transmission rate corresponding to N × fixed coding rate is required.

  In the video reproduction client 3, the data receiving unit 104 receives N streams. The decoding unit 105 decodes N streams. The video display unit 106 displays a partial video.

  Here, a video composed of M streams (whole stream) is referred to as “whole video”, a video composed of N (≦ M) streams (partial streams) is referred to as “partial video”, A video composed of streams is defined as “unit video”. That is, the whole video is composed of M unit videos, and the partial video is composed of N unit videos.

Hideaki Kimata, Shinya Shimizu, Yutaka Kunita, Megumi Isogai, and Yoshimitsu Ohtani, "Panorama video coding for user-driven interactive video application," IEEE International Symposium on Consumer Electronics (ISCE) 2009, 2009.

  In the above-described conventional technology, there is a problem in that a partial video composed of N partial streams cannot be transmitted when transmission rate <N × fixed coding rate. Further, when the encoding rate is lowered too much so that the transmission rate ≧ N × fixed encoding rate, there is a problem that the image quality is deteriorated.

  The present invention has been made in view of such circumstances, and its purpose is video multiplexing capable of transmitting an optimal high-quality video according to the transmission rate without excessively reducing the encoding rate. An apparatus, a video multiplexing method, and a program are provided.

To solve the problems described above, the present invention is a video multiplexer that is encoded and transmitted and multiplexing a plurality of raw stream constituting the video at a predetermined transmission band, each of the plurality of raw streams the encoded by a plurality of coding rates, encoding means for multiplexing the plurality of coded streams generated by the encoding, the predetermined number corresponding to the video display area for displaying a portion of the image of the raw stream selection means for selecting a combination, the evaluation value calculating means for calculating the quality evaluation value for each of said plurality of coded streams generated by the encoding means, said plurality of code generated by the encoding means from the stream, the respective predetermined number of viable streams included in the combination that is selected by coding by the coding rate of the underlying by said selection means Identify made coded stream, the predetermined number of so that Do within transmission bandwidth sum of said predetermined coding rate of the coded stream after the replacement, the from the identified said encoded stream The encoded stream whose quality evaluation value calculated by the evaluation value calculating means is lower than the other specified encoded streams is selected, and the selected encoded stream is selected from the same raw stream as the encoded stream. A combination of the predetermined number of encoded streams is determined by replacing the encoded stream generated by encoding at a higher encoding rate than the basic encoding rate, and the predetermined number of sets of the raw streams a reference information creating means for creating reference information specifying a combination of the predetermined number of coded streams determined from the combined, the Based on the reference information created by irradiation information creating means, selection instructing means for instructing a combination of encoded streams of the predetermined number corresponding to the combination of said predetermined number of raw streams to be selected by said selection means When a video multiplexing device characterized by comprising a transmitting means for transmitting the predetermined number of coded stream combination instructed by the selection instruction means.

The present invention further includes video dividing means for generating a plurality of raw streams by dividing the input video data into small areas in the above invention, wherein the encoding means is divided by the video dividing means. each of the plurality of raw streams, coded in a plurality of coding rates, and wherein the multiplexing the plurality of coded streams generated by the encoding.

In the above invention, the present invention further comprises designation means for designating the combination of the predetermined number of raw streams, and the selection instruction means is instructed to change the combination of the raw streams by the designation means. , based on the reference information created by the reference information creating means, new combinations of encoded streams of the predetermined number corresponding to the combination of said predetermined number of raw streams to be selected by said selection means It is characterized by instructing.

The present invention, in the above invention, the further comprises a specifying means for specifying a combination of a predetermined number of live streams, the reference information creating means, the combination of said predetermined number of raw streams changed by the specifying means When instructed, the from among the plurality of coded streams generated by the encoding means encodes the coding rate to the underlying each of said predetermined number of raw streams included in the combination that is change instruction identify the generated encoded stream Te, the predetermined number of so that Do within transmission bandwidth sum of said predetermined coding rate of the coded stream after the replacement from among the identified said encoded stream The quality evaluation value calculated by the evaluation value calculation means selects an encoded stream that is lower than the other specified encoded streams, and is selected. The encoded stream is replaced with the encoded stream generated by encoding from the same raw stream as the encoded stream at an encoding rate higher than the basic encoding rate. A combination is determined, and reference information specifying a combination of the predetermined number of encoded streams determined from a combination of the predetermined number of raw streams instructed to change is newly created.

Further, in order to solve the above problems, the present invention is a video multiplexing method for transmitting and encoding and multiplexing the plurality of raw stream constituting the video at a predetermined transmission band, said plurality of raw streams Are encoded at a plurality of encoding rates, a plurality of encoded streams generated by the encoding are multiplexed, and a predetermined number of raw streams corresponding to a video display area for displaying a part of the video selecting a combination, a step of calculating the quality evaluation value for each of the generated plurality of coded streams from among the generated plurality of encoded streams, said combination selected wherein each predetermined number of viable streams to identify coded stream generated by coding by the coding rate of the underlying, the predetermined number of the replaced included In so that Do sum of the encoding rate is within the predetermined transmission band of the encoded stream, the encoded lower code than stream the quality evaluation value is other specified from among the identified said encoded stream An encoded stream is selected, and the selected encoded stream is replaced with the encoded stream generated by encoding from the same raw stream as the encoded stream at an encoding rate higher than the basic encoding rate. Determining a combination of the predetermined number of encoded streams, and creating reference information for identifying the combination of the predetermined number of encoded streams determined from the combination of the predetermined number of raw streams ; based on the reference information created in the above, the predetermined number of code corresponding to the combination of said predetermined number of raw stream to be selected A step of instructing a combination of a stream, a video multiplexing method characterized by comprising the step of transmitting a predetermined number of encoded streams of the indicated combination.

According to the present invention, in the above invention, the step of generating a plurality of raw streams by dividing the input video data into small regions, and each of the plurality of divided raw streams at a plurality of encoding rates. Encoding and multiplexing a plurality of encoded streams generated by the encoding .

The present invention, in the above invention, the step of specifying a combination of the predetermined number of raw stream, the predetermined number of live streams combination is change instruction, based on the reference information, said selected And a step of newly instructing a combination of the predetermined number of encoded streams corresponding to the combination of the predetermined number of raw streams to be included.

The present invention, in the above invention, the step of specifying a combination of the predetermined number of raw stream, the predetermined number of the raw stream combination is change instruction, the plurality of generated coded stream from within to identify underlying coded encoded stream generated by encoding by the rate of the respective predetermined number of viable streams included in the alignment change indicated the set, the predetermined number of coded after replacement in so that Do sum of the encoding rate is within the predetermined transmission band of the stream is lower than the coded stream in which the quality evaluation value is other specified from among the identified said coded stream coded stream And the selected encoded stream is higher than the basic encoding rate from the same raw stream as the encoded stream. Goka rate by replacing said encoded stream generated by encoding to determine the combination of the predetermined number of coded streams, determined from a combination of the raw stream of the predetermined number of the change instruction the And a step of newly creating reference information specifying a combination of a predetermined number of encoded streams.

Further, in order to solve the above problems, the present invention causes a computer of a video multiplexer for transmitting encoded and multiplexing a plurality of raw stream constituting the video at a predetermined transmission band, said plurality of raw streams Are encoded at a plurality of encoding rates, a plurality of encoded streams generated by the encoding are multiplexed, and a predetermined number of live images corresponding to a video display area for displaying a part of the video. selection function of selecting a combination of the stream, the encoded evaluation value calculating function of calculating the quality evaluation value for each of said plurality of coded streams generated by the function, the plurality of code generated by the encoding function from the stream, the coding rate to the underlying said respective predetermined number of viable streams included in the combination selected by the selection function Identify the coded stream generated by Goka, the predetermined number of so that Do within transmission bandwidth sum of said predetermined coding rate of the coded stream after the replacement, the identified the coded stream The quality evaluation value calculated by the evaluation value calculation function is selected from among the encoded streams that are lower than the other specified encoded streams, and the selected encoded stream is the same as the encoded stream. A combination of the predetermined number of encoded streams is determined by replacing the encoded stream generated by encoding from a raw stream at a higher encoding rate than the basic encoding rate, and the predetermined number of the raw streams are determined. reference information creating unit that creates reference information specifying a combination of the predetermined number of coded streams determined from a combination of streams , Selection on the basis of the reference information created by reference information creating function, and instructs the combination of the selected functional said predetermined number of which corresponds to the combination of said predetermined number of raw streams to be selected by the encoding stream instruction function, a program, characterized in that to realize a transmission function of transmitting the predetermined number of coded stream combination instructed by the selection instruction function.

  According to the present invention, it is possible to transmit an optimum high-quality video according to the transmission rate without excessively reducing the encoding rate.

It is a block diagram which shows the structure of the video transmission system by 1st Embodiment of this invention. It is a flowchart for demonstrating the evaluation value calculation procedure with respect to each data ID by this 1st Embodiment. It is a flowchart for demonstrating the preparation procedure of the reference information required in the case of command conversion in this 1st Embodiment. It is a conceptual diagram which shows the corresponding information of stream ID and data ID in the 1st embodiment. It is a flowchart for demonstrating the detailed procedure of the reference information preparation in this 1st Embodiment. It is a conceptual diagram which shows the priority data ID information for the image quality improvement in the 1st embodiment. It is a conceptual diagram which shows the example of creation of the stream request information in the first embodiment. It is a conceptual diagram which shows the reference information in this 1st Embodiment. It is a flowchart for demonstrating the command transmission procedure of the video reproduction client 30 in this 1st Embodiment. It is a flowchart for demonstrating the command reception of the video transmission server 20 in this 1st Embodiment, and a data transmission procedure. It is a flowchart for demonstrating the data reception of the video reproduction client 30 in this 1st Embodiment, and a video display procedure. It is a block diagram which shows the structure of the video transmission system by this 2nd Embodiment. It is a conceptual diagram which shows an example of the video division | segmentation by the video division part 301 in this 2nd Embodiment. It is a flowchart for demonstrating the calculation procedure of SNR with respect to each data ID in this 2nd Embodiment. It is a conceptual diagram which shows data ID provided with the bit rate (500 kbps) in the 2nd embodiment. It is a conceptual diagram which shows data ID provided with the bit rate (1 Mbps) in the 2nd embodiment. It is a conceptual diagram which shows data ID provided with the bit rate (1.5 Mbps) in this 2nd Embodiment. It is a conceptual diagram which shows the correspondence information of tile ID in this 2nd Embodiment and video coding data ID. It is a conceptual diagram which shows the example of the display pattern in the 2nd embodiment. It is a conceptual diagram which shows the example of creation of the stream request information in the second embodiment. It is a conceptual diagram which shows the reference information in this 2nd Embodiment. It is a block diagram which shows the structure of the video transmission system by 3rd Embodiment of this invention. It is a block diagram which shows the structure of the video transmission system by 4th Embodiment of this invention. It is a flowchart for demonstrating the reference information preparation procedure in the multiplexing apparatus 10 by 5th Embodiment of this invention. It is a conceptual diagram which shows the example of the ascending order sort of data ID which made the evaluation value the key in the multiplexing apparatus 10 in this 5th Embodiment. It is a block diagram which shows the structure of the video transmission system by a prior art.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
A. First Embodiment FIG. 1 is a block diagram showing a configuration of a video transmission system according to a first embodiment of the present invention. In the multiplexing apparatus 10, the encoding unit 201 receives video having synchronized M stream configurations, encodes and multiplexes each stream at a plurality of encoding rates, and generates video data. The data storage unit 202 stores this video data. The reference information creation unit 203 creates reference information using the video data stored in the data storage unit 202 and stores the reference information in the reference information storage unit 204. Note that data multiplexed in the order of reference information and video data may be stored in the data storage unit 202.

  In the video transmission server 20, the data management unit 205 makes N (≦ M) stream requests to the data storage unit 202 in accordance with the “stream request information” received from the command reception unit 206 from the video reproduction client 30. Thus, N streams are extracted from the M streams stored in the data storage unit 202 and transferred to the data transmission unit 207. The data transmission unit 207 transmits N streams to the video reproduction client 30. The command receiving unit 206 receives a command from the video playback client 30. The reference information transmission unit 208 takes out the reference information data from the reference information storage unit 204 and sends it to the video reproduction client 30.

  In the video reproduction client 30, the data reception unit 212 receives N streams. The decoding unit 213 decodes N streams. The video display unit 214 displays the decoded partial video. The reference information receiving unit 216 receives the reference information transmitted from the video transmission server 20 and stores it in the reference information storage unit 210. The stream combination designation unit 215 accepts a change input for stream combination designation. When the stream combination designation is changed, the command conversion unit 211 refers to the reference information stored in the reference information storage unit 210 and converts “stream combination information” into “stream request information”. The command transmission unit 209 transmits a command to the video transmission server 20 in accordance with “stream request information”.

  It should be noted that the data storage unit 202 and the reference information storage unit 204 in the multiplexing device 10 are functionally integrated, the data transmission unit 207 and the reference information transmission unit 208 in the video transmission server 20 are functionally integrated, and the data in the video playback client 30 is integrated. A configuration in which the reception unit 212 and the reference information reception unit 216 are functionally integrated is also possible.

  In FIG. 1, in the video playback client 30, the command conversion unit 209 converts “stream combination information” into “stream request information” with reference to “reference information”. By having “reference information” with the video playback client 30, it is also possible to implement a configuration in which the command transmission unit 209 directly transmits “stream combination information” as a command without performing command conversion in the command conversion unit 211. is there. In the case of this configuration, “stream combination information” can be converted to “stream request information” by referring to the reference information on the video transmission server 20 side.

Next, the video multiplexing / transmission method of the first embodiment will be described in detail.
In the first embodiment, for example, when video composed of 12 synchronized input streams is transmitted at 10 Mbps, reference information is transmitted before video data is transmitted. The reference information creation procedure in the reference information creation unit 203 in the multiplexing apparatus 10 shown in FIG. 1 uses an evaluation value calculation procedure for a result obtained by encoding each input stream at a plurality of bit rates, and an evaluation value calculation result. The reference information creation procedure.

  FIG. 2 is a flowchart for explaining an evaluation value calculation procedure for each data ID by the encoding unit 201 and the reference information creation unit 203 according to the first embodiment. First, the encoding unit 201 assigns stream IDs to a plurality of synchronized input streams. Next, each input stream is encoded at a plurality of encoding rates (bit rates) (step S001), and a data ID is assigned to the encoded stream (step S002). Subsequently, the reference information creation unit 203 creates correspondence information between the stream ID and the video encoded data ID (step S003), and finally calculates an evaluation value for each data ID (step S004).

  FIG. 3 is a flowchart for explaining a reference information creation procedure by the reference information creation unit 203 necessary for command conversion by the command conversion unit 211 according to the first embodiment. Before creating reference information, the reference information creating unit 203 first sets a “transmission rate” indicating a transmission band and a “stream combination” composed of a combination of data IDs (step S201). The “base bit rate” indicating the encoding rate to be set and the “improved bit rate” indicating the encoding rate larger than the basic encoding rate are set (step S202).

  Next, the number of data IDs targeted for image quality improvement is calculated from the data IDs at the base bit rate in the “stream combination” (step S203). Next, using the evaluation value for each data ID calculated in step S004, order information of the data ID that needs to be improved in image quality preferentially is created (step S204). The number of data IDs calculated in step S203 is changed from the data ID at the base bit rate in the “stream combination” to the data ID at the improved bit rate (step S205). Finally, reference information including correspondence information between the “stream combination information” and the “stream request information” composed of the data ID is created (step S206).

  For example, transmission rate: 10 Mbps, base bit rate: 500 kbps, improved bit rate: 1 Mbps, “stream combination” is stream ID (7, 8, 9, 13, 14, 15, 19, 20, 21, 25, 26, 27 ), When the evaluation value is an average SNR (average value of SNR (signal-noise ratio) of all frames), correspondence information between the stream ID and the data ID is as shown in FIG.

  Further, the SNR of each frame is calculated by the following equation (1).

SNR = 10 * log ₁₀ (MAX ² / MSE) (1)

  The unit is decibel [dB]. MAX is the maximum pixel value that the original image can take. For example, the value of MAX when the bit depth is 8 bits is 255. The MSE in Equation (1) is a mean square error, and the square sum of the difference between the tile-size original image and the local decoded image is divided by the number of pixels of the tile size. Desired.

  In the case of an encoding mode (for example, full intra mode) in which a local decoded image cannot be obtained by the encoding unit 201, local decoded image creation (decoding) needs to be performed separately from the encoding procedure. .

  FIG. 5 is a flowchart for explaining detailed procedures for creating reference information in the multiplexing apparatus 10 according to the first embodiment. The reference information creation unit 203 first sets the “stream combination” (stream ID (7, 8, 9, 13, 14, 15, 19, 20, 21, 25, 26, 27), Nt = 12), transmission The bit rate (Max_bitrate) is set to 10 Mbps (step S301). Next, the base bit rate (tile_bitrate_base) = 500 kbps and the improved bit rate (tile_bitrate_up) = 1 Mbps are set (step S302).

  Next, Nt (number of stream IDs of “stream combination”) × tile_bitrate_base (base bit rate) <Max_bitrate (transmission bit rate) (first condition), and M1 (Max_bitrate−Nt × tile_bitrate_base)> M2 It is determined whether (tile_bitrate_up-tile_bitrate_base) or a second condition of M1 = M2 is satisfied (step S303).

  If the first condition or the second condition is not satisfied, tile_bitrate_base = tile_bitrate_base-Δtile_bitrate_base and tile_bitrate_up = tile_bitrate_up-Δtile_bitrate_up are set (step S304), and the process returns to step S303.

  On the other hand, in the setting condition described above, Nt = 12 based on the number of stream IDs of the “stream combination”, so Nt × title_bitrate_base = 12 × 0.5 Mbps = 6 Mbps <Max_bitrate = 10 Mbps, which satisfies the first condition, Since tile_bitrate_up = 1 Mbps, M1 = 10−12 × 0.5 = 4 Mbps, M2 = 1−0.5 = 0.5 Mbps, and the second condition of M1> M2 is satisfied (YES in step S303).

  When the first condition and the second condition are satisfied, Mup = INT (4 / 0.5) = 8 is calculated (step S305), and the data IDs at the 12 base bit rates in the “stream combination” are calculated. Sorting in ascending order by using the SNR of the Y component for the key as a key (step S306). Thereby, the sort result shown in FIG. 6 is obtained.

  Next, eight are selected from Mup = 8 in ascending order of SNR (step S307), the correspondence information of “data ID” shown in FIG. 4 is referred to, and the data ID at the base bit rate tile_bitrate_base is as shown in FIG. In addition, the data ID is changed to the improved bit rate tile_bitrate_up (step S308). Finally, as shown in FIG. 8, reference information including correspondence information between “stream combination” and “stream request information” composed of data IDs is created (step S309).

  Next, a series of procedures from command transmission from the video playback client 30 to video display in the video transmission system according to the first embodiment will be described with reference to FIGS.

  FIG. 9 is a flowchart for explaining the command transmission procedure of the video reproduction client 30. First, the “stream combination” is designated by the stream combination designation unit 215 (step S501), the “stream combination” is converted to “stream request information” by the command conversion unit 211 (step S502), The command transmission unit 209 transmits a command (step S503).

  FIG. 10 is a flowchart for explaining command reception and data transmission procedures of the video transmission server 20. First, the command reception unit 206 receives a command (step S401), interprets the request data ID included in the command (step S402), and sends a stream corresponding to the request data ID from the data storage unit 202 to the data transmission unit. Transfer to 207 (step S403). Finally, the data transmission unit 207 transmits the stream (step S404).

  FIG. 11 is a flowchart for explaining the data reception and video display procedure of the video playback client 30. First, the data receiving unit 212 receives a stream (step S601), and then the decoding unit 213 decodes the stream (step S602). Finally, the decoded data is displayed on the video display unit 214 (step S603).

B. Second Embodiment Next, a second embodiment of the present invention will be described.
FIG. 12 is a block diagram showing the configuration of the video transmission system according to the second embodiment. It should be noted that portions corresponding to those in FIG. In the multiplexing apparatus 10, the video dividing unit 301 divides input video data into M streams (tile division). The encoding unit 201 encodes and multiplexes each stream at a plurality of encoding rates to create video data.

  As described above, in the second embodiment, the video dividing unit 301 divides input video data into a plurality of synchronized streams. For example, when the input video size is 3840 × 1920 pixels and the tile size is 640 × 320 pixels, the division result shown in FIG. 13 is obtained. At this time, if the video display area size is 1920 × 1080 pixels, the video display pattern is 12 patterns.

  FIG. 14 is a flowchart for explaining the SNR calculation procedure for each data ID. First, the input video of 3840 × 1920 pixels is divided into 6 × 6 = 36 tiles as shown in FIG. 13 (step S101). Next, each tile is encoded at a plurality of bit rates (for example, 500 kbps, 1 Mbps, and 1.5 Mbps) (step S102). Subsequently, a data ID is assigned to the encoding result of each tile (step S103). For example, FIGS. 15, 16, and 17 show data IDs assigned at a plurality of stages of bit rates (500 kbps, 1 Mbps, and 1.5 Mbps).

  Next, correspondence information between the tile ID and the encoded video data ID as shown in FIG. 18 is created (step S104). Finally, an average SNR for each data ID (the SNR of all frames) from the tile-size original image obtained after tile division of the input image and the tile-size local decoded image obtained in the process of the encoding unit 201. (Average value) is calculated (step S105).

  Here, the SNR of each frame is calculated by Expression (1) as in the first embodiment. In the case of an encoding mode (for example, full intra mode) in which a local decoded image cannot be obtained by the encoding unit 201, local decoded image creation (decoding) needs to be performed separately from the encoding procedure. is there.

  In the second embodiment, the reference information creation procedure required for command conversion in the command conversion unit 211 is the same as that in the first embodiment (FIG. 3), and therefore only different points will be described. To do.

  For example, as shown in FIG. 13, an example in which the tile size is 640 × 320 pixels, the input video size is 3840 × 1920 pixels, and the video display area size is 1920 × 1080 pixels will be described. In the case as shown in FIG. 13, when it is assumed that the video display area is not enlarged / reduced and only parallel movement is performed, the total display pattern is 12 patterns of display patterns P1 to P12 as shown in FIG. . The stream to which each tile ID is assigned as an input image, the result of encoding at 500 kbps shown in FIG. 15, the result of encoding at 1 Mbps shown in FIG. 16, and the result of encoding at 1.5 Mbps shown in FIG. When the data ID is assigned to the converted result, the correspondence relationship between the tile ID and the data ID can be represented in FIG.

  Here, in the second embodiment, a case where the transmission rate is 10 Mbps, the base bit rate is 500 kbps, the improved bit rate is 1 Mbps, or 1.5 Mbps, and the evaluation value is an average SNR (average value of SNR of all frames) will be described. The reference information creation procedure in the multiplexing apparatus 10 according to the second embodiment is basically the same as that in the first embodiment (FIG. 5), and will be described with reference to FIG.

  In step S301, the “display pattern P6” shown in FIG. 19 is set as the “stream combination”. Since the transmission rate is 10 Mbps, Max_bitrate = 10 Mbps is set. Next, the case where the base bit rate title_bitrate_base = 500 kbps and the improved bit rate tile_bitrate_up = 1 Mbps are set in step S302 will be described.

  Since the number of tiles in the “display pattern P6” shown in FIG. 19 is 12 tiles, Nt = 12. Therefore, in step S303, Nt × title_bitrate_base = 12 × 0.5 Mbps = 6 MbpsMax_bitrate = 10 Mbps, which satisfies the condition, and further, when tile_bitrate_up = 1 Mbps, M1 = 10−12 × 0.5 = 4 Mbps, M2 = 1-0 .5 = 0.5 Mbps, which satisfies the condition of M1> M2.

  If the above condition is satisfied, Mup = INT (4 / 0.5) = 8 can be calculated in step S305. Next, the data ID at the base bit rate corresponding to the 12 tile IDs in the “display pattern P6” is selected from the correspondence information between the tile ID and the video encoded data ID shown in FIG. 18, and in step S306, Sorting in ascending order using the average SNR of the Y component relating to the data ID as a key. Thereby, the sort result shown in FIG. 6 is obtained.

  Next, from Mup = 8 calculated in step S305, eight are selected in ascending order of SNR in step S307, and in step S308, the correspondence information of “data ID” shown in FIG. 18 is referred to, and the base bit rate tile_bitrate_base is selected. Is changed to the data ID at the improved bit rate tile_bitrate_up. Finally, in step S309, as shown in FIG. 8, reference information including correspondence information of “stream request information” composed of “stream combination” and data ID is created.

  On the other hand, when tile_bitrate_up = 1.5 Mbps is set, in step S303, M1 = 10−12 × 0.5 = 4 Mbps, M2 = 1.5−0.5 = 1.0 Mbps, and the condition of M1> M2 Therefore, in step S305, Mup = INT (4 / 1.0) = 4 can be calculated.

  Next, in step S306, sorting is performed in ascending order using the SNRs of the Y components related to the data IDs corresponding to the number of twelve tiles in the display pattern P6 as keys. Thereby, the sort result shown in FIG. 6 is obtained. Next, from Mup = 4 calculated in S305, four are selected in ascending order of SNR in step S307, and in step S308, the correspondence information of “data ID” shown in FIG. 18 is referred to, and the base bit rate tile_bitrate_base is set. As shown in FIG. 20, the data ID is changed to the data ID at the improved bit rate tile_bitrate_up. Finally, in step S309, as shown in FIG. 21, reference information including correspondence information between “stream combination” and “stream request information” composed of the data ID is created.

  In the second embodiment, a series of procedures from command transmission to video display in the video playback client is the same as that in the first embodiment described above (FIGS. 9 to 11), and a description thereof will be omitted.

C. Third Embodiment Next, a third embodiment of the present invention will be described.
FIG. 22 is a block diagram showing the configuration of the video transmission system according to the third embodiment. It should be noted that portions corresponding to those in FIG. In the multiplexing apparatus 10, the evaluation value calculation unit 403 calculates an evaluation value using the video data stored in the data storage unit 402 and stores the evaluation value in the calculation value storage unit 404. Note that data multiplexed in the order of evaluation values and video data may be stored in the data storage unit 402.

  In the video transmission server 20, the evaluation value transmission unit 408 extracts the evaluation value information from the evaluation value storage unit 404 and sends it to the video reproduction client.

  In the video reproduction client 30, the evaluation value receiving unit 415 receives the evaluation value information transmitted from the video transmission server 20 and stores it in the evaluation value storage unit 416. When the stream combination designation is changed, the reference information creation unit 418 extracts the evaluation value information from the evaluation value storage unit 416, creates the reference information, and stores the reference information in the reference information storage unit 210.

  The data storage unit 202 and the evaluation value storage unit 404 in the multiplexing apparatus 10 are functionally integrated, the data transmission unit 207 and the evaluation value transmission unit 408 in the video transmission server 20 are functionally integrated, and the data in the video playback client 30 is integrated. A configuration in which the receiving unit 212 and the evaluation value receiving unit 415 are functionally integrated is also possible.

  The third embodiment is a modification of the above-described first embodiment. As can be seen by comparing the configuration of FIG. 1 with the configuration of FIG. 22, the multiplexing apparatus 10 does not create reference information. Reference information is created by the video playback client 30.

  Therefore, the information transmitted to the video transmission client 30 is not the information shown in FIG. 8 and FIG. 21, but the correspondence information between the stream ID and the video encoded data ID shown in FIG. 4 and the sort shown in FIG. This is the result (here, the average SNR calculation result for all data IDs). The creation of the reference information in the video playback client 30 is the same as in the first embodiment described above. That is, the setting of “stream combination” in step S301 in FIG. 5 is performed by the stream combination designation unit 215 shown in FIG. 22, and the reference information according to the procedure shown in FIG. 5 according to the change in designation of “stream combination”. Create.

D. Fourth Embodiment Next, a fourth embodiment of the present invention will be described.
FIG. 23 is a block diagram showing the configuration of the video transmission system according to the fourth embodiment. In addition, the same code | symbol is attached | subjected to the part corresponding to FIG.1, FIG12, FIG.22, and description is abbreviate | omitted. As is apparent from FIG. 23, the video transmission system according to the fourth embodiment is different from the video transmission system according to the first embodiment shown in FIG. 1 in the configuration of the second embodiment shown in FIG. The configuration of the third embodiment shown in FIG.

  That is, in the video transmission system according to the fourth embodiment, in the multiplexing device 10, the video dividing unit 301 divides input video data into M streams (tile division), and the evaluation value calculating unit 403 An evaluation value is calculated using the video data stored in the data storage unit 503 and stored in the calculated value storage unit 404. In the video transmission server 20, the evaluation value transmission unit 408 extracts the evaluation value information from the evaluation value storage unit 404 and sends it to the video reproduction client 30. In the video reproduction client 30, the evaluation value receiving unit 415 receives the evaluation value information transmitted from the video transmission server 20, stores the evaluation value information in the evaluation value storage unit 416, and the reference information creation unit 418 receives the stream combination. When the designation is changed, the evaluation value information is extracted from the evaluation value storage unit 416, the reference information is created, and the reference information is stored in the reference information storage unit 210.

  The fourth embodiment is a modification of the above-described first embodiment. As can be seen by comparing the configuration of FIG. 12 with the configuration of FIG. 23, the multiplexing apparatus 10 does not create reference information. Reference information is created by the video playback client 30.

  Therefore, the information transmitted to the video transmission client 30 is not the information shown in FIGS. 8 and 21, but the information shown in FIGS. 18 and 6 (here, the average SNR calculation result for all data IDs). . The creation of the reference information in the video playback client 30 is the same as in the second embodiment described above, and the “stream combination” in step S301 in FIG. 5 is set in the stream combination designation unit 215 in FIG. Reference information may be created according to the procedure shown in FIG. 5 in accordance with the change in designation of “stream combination”.

E. Fifth Embodiment Next, a fifth embodiment of the present invention will be described.
The configuration of the video transmission system according to the fifth embodiment is the same as the configuration of the video transmission system according to the second embodiment shown in FIG. In the fifth embodiment, the transmission rate is 11 Mbps, the base bit rate is 500 kbps, the improved bit rate is 1 Mbps, and 1.5 Mbps (two-stage improvement), and the evaluation value is an average SNR (average value of SNR of all frames). The reference information creation procedure in the case of having been performed will be described.

  FIG. 24 is a flowchart for explaining a reference information creation procedure in the multiplexing apparatus 10 according to the fifth embodiment. First, a “transmission rate” indicating a transmission band is set to 11 Mbps, and a “stream combination” configured by a combination of data IDs (stream ID = (7, 8, 9, 13, 14, 15, 19, 20) is set. , 21, 25, 26, 27)) (step S 701), and then “base bit rate” = 500 kbps indicating the basic encoding rate, indicating an encoding rate higher than the basic encoding rate “Improved bit rate” = 1 Mbps and 1.5 Mbps are set (step S702).

  Next, as shown in FIG. 25, the data ID is sorted in ascending order using the evaluation value (average SNR indicated by SNR_Y500) as a key (step S703), and the data ID given to the encoded data by the base bit rate = Of (107, 108, 109, 113, 114, 115, 119, 120, 121, 125, 126, 127), the data ID = 121 indicating the lowest SNR was encoded at 1 Mbps on one stage. The data ID assigned to the data is changed to 221 (step S704). After the change, data ID = (107, 108, 109, 113, 114, 115, 119, 120, 221, 125, 126, 127).

  As a result, the total bit rate is 11 × 0.5 Mbps + 1 × 1 Mbps = 6.5 Mbps <transmission rate: 11 Mbps (NO in step S705), and the process returns to S703. When the data ID = 221 is further changed after the sorting and the data ID change are performed a plurality of times, the data ID is changed to the data ID = 321 given to the data encoded at 1.5 Mbps which is one step higher. .

  If YES in step S705, it indicates that the total bit rate exceeds the transmission rate, so the previous data ID change is restored (step S706). In this case, the data ID change is performed 10 times, and the result shown in FIG. 25 can be obtained. Here, SNR_Y indicates an average SNR corresponding to data corresponding to the changed data ID. Next, reference information is created based on the data IDs indicated by hatching in FIG. 25 (step S707).

  FIG. 25 shows the data ID before change and the average SNR for the data ID associated with FIG.

  According to the first to fifth embodiments described above, N images suitable for the transmission rate from video data encoded and multiplexed at a plurality of bit rates with respect to a video having a plurality of synchronized stream configurations. Thus, even if transmission rate <N × fixed coding rate, partial video composed of N streams can be transmitted, and transmission rate ≧ N × fixed code. Therefore, it is possible to transmit an optimal high-quality video according to the transmission rate without reducing the encoding rate too much so that the encoding rate is achieved.

DESCRIPTION OF SYMBOLS 10 Multiplexer 20 Video transmission server 30 Video reproduction client 201 Encoding part 202 Data storage part 203 Reference information creation part 204 Reference information storage part 205 Data management part 206 Command reception part 207 Data transmission part 208 Reference information transmission / reception part 209 Command transmission Unit 210 reference information storage unit 211 command conversion unit 212 data reception unit 213 decoding unit 214 video display unit 215 stream combination designation unit 216 reference information reception unit

Claims (9)

A video multiplexing device that encodes and multiplexes a plurality of raw streams constituting a video and transmits in a predetermined transmission band,
Each of the plurality of raw streams encoded with multiple coding rates, encoding means for multiplexing the plurality of coded streams generated by the coding,
Selection means for selecting the combination of the raw stream of predetermined number corresponding to the video display area for displaying a portion of the image,
An evaluation value calculation means for calculating the quality evaluation value for each of said plurality of coded streams generated by said encoding means,
Generated by encoding each of the predetermined number of raw streams included in the combination selected by the selection unit at a basic encoding rate from the plurality of encoded streams generated by the encoding unit. identify the encoded stream, the predetermined number of so that Do within transmission bandwidth sum of said predetermined coding rate of the coded stream after the replacement, the evaluation from among the identified said encoded stream Selecting an encoded stream whose quality evaluation value calculated by the value calculating means is lower than that of the other specified encoded stream, and selecting the selected encoded stream from the same raw stream as the encoded stream; Replace with the encoded stream generated by encoding at a higher encoding rate than the basic encoding rate. Determining a combination of a predetermined number of coded streams, and reference information creating means for creating reference information specifying a combination of the predetermined number of the predetermined number determined by the combination of the raw stream of the coded stream ,
A selection instruction for instructing a combination of the predetermined number of encoded streams corresponding to the combination of the predetermined number of raw streams to be selected by the selection unit based on the reference information generated by the reference information generation unit Means,
Video multiplexing device characterized by comprising a transmitting means for transmitting the predetermined number of coded stream combination instructed by the selection instruction means. Video division means for generating a plurality of raw streams by dividing the input video data into small regions,
The encoding means includes
Each of the divided plurality of raw stream by the image dividing means encodes a plurality of coding rates, according to claim 1, wherein the multiplexing the plurality of coded streams generated by the coding Video multiplexer. Further comprising specifying means for specifying a combination of the predetermined number of raw streams;
The selection instruction means includes
When the combination of the raw stream is changed indicated by the specifying means, on the basis of the reference information created by the reference information creating means, the combination of said predetermined number of raw streams to be selected by said selection means The video multiplexing apparatus according to claim 1 or 2, wherein a new instruction is given for a combination of the predetermined number of encoded streams corresponding to . Further comprising specifying means for specifying a combination of the predetermined number of raw streams;
The reference information creating means includes
When the combination of the predetermined number of raw stream is changed indicated by the designation unit, wherein the predetermined from the plurality of coded streams generated by said coding means, included in the combination that is change instruction A coded stream generated by encoding each of a number of raw streams at a basic coding rate is specified, and the sum of coding rates of the predetermined number of encoded streams after replacement is within the predetermined transmission band. in such so that, to select a lower encoded stream than the encoded stream the quality evaluation value calculated is another identified by the evaluation value calculating means among the identified said encoded stream, are selected The encoded stream is changed from the same raw stream as the encoded stream to a higher encoding rate than the basic encoding rate. Ri is replaced with the coded stream generated by coding the predetermined number to determine the combination of the coded stream, the change instruction has been said predetermined number of determined from a combination of said predetermined number of said raw stream The video multiplexing apparatus according to claim 1 or 2, wherein reference information for specifying a combination of encoded streams is newly created. A video multiplexing method for encoding and multiplexing a plurality of raw streams constituting a video and transmitting in a predetermined transmission band,
A step of multiplexing each of the plurality of raw streams, coded in a plurality of coding rates, a plurality of coded streams generated by the coding,
Selecting a combination of the raw stream of predetermined number corresponding to the video display area for displaying a portion of the image,
Calculating the quality evaluation value for each of the generated plurality of coded streams,
From among the plurality of generated encoded streams, specify an encoded stream generated by encoding each of the predetermined number of raw streams included in the selected combination at a basic encoding rate, said predetermined number of so that Do within transmission bandwidth sum of said predetermined coding rate of the coded stream after the replacement, the said quality evaluation value is other specified from among the identified said encoded stream Generated by selecting an encoded stream lower than the encoded stream and encoding the selected encoded stream from the same raw stream as the encoded stream at an encoding rate higher than the basic encoding rate. A combination of the predetermined number of encoded streams is determined to replace the encoded stream, and the predetermined number of raw streams is determined. And creating reference information specifying a combination of the coded stream a predetermined number determined from combinations above,
Instructing a combination of the predetermined number of encoded streams corresponding to a combination of the predetermined number of raw streams to be selected based on the created reference information;
Video multiplexing method characterized by including the step of transmitting the indicated combinations of the predetermined number of coded streams. Generating a plurality of raw streams by dividing input video data into small areas;
6. The method further comprising: encoding each of the plurality of divided raw streams at a plurality of encoding rates, and multiplexing the plurality of encoded streams generated by encoding. Video multiplexing method. Designating a combination of the predetermined number of raw streams;
When the combination of the predetermined number of raw streams is instructed to change, based on the reference information, the combination of the predetermined number of encoded streams corresponding to the combination of the predetermined number of raw streams to be selected The video multiplexing method according to claim 5 or 6, further comprising the step of: Designating a combination of the predetermined number of raw streams;
When the combination of the predetermined number of raw stream is change instruction, from among the generated plurality of coded streams, underlying each said predetermined number of raw streams included in the combination that is change instruction identify the coded stream generated by coding by the coding rate, the predetermined number of so that Do within transmission bandwidth sum of said predetermined coding rate of the coded stream after replacement, identified the An encoded stream having a quality evaluation value lower than the other specified encoded streams is selected from the encoded streams, and the selected encoded stream is selected from the same raw stream as the encoded stream from the basic stream. The predetermined number of codes replaced with the encoded stream generated by encoding at a higher encoding rate than the encoding rate Determining a combination of streams, further comprising the steps of newly created reference information specifying a combination of changes indicated said predetermined number of coded streams determined from a combination of said predetermined number of said raw stream The video multiplexing method according to claim 5 or 6, further comprising: To a computer of a video multiplexing device that encodes and multiplexes a plurality of raw streams constituting a video and transmits them in a predetermined transmission band,
Each of the plurality of raw streams, coded in a plurality of coding rates, encoding function for multiplexing a plurality of coded streams generated by the coding,
Selection function of selecting a combination of the raw stream of predetermined number corresponding to the video display area for displaying a portion of the image,
Evaluation value calculating function of calculating the quality evaluation value for each of the encoding function of the plurality of generated by the encoding stream,
Generated by encoding each of the predetermined number of raw streams included in the combination selected by the selection function at a basic encoding rate from the plurality of encoded streams generated by the encoding function. identify the encoded stream, the predetermined number of so that Do within transmission bandwidth sum of said predetermined coding rate of the coded stream after the replacement, the evaluation from among the identified said encoded stream The encoded stream having the quality evaluation value calculated by the value calculation function is lower than the other specified encoded streams, and the selected encoded stream is selected from the same raw stream as the encoded stream. Replace with the encoded stream generated by encoding at a higher encoding rate than the basic encoding rate. Determining a combination of a predetermined number of encoded stream, reference information creating function of creating reference information specifying a combination of the predetermined number of the predetermined number determined by the combination of the raw stream of coded streams,
A selection instruction for instructing a combination of the predetermined number of encoded streams corresponding to a combination of the predetermined number of raw streams to be selected by the selection function based on the reference information generated by the reference information generation function function,
A program characterized by implementing the transmission function of transmitting the predetermined number of coded stream combination instructed by the selection instruction function.

Images