JP2002171529A - Video encoder and method, recording medium, and decoder - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video encoder and method that attains multi-angle broadcasting for a frequency band by one channel in digital TV broadcasting. SOLUTION: A bit allocator 41 allocates bits on the basis of multi-angle information 12 and outputs bit allocation information 43. A 1st encoder 201 and a 2nd encoder 202 respectively encode a main video image 101 and audio 102 and a multi-angle video image 111 and audio 112 on the basis of the multi- angle information 12 and the bit allocation information 43. A multiplexer 51 multiplexes transport streams outputted from the 1st encoder 201 and the 2nd encoder 202 into one broadcast transport stream 8 on the basis of the multi- angle information 12 and the bit allocation information 43.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video encoding method that enables multi-angle broadcasting in a band corresponding to one channel in digital TV broadcasting.

[0002]

2. Description of the Related Art In digital TV broadcasting, digital video data is compressed and encoded at an appropriate transmission rate, thereby making the most of a limited transmission bandwidth and allowing programs of many broadcasting stations to be broadcast. It is possible to broadcast at the same time. The video compression technology that made this possible is MPEG2 (Moving Pic
Environment Expert Group) standard (ISO / IEC13818)
-2).

[0003] First, video encoding based on the MPEG2 standard will be briefly described. FIG. 18 shows a block diagram of a video encoder based on MPEG2. In the frame rearranging circuit 32, the input video sequence 211 is converted into an intra-coded frame (hereinafter, referred to as an I-picture), an inter-coded frame (hereinafter, referred to as a P-picture), and an interpolated encoded frame for each frame. (Less than,
The frames are rearranged in the coding order according to the coding type of the B picture), and the motion estimation circuit 331 calculates a motion vector. The motion compensation circuit 332 calculates the data output from the adder 391 and the motion estimation circuit 33
The motion-compensated data is generated from the motion vector output from 1 and output to the differentiator 392. The motion-compensated data is decoded by the inverse quantization circuit 352 and the inverse DCT circuit 342 as the previous motion compensation for the next picture to be encoded, and output to the adder 391. The differentiator 392 calculates the difference between the data output from the motion estimator 331 and the data output from the motion compensator 332, and the DCT circuit 341 converts the video signal output from the differentiator 392 into a discrete cosine transform (DCT). Transform) processing. The quantization circuit 351 performs a quantization process on the transform coefficient output from the DCT circuit 241 using a quantization width given by the quantization control circuit 38.
The transform coefficient output from the quantization circuit 351 is encoded,
Output to the buffer 37. The buffer 37 is a VBV (Video Buffer) for simulating a buffer state at the time of decoding.
eringVerifier)
The quantization control circuit 38 obtains a target code amount calculated from the generated code amount output from the buffer 37, the bit rate, the remaining buffer amount, and other conditions, and calculates a quantization width to be given to the macroblock based on the target code amount. Quantization circuit 35
Output to 1. The data output to the buffer 37 is output as a video stream 221.

[0004] In the case of transmission with a predetermined bandwidth such as broadcast, CBR (Constant Bit Rate) coding for performing coding by fixing the video coding bit rate can be applied. However, actual video contains a variety of scenes, such as scenes that are hard to compress due to severe movement, and scenes that are easy to compress with little movement. If the bit rate is fixed as in CBR, scenes with severe movement, etc. Image quality degradation becomes noticeable. Therefore, in order to improve the image quality, a VBR (Variable) that allocates a small bit rate to a scene that is easily compressed to reduce an unnecessary bit amount and allocates a large bit rate to a scene that is difficult to compress.
Bit Rate) coding. There are roughly two types of VBR encoding methods, one is an encoding method in which encoding is performed only once, and the other is representative of a generally known two-pass encoding. The encoding is performed a plurality of times.

Here, a typical two-pass encoding will be described as an example of a method of performing encoding a plurality of times. 2
FIG. 19 shows an example of the pass video encoder. The input video signal 1 is input to the selector 141. In the first encoding, the selector 141 is connected to the terminal 2111, and the video signal 1 is input to the one-pass video encoder 302. In the one-pass video encoder 302, encoding similar to that of the MPEG2 video encoder 30 described with reference to FIG. 18 is performed, and encoding difficulty information 46 such as encoded data or generated code amount is set as a target in the second encoding. It is input to a two-pass bit allocator 45 for assigning a code amount. Here, when the data amount is large, the data obtained on the disk 72 can be recorded. The two-pass bit allocator 45 calculates a second target code amount from the obtained data and coding conditions such as the second bit rate. Next, the second encoding is performed.
In this case, the selector 141 is connected to the terminal 2112, and the video signal 1 exactly the same as the first time is input to the two-pass video encoder 303, and the two-pass bit allocator 4
The second encoding is performed based on the target code amount calculated in step 5, and a video stream 221 is output.

In broadcasting, a system stream generated by multiplexing video and audio is a transport stream (TS: Transport Stream) defined by the MPEG2 system standard (ISO / IEC13818-1).
am) is used. The transport stream is a sequence of fixed-length packets and is a CBR, but has a configuration in which a plurality of programs can be multiplexed. On the receiving side, by reading a PID (Packet IDentifier), which is a unique ID assigned to identify a packet of each program, it is possible to extract only packets necessary for the program currently being received. When using VBR in broadcasting, the maximum bit rate is limited by the bandwidth, so once the bandwidth that can be transmitted at the maximum bit rate is secured,
Even if coding is always performed at the highest bit rate or when coding is performed at a lower average bit rate, the same bandwidth is used. One method of compensating for such a drawback is a statistical multiplexing method in which a band is shared among a plurality of programs and a band is dynamically allocated while measuring the degree of difficulty of encoding between the programs. The statistical multiplexing method is suitable when a broadcasting station having a wide bandwidth capable of broadcasting many programs wants to perform a broadcast with emphasis on image quality.

By the way, there is a function called a multi-angle in which a plurality of videos from different angles are prepared in a certain scene of one content, and which one can be selected at the time of reproduction. In the case of realizing multi-angle by broadcasting, if the receiver can select one video scene different from the main broadcasting as in, for example, Japanese Patent Application Laid-Open No. 8-32887, there is a disadvantage that the band used is doubled. there were. If the statistical multiplexing method is used, in this case, the band to be used can be less than twice. However, in the case of assuming a multi-angle video in which only some scenes are different from a long video, redundant data is transmitted for a long time, so that it cannot be said that band use efficiency is high. As described above, the only way to broadcast multi-angle video in broadcasting is to increase the number of programs and the band to be used.

[0008]

However, there are some differences, such as when it is desired to view only a certain specific scene at different angles and contents, or when it is desired to perform rating (Rating) as a viewing restriction on some scenes. There are cases where a user wants to broadcast content but does not want to increase the band used. As a conventional method capable of realizing multi-angle broadcasting without increasing the band, Japanese Patent Application Laid-open No.
00-500632. In this scheme, auxiliary data can be inserted instead of the filling packet on the stream. However, since the main purpose is multiplexing of auxiliary data, when video data is added, a mechanism for transmitting the video data in time to reproduce it has not been considered.

[0009] The present invention solves the above-mentioned conventional problems. In digital TV broadcasting, when broadcasting content including multi-angle video produced in advance, multi-angle broadcasting is performed using the bandwidth of one channel. The purpose is to provide.

[0010]

To achieve the above object, the present invention provides a method for encoding a main image and a multi-angle image based on multi-angle information including at least a start time and an end time of the multi-angle image. A first coder that encodes a main video at a variable bit rate (VBR) based on multi-angle information and bit allocation information output by the bit allocator, and a multi-angle video And a multiplexer for multiplexing the encoded stream into one broadcast transport stream.

With this configuration, the multi-angle video coding and multiplexing method and apparatus of the present invention can transmit a multi-angle video in advance based on time information including a start time and an end time of the multi-angle video and bit allocation information. The band is reserved in the time zone before the start of the multi-angle video, the main video is encoded at the average bit rate excluding the secured bandwidth rate, and the separately encoded multi-angle video is encoded on the transport stream. , It is possible to transmit a multi-angle video without increasing the bandwidth.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first invention of the present invention is a video coding apparatus for coding a main video and a sub-video accompanying the main video as a coded stream of a predetermined bit rate. A bit allocator that outputs bit allocation information indicating bit allocation of the main video and the sub video in the coded stream based on time information including a start time and an end time of a sub video, and the main video is coded. A video encoding apparatus, comprising: a stream that multiplexes a stream and a stream obtained by encoding the sub-image based on the bit allocation information, and a multiplexer that generates the encoded stream. Bandwidth before the start of the sub-picture, and the average bit rate excluding the rate of the secured bandwidth The main video coded by preparative, with sub-picture which is separately encoded, can be transmitted by multiplexing on a transport stream, the multi-angle video without increasing the bandwidth. Although it is necessary to prepare video contents in advance, it is possible to encode and transmit the main video and the sub-video almost in real time with a small delay.

According to a second aspect of the present invention, in the first aspect, an encoder for encoding a main video and a sub-video based on time information and bit allocation information output by a bit allocator, and an encoded stream Is a video encoding device characterized by having a recording medium for temporarily storing a main video and a sub video, which are encoded and stored serially using an encoder, and multiplexed later. Can be sent. The advantage of this method is that only one encoder is required, and that the quality of the coded video is checked before transmission, and if the quality is not sufficient, processing such as re-encoding can be performed.

[0014] In a third aspect of the present invention based on the first aspect, the bit allocator is configured to determine whether the main video and the main video are based on at least coding difficulty information including a coding amount generated when the video is coded. Based on the bit allocation in the coded stream of the sub-picture, the 1-pass encoder that encodes the main picture and the sub-picture in order to obtain the encoding difficulty information, and based on the bit allocation information output by the time information and the bit allocator, Encoding the main image and the sub-image 2
A video encoding apparatus comprising: a pass encoder; and a recording medium for temporarily storing a stream encoded by the one-pass encoder and the two-pass encoder. 2
It can be path encoded, multiplexed and transmitted.
The advantage of this method is that the video encoding difficulty information is extracted at the time of one-pass encoding, and the bits are assigned at the time of two-pass encoding of the main video and the multi-angle video based on the extracted encoding difficulty information. By having the pass bit allocator, the image quality of all video scenes including the main video and the multi-angle video can be made uniform, and bit allocation and the like according to the visual characteristics can be performed.

[0015] A fourth invention of the present invention is a decoding device for decoding a stream generated by the first to third video coding devices, wherein a stream in which a sub-video is coded, or A recording medium for temporarily storing a stream obtained by encoding the sub-picture and audio accompanying the sub-picture, external input means for externally setting whether or not to select the sub-picture; and The time information of the start time and the end time of the sub-video is read, and when the video selected by the external input means is the sub-video, a switching signal to the output video and the output audio according to the time information is generated. A decoding device, comprising: a sub-picture setting device; a disc for recording multi-angle video and audio; By using a multi-angle setting device that generates an output video and audio switching signal according to information on the start time and end time of the single video, it is possible to configure a decoding device that can select a video on the receiving side. Become.

A system stream generated by multiplexing video and audio according to the present invention is a transport stream (TS) defined by the MPEG2 system standard (ISO / IEC13818-1).
This will be described with reference to FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1) In the present embodiment, when encoding video content including a multi-angle video prepared in advance, the main video and the multi-angle video are encoded in parallel in real time, and with a small delay. A video encoding device that multiplexes a transport stream and outputs one broadcast transport stream will be described.

FIG. 1 shows a block diagram of a video encoding apparatus according to the present embodiment.

In FIG. 1, reference numeral 6 denotes a first encoder 20.
1, second encoder 202, and multiplexer 5
External controllers 41 and 4 for externally controlling 1
Reference numeral 1 denotes a bit allocator that performs bit allocation based on the multi-angle information 12 and outputs bit allocation information 43. Reference numeral 51 denotes a multiplexed transport stream output from the first encoder 201 and the second encoder 202 and an output stream 8 Multiplexer that outputs
01 is the main video, 102 is the audio accompanying the main video, 111
Is a multi-angle video, 112 is audio accompanying the multi-angle video, 201 is bit allocation information 43 and multi-angle information 12 output from the bit allocator 41.
The first encoder 202 encodes the main video 101 and the audio 102 based on the multi-angle video 111 and the audio 1 based on the bit allocation information 43 and the multi-angle information 12 output from the bit allocator 41.
12 is a second encoder that encodes the H.12. Here, the operation of the video encoding device configured as described above will be described below.

The main video 101 and audio 102 are
And the like. Multi-angle video 1
11 and the audio 112 are recorded on a recording medium that can be reproduced simultaneously with the main video such as a VTR different from the main video.

The multi-angle information 12 will be described with reference to an example shown in FIG. In FIG. 2, the main video exists only from time 0 to time T, and the multi-angle video exists only from time t1 to t2 and only from time t3 to t4. The minimum multi-angle information in this case can be provided as a table describing the start time and end time of the main video and the multi-angle video as shown in (Table 1).

[0023]

[Table 1]

In Table 1, "Primary" indicates a main image, and "Multi-angle" indicates a multi-angle image. Note that the multi-angle information is not limited to the format shown in (Table 1), but can describe various information related to video such as bit rate information. Where C
FIG. 3 shows an example of bit allocation when BR coding is used. In FIG. 3, RMAX indicates the bit rate allowed to be occupied by video only in the final output stream 8. As shown in FIG. 3, when coding is performed and simple multiplexing is performed, it can be confirmed that the bandwidth must be twice as large as 2 × RMAX.

Hereinafter, using the multi-angle information 12,
The operation of the bit allocator 41 for allocating bits will be described.

FIG. 4 shows an example of bit allocation using the bit allocator 41. First, the total bit amount B12 allocated to the multi-angle video in the section [t1, t2] is determined. Next, B12 is subtracted from the total bit amount RMAX × t1 that can be transmitted in the band of the section [0, t1], and the number of remaining bits is allocated to the main video. Similarly, the interval [t
[T1, t3], and the total bit amount RMAX × (t3-t1) that can be transmitted in the band of the section [t1, t3] is calculated as B
By subtracting 34 and allocating the remaining number of bits to the main video, the result of the bit allocation information is obtained as shown by the shaded portion in FIG. When a stream coded based on the bit allocation obtained in this way is multiplexed, the data arrangement is as shown in FIG. From FIG. 5, it can be confirmed that the video bit rate and RMAX required as the conditions of the output stream 8 are satisfied.

An example of the bit allocation information 43 in FIGS. 4 and 5 is shown in (Table 2).

[0028]

[Table 2]

In Table 2, Primary is the main video, Play is the multi-angle video at the time of reproduction, Tran is
“sport” indicates a multi-angle image when multiplexing the stream. Here, the bit rate RMAX = 9 Mbp
s. According to (Table 2), at time 0, the main video is 6 Mbps, and the scene 1 of the multi-angle video is 3M.
Each is encoded and multiplexed at bps. Next, at time t1, the main video is encoded at 7.2 Mbps and the scene 2 of the multi-angle video is encoded at 1.8 Mbps and multiplexed. Also, it can be confirmed that the total bit rate of the main video and the multi-angle video is the upper limit of 9 Mbps. On the playback side, on the other hand, at time t1, scene 1 of the multi-angle video is played at 6.6 Mbps. Hereinafter, the bit allocation information can be similarly interpreted. Table 2 is an example of bit allocation information, and the description format is not limited to this as long as it can describe the start / end time of multi-angle video and coding bit rate information. Also, the time in the display order of the display etc. was used as the time at the time of bit allocation instead of the time in the encoding order, but this is for the sake of simplicity of explanation and has an essential effect on the implementation of the invention. Note that there is no.

Next, in FIG. 1 showing the overall configuration, bit allocation information 43 obtained from a bit allocator 41 is shown.
And the multi-angle information 12, the first encoder 201 encodes main video and audio, and the second encoder 202 encodes multi-angle video and audio. The operation of the encoders 201 and 202 will be described with reference to the block diagram inside the encoder shown in FIG.

In FIG. 6, reference numeral 20 denotes an encoder, and 23 receives an audio coded information 252 output from a coded information setting unit 25 and receives an input audio 2 accompanying an input video.
An audio encoder that encodes 12 and outputs an audio stream. A system 24 generates a transport stream 29 by multiplexing the video stream 221 and the audio stream 222 from system encoding information 253 output from an encoding information setting unit 25. The encoder 25 generates video coding information 43, audio coding information 252, and coding information for generating system coding information 253 from the bit allocation information 214, the multi-angle information 12, and the coding information input from the external setting means 216. The setting unit 26 processes an external control signal to enable the synchronous operation of the coding information setting unit 251, the video encoder 30, the audio encoder 23, and the system encoder 24. The coding controller 30 includes a video coding information 251 and a code. Chemical Controller 26 an input image 212 from the control signal inputted by coding from a video stream 2
The video encoders 30 and 216 which output 21 are external setting means for setting encoding information externally. An encoding controller 251 of the video encoder 30, the audio encoder 23, and the system encoder 24 indicates video encoding information. Here, in FIG. 6, the same reference numerals are assigned to the same information as in FIG.

First, for the first encoder 201, the input video 211 is composed of the main video 101 and the input audio 2 shown in FIG.
Reference numeral 12 corresponds to the audio 102 attached to the main video in FIG.
The start time and the end time of the main video 211 are given as the multi-angle information 12, and the bit allocation information 43 calculated by the bit allocator 41 is given as the bit allocation information 43. In the coding information setting unit 25, video coding information 251, audio coding information 252, and system coding information 253 are generated from the bit allocation information 43 and the multi-angle information 12. Video coding information 25
For example, VB such as a target code amount in GOP units
Gives information for R encoding. Audio encoding information 25
As 2, a bit rate reserved for audio is given. As the system encoding information 253, the multi-angle information 12 and the bit allocation information 43 are given in an appropriate format. Further, information set by the external setting means 216 can be given to each piece of encoded information. For example, as the system encoding information 253, a PID (Packet Ident
ifier).

The video encoder 30 converts the main image 211 into a VB based on the video encoding information 251 in real time.
R encoding is performed to obtain a video stream 221.

FIG. 7 is a block diagram showing the configuration of the video encoder 30. As shown in FIG. 7, the video encoder 30 has a configuration based on the MPEG2 standard described in the related art, and video encoding information 251 is input to the quantization control circuit 38. Here, an important point when encoding the main video is the occupancy of the VBV buffer at the time of encoding. As described above, the VBV buffer corresponds to the buffer 37 in FIG.

FIG. 8 shows the occupancy of the VBV buffer during encoding. As shown in FIG. 8, when switching from the main video to the multi-angle video and when switching from the multi-angle video to the main video (time t1, t2, t3,
At time t4), the quantization control circuit 38 performs encoding control to match the buffer occupancy in the decoder. That is, encoding control is performed so as to satisfy (Equation 1).

[0036]

(Equation 1)

As described above, by limiting the buffer occupancy, even when a multi-angle video is selected in the decoder, the buffer occupancy of the decoder can be guaranteed by the VBV buffer simulation performed at the time of encoding. There are several ways to guarantee this buffer occupancy, but the simplest method is to provide the buffer occupancy value in advance as a fixed value at all times when the buffer position needs to be matched. It is. In addition, there is a method in which the other follows the buffer occupancy of the main video or the multi-angle video that has been encoded first, but this method is used in the present embodiment for simultaneously encoding the main video and the multi-angle video. It is complicated to apply to the form.

As described above, the video encoder 30
Performs encoding that guarantees buffer occupancy. At the same time, the audio encoder 23 encodes the audio 211 attached to the main video in real time based on the audio encoding information 252, and obtains an audio stream 221.

The system encoder 24 is assumed to be a system encoder 241 as shown in FIG. 9 in the first encoder. The operation of the system encoder 241 is shown in FIG.
This will be described with reference to FIG.

In FIG. 9, reference numeral 243 denotes a video packetizer for generating video packets from the input video stream 221;
An audio packetizer 245 for generating an audio packet from a temporary packet generator 245 for generating a temporary packet based on the system coding information 253,
6 multiplexes the video packet output from the video packetizer 243, the audio packet output from the audio packetizer, and the temporary packet output from the temporary packet generator 245, and multiplexes the transport stream 29.
1, a system multiplexer 247, a video packetizer 243, an audio packetizer 244, a temporary packet generator 245, and a system multiplexer 24.
A clock generator 249 for synchronizing 6 is a control signal from the encoding controller 26 in FIG.

First, the video packetizer 243 obtains information such as a PID from the system coding information 253, and generates a video packet. Audio packetizer 2 at the same time
44 also generates audio packets. The temporary packet generator 245 generates a temporary packet based on the bit allocation information and the PID included in the system encoding information 253. Here, the temporary packet is a packet to be inserted at a position where a multi-angle stream is inserted in order to multiplex the multi-angle stream later. The system multiplexer 246 multiplexes the generated video packets, audio packets, and provisional packets,
Generate the transport stream 291. The above operation is performed in synchronization with the clock generated by the clock generator 247. In the first encoder, the main video, audio,
Transport stream 29 in which temporary packets are multiplexed
Get 1.

On the other hand, in FIG. 1 showing the overall structure, for the second encoder 202, the input video 211 corresponds to the multi-angle video 111 and the input audio 212 corresponds to the audio 112 attached to the multi-angle video. The start time and end time of the multi-angle video are given as the multi-angle information 213, and the bit allocation information derived by the bit allocator 41 is given as the bit allocation information 214. In FIG. 6 showing the inside of the encoder, the second encoder 2
02, the video encoding information 251, the audio encoding information 252, and the system encoding information 253 generated by the encoding information setting unit 25 correspond to the first encoder 201.
Is the same as

The video encoder 30 converts the multi-angle image 211 into a VB based on the video encoding information 251.
R encoding is performed to obtain a video stream 221. At the same time, the audio encoder 23 outputs the audio encoded information 25
2 based on the audio 21 attached to the multi-angle video
1 to obtain an audio stream 222. The system encoder 24 is the second encoder shown in FIG.
Is assumed to be the system encoder 242 shown in FIG.
The operation of the system encoder 242 will be described with reference to FIG.

In FIG. 10, reference numeral 249 denotes a control signal from the encoding controller 26, and 253 denotes an encoding information setting unit 2.
5, 243 is a video packetizer for generating video packets from the input video stream 221, and 244 is an input audio stream 22.
An audio packetizer 246 for generating an audio packet from the video packetizer 2 and the video packetizer 246 output from the video packetizer 243;
4 is a system multiplexer for multiplexing the audio packets output from 4 and generating a transport stream 292, and 247 is a video packetizer 243, audio packetizer 244, and system multiplexer 246.
This is a clock generator for synchronizing. Here, in FIG. 10, the same components as those in FIG. 6 are assigned the same numbers.

First, the video packetizer 243 obtains information such as PID from the system encoding information 253, and generates a video packet. Audio packetizer 2 at the same time
44 also generates audio packets. The system multiplexer 246 multiplexes the generated video packets and audio packets, and generates a transport stream 291. Here, it should be noted that when encoding a multi-angle video, the encoding may be performed with a shorter start time without performing the encoding based on the start and end times of the multi-angle. is there.

FIG. 11 shows an example of changing the encoding time. FIG.
In FIG. 1, the upper diagram shows the time at which the multi-angle video should be originally reproduced. If encoding is performed at this playback time, since the time at which the encoded data is to be multiplexed has already passed, it is necessary to accumulate the main video stream in a buffer and wait until the multi-angle video stream arrives. Occurs. Therefore, as shown in the lower part of FIG. 11, if the encoding time is advanced and the coding is performed in a vacant time,
The delay of the main image can be reduced. in this case,
In FIG. 10, it is necessary to shift the time information described in each packet using the system encoding information 253 to the time at which the multi-angle video and audio are reproduced, and write the time information. In this case, for a multi-angle video encoded between time t1 and time t2, time 0 to time t
2-t1, for a multi-angle video encoded from time t3 to time t4, from time Ta to time Ta + t4
Describe -t3. By changing the encoding time in this way, when the main video and the multi-angle video are finally multiplexed, the delay due to the buffer is reduced, so that encoding with less delay can be realized as a whole system. Becomes As described above, in the case of the second encoder, a transport stream 29 in which multi-angle video and audio are multiplexed is obtained.

In FIG. 1 showing the overall structure, a multiplexer 51 multiplexes the output transport stream of the first encoder 201 and the output transport stream of the second encoder 202 obtained as described above.

The operation of the multiplexer 51 will now be described with reference to FIG.
2 will be described.

In FIG. 12, reference numeral 511 denotes a main video / audio transport stream of the main video and audio output from the first encoder 201, and 512 denotes a multi-angle video and audio multi-stream output from the second encoder 202. An angle video / audio transport stream 513 is a main video / audio buffer which is a first buffer for storing a main video / audio transport stream 511, and a multi-angle video / audio transport stream 512 is a second buffer 512. The multi-angle video / audio buffer 515 to be stored, which is a buffer of 515, monitors the transport stream 511 of the main video / audio,
A PID determiner 8 is a data rewriting means for detecting a PID of a provisional packet to be filled with the multi-angle video / audio transport stream 512, and 8 is a multiplexed output transport stream.

The main video / audio transport stream 511 input in real time is stored in the main video / audio buffer 513. On the other hand, multi-angle video
The audio transport stream 512 is stored in the multi-angle video / audio buffer 514. The PID determiner 515 detects the PID of the temporary packet in the main video / audio transport stream 511, and if the packet is a temporary packet, passes the switch 516 to replace the packet with a multi-angle video / audio buffer. As described above, the output transport stream 8 is obtained.

Here, the concept of packet replacement will be described with reference to FIG.

In FIG. 13, V is a video packet, A
Indicates a voice packet, and T indicates a temporary packet. The upper packet sequence is a main video / audio transport stream, and the lower packet sequence is a multi-angle video / audio transport stream. When detecting the provisional packet T on the main video / audio transport stream, the PID determination unit 515 connects the switch 516 to the main video / audio buffer and replaces the provisional packet with a multi-angle video / audio transport stream packet. .
By performing such operations, the number of packets of the main video / audio transport stream does not change before and after the replacement. As described above, since the temporary packet is inserted based on the bit allocation information, multiplexing can be performed only by replacing the packets in order as shown in FIG.

FIG. 14 shows a decoder for receiving the transport stream generated in the present embodiment.

In FIG. 14, reference numeral 8 denotes a received transport stream; 71, a disk controller for reading and writing to a disk 72; 72, a disk for recording multi-angle video and audio; 90, a decoder; A multi-angle setting unit 96 is a sub-picture setting unit that obtains multi-angle information from the transport stream stored on the disc 72 and provides switching information between video and audio. Reference numeral 96 denotes a video and audio output from the multi-angle setting unit 95. A switch 515 for switching between main video and audio, multi-angle and audio based on the switching information, 515 a PID determiner for determining the PID of the transport stream, 516 a switch that operates based on the determination result of the PID determiner 515, 911
Is a video output from the decoder 90, 912 is an audio output from the decoder 90 913 is an external input means for providing multi-angle setting information to the multi-angle setting device 95, 913 is a multi-angle setting information to the multi-angle setting device 921 is a video buffer for storing a video stream, 922 is an audio buffer for storing an audio stream, 931 and 932 are demultiplexers for decomposing a transport stream into video and audio streams, 9
41 is a video decoder for decoding a video stream stored in the video buffer 921, 942 is an audio decoder for decoding an audio stream stored in the audio buffer 922, and 9201 is for storing transport streams of main video and audio. Reference numeral 9202 denotes a buffer for storing multi-angle video and audio transport streams.

In the transport stream 8 received by the decoder 90, the PID determiner 515 determines whether the PID that can be known from the program map table or the like is a multi-angle PID. The switch 516 is operated based on the determination result, and the main video and audio transport streams are transferred to the buffer 9201, and the multi-angle video and audio transport streams are transferred to the buffer 9202. In the buffer 9202, the transferred stream is accumulated on the disk 72 through the disk controller 71. The main video and audio transport streams are transmitted to the demultiplexer 9.
The video stream is decomposed into video and audio streams by 31 and transferred to a video buffer 921 and an audio buffer 922, respectively. The video decoder 941 and the audio decoder 942 respectively determine a video stream and a video stream based on the time stamp information obtained by the demultiplexer 931.
Decode the audio stream. Disk 72
Are transmitted to the demultiplexer 932 under the control of the disk controller 71, and are decomposed into video and audio streams by the demultiplexer 932, and the video buffer 921 and the audio buffer 922, respectively.
And decoded by the video decoder 941 and the audio decoder 942.

On the other hand, the multi-angle setting device 95 to which the multi-angle setting information is inputted from the external input means 913
Is connected to the disk 72 via the disk controller 71.
The start time and the end time of multi-angle video and audio are confirmed from the time stamp of the transport stream stored in the. When the main input is set to the main video and audio from the external input means 913, the switch 96 is always set to the main video and audio side. When the multi input is set to the multi-angle video and audio, the switch 96 is set to the start time obtained from the disc. To the multi-angle video and audio side, and return to the main video and audio side at the end time.

As described above, the reproduction output can be switched by the decoder. In the present embodiment, a disk and a disk controller are used to store a stream. However, the present invention is not limited to this, as long as it meets the purpose of storing a stream.

In this embodiment, the multi-angle image is 1
The two cases have been described.
Since the same method can be used to encode one or more codes, the details will not be described.

Further, in the present embodiment, the description has been made using different system encoders when encoding the main video and the multi-angle video, but this is for the purpose of simplifying the description. Actually, the two system encoders differ only in the presence / absence of the provisional packet generator. Therefore, using the system encoder having the provisional packet generator in FIG. 9, the provisional packet generator is used in the system encoder in the second encoder. If used without operation, the first encoder and the second encoder can have exactly the same configuration.

Also, in the present embodiment, an example has been shown in which the multi-angle video of FIG. 2 is encoded as shown in FIG. 5 with respect to bit allocation. In this example, the time zone for reproducing the multi-angle video and the time zone for encoding do not temporally overlap at all. However, if the total delay time that occurs between the time when the multiplexed data is stored on the decoder disk and the time when it is played back is calculated, and the minimum delay time is guaranteed, the playback time zone of the multi-angle video and the encoding time Time zones may overlap.

(Embodiment 2) In Embodiment 2 of the present invention, when encoding video content including a multi-angle video prepared in advance, the main video and the multi-angle video are encoded using one encoder. A description will be given of a video encoding apparatus that serially encodes and accumulates the information on a disk, and subsequently outputs one multiplexed broadcast transport stream read from the disk and multiplexed.

FIG. 15 is a block diagram of a video encoding apparatus according to the present embodiment.

In FIG. 15, reference numeral 6 denotes a bit allocator 4
1, an external controller for externally controlling the encoder 20, the disk controller 71, and the multiplexer 52; 8, a transport stream output from the multiplexer 52; 41, multi-angle information 1
2, a bit allocator that performs bit allocation based on
0 is a video 13 based on the bit allocation information 43 output from the bit allocator 41 and the multi-angle information 12.
1 and an encoder for encoding the audio 132; 52, a multiplexer for multiplexing the video and audio streams read from the disk 72 into the transport stream 8; 71, a disk controller for reading from and writing to the disk 72; Controller 71
, A disk for recording the stream output from the encoder 20 according to the above, 131 is all video including the main video and multi-angle video, and 132 is audio accompanying the video. In FIG. 15, components having the same operations as those of the video encoding device according to the first embodiment are assigned the same reference numerals.

The operation of the video encoding apparatus configured as described above will be described below.

All the video 131 including the main video and the multi-angle video and the audio 132 accompanying the video are
And the like. As an example, an example in which the multi-angle video shown in FIG. 2 is recorded in the order shown in FIG. 16 will be described. The main video and the multi-angle video may be recorded in any way as long as they can be reproduced. The multi-angle information 12 is the same as that described in the first embodiment. However, when the video is divided and recorded on a recording medium such as a VTR, the format must be compatible with the video. The operation of bit allocator 41 for performing bit allocation using multi-angle information 12 is the same as that described in the first embodiment.

Next, using the bit allocation information 43 obtained from the bit allocator 41 and the multi-angle information 12, the encoder 20 encodes all video and audio. The configuration inside the encoder is the same as that described in Embodiment 1 with reference to FIG. As described in the first embodiment, the encoder for the main video and the encoder for the multi-angle video can have exactly the same configuration. The stream output from the encoder 20 will be described with reference to FIG. A multi-angle video source as shown in FIG. 2 is encoded in the order shown in FIG. In this case, at the time [0, T), the main video is encoded, and the transport stream 1 is obtained. Next, the multi-angle scene 1 is encoded at the time [T, T + T1), and the transport stream 2 is obtained. Finally, a multi-angle scene 2 is encoded at a time of [T + T1, T + T1 + T2), and a transport stream 3 is obtained. As described above, at least the main video and the multi-angle video need to be separated into separate streams.

The disk controller 71 stores the transport stream output from the encoder on the disk 72. The multiplexer 52 multiplexes a plurality of transport streams stored on the disk 72. The operation of the multiplexer 52 is different from that of the multiplexer 51 described in the first embodiment in that the operation is sequentially read from the disk and multiplexed. However, since the operation can be basically performed with the same configuration, detailed description is omitted. The operation of the decoder is as described in the first embodiment.

Although the present embodiment has been described with reference to a case where there is one multi-angle image, a case where two or more multi-angle images are encoded in parallel can be realized using the same method. Do not write.

In this embodiment, an example is shown in which a disk and a disk controller are used. However, a medium that can store and read a stream is not limited to a disk.

In the present embodiment, the encoder 20 generates the transport stream. However, the encoder 20 may output the video and audio elementary streams. In this case, it is possible to adopt a configuration in which each elementary stream is stored in a storage medium and converted into a transport stream by a system encoder prepared in the multiplexer when the multiplexer 52 multiplexes the elementary streams.

(Embodiment 3) In Embodiment 3 of the present invention, when encoding video content including a multi-angle video prepared in advance, the main video and the multi-angle video are encoded once to encode the video. Measure the encoding difficulty information, perform bit allocation based on the measured encoding difficulty information, encode the main video and the multi-angle video in series based on the obtained bit allocation information, and accumulate them on the disc. A video encoding apparatus that outputs one broadcast transport stream read from a disk and multiplexed later will be described.

FIG. 17 is a block diagram of a video encoding apparatus according to the present embodiment.

In FIG. 17, reference numeral 6 denotes an external controller for externally controlling the 2-pass bit allocator 42, the 1-pass encoder 203, the 2-pass encoder 204, the disk controller 71, and the multiplexer 52.
42 is a two-pass bit allocator that performs bit allocation based on the multi-angle information 12 and the encoding difficulty information 46, 44 is the bit allocation information output from the two-pass bit allocator, and 46 is the encoding output from the one-pass encoder 203. Difficulty level information, 52 is a multiplexer for multiplexing the video and audio streams read from the disc into the transport stream 8, a multiplexer 71 is a disk controller for reading and writing to the disc, and 72 is a stream that records the stream output from the encoder 20. Disc 141, all video including the main video and multi-angle video, 142 is audio accompanying all the video,
151 is an input video terminal of the 1-pass encoder 203;
2 is an input video terminal of the two-pass encoder 204, 161 is an input audio terminal of the one-pass encoder 203, 162 is an input audio terminal of the two-pass encoder 204, and 203 is a video 141 and an audio 142 based on the multi-angle information 12.
The first pass encoder 204 performs the first encoding of the video 1 based on the bit allocation information 44 and the multi-angle information 12 output from the two-pass bit allocator 42.
This is a two-pass encoder that performs a second encoding of the audio 41 and the audio 142. Here, in FIG. 17, the same numbers are assigned to components that perform the same operations as those of the video encoding devices according to the first and second embodiments.

The operation of the video encoding apparatus configured as described above will be described below.

All the images 141 including the main image and the multi-angle image and the sound 142 attached to all the images are
It is recorded on a recording medium such as a VTR. The recording method is the same as the method described in the second embodiment. The multi-angle information 12 is the same as that described in the first embodiment. However, when the video is divided and recorded on a recording medium such as a VTR, the format must be compatible with the video.

When one-pass encoding is performed, the image 141
Is connected to the terminal 151, and the audio 142 is connected to the terminal 161. The one-pass encoder 203 outputs the multi-angle information 1
2, encoding of all video and audio including the main video and the multi-angle video is performed. The details of the one-pass encoding are as described with reference to FIG. 19 in the related art. The encoding difficulty information 46 obtained by one-pass encoding is temporarily stored in the disk 72 through the disk controller 71. Here, the encoding difficulty information 46 assumes the amount of generated code output from the one-pass encoder 203, but the
May be processed and edited after the information output from is accumulated on the disk 72.

The 2-pass bit allocator 42 uses the multi-angle information 12 and the encoding difficulty information 46 read from the disk 72 to calculate bit allocation information 44 to be input to the 2-pass encoder 204. As a result, it is possible to assign bits according to the degree of difficulty of an actual video including a multi-angle video. The operation of the 2-pass bit allocator 42 that performs bit allocation is basically the same as that described in the first embodiment, but by using the encoding difficulty information 46, the bit allocation for further improving the image quality can be performed. It is possible to do. When the two-pass encoding is performed, the video 141 is connected to the terminal 152 and the audio 142 is connected to the terminal 162. The two-pass encoder 204
Using the bit allocation information 44 obtained from the two-pass bit allocator 42 and the multi-angle information 12, all the video 141 and audio 142 are encoded. The configuration inside two-pass encoder 204 is the same as that described in the first embodiment. The stream output from the 2-pass encoder 204 is as described in the second embodiment with reference to FIG. In addition, when a transport stream is generated by performing system encoding on a main video, the first embodiment is used.
The insertion of the temporary packet described in (1) is performed. The disk controller 71 stores the transport stream output from the encoder on the disk 72.

The multiplexer 52 performs the same operation as that described in the second embodiment. The operation of the decoder is as described in the first embodiment.

Although the present embodiment has been described with reference to a case where there is one multi-angle image, a case where two or more multi-angle images are encoded in parallel can be realized by using the same method. do not do.

In the present embodiment, an example is shown in which a disk and a disk controller are used. However, any medium that can store and read a stream is not limited to a disk.

In this embodiment, the two-pass encoder 204 generates a transport stream. However, the two-pass encoder 204 may output a video and an audio elementary stream. In this case, it is possible to adopt a configuration in which each elementary stream is stored in a storage medium and converted into a transport stream by a system encoder when multiplexed by the multiplexer 52.

In the present invention, two-pass encoding has been described as an example. However, even when encoding is performed three or more times, there is essentially no change in the processing means, and the encoding can be easily performed. Further, in the present embodiment, the number of encoders is set to two for ease of explanation, but all the images including the main image and the multi-angle image are encoded a plurality of times using one encoder. The configuration is also
It can be realized similarly.

[0083]

As is apparent from the above description, according to the present invention, in the case of broadcasting a content including a multi-angle video that has been produced in advance in digital TV broadcasting, the time information on the presence of the multi-angle video and Based on the bit allocation information, a band capable of transmitting the multi-angle video in advance is secured in a time period before the start of the multi-angle video, and the main video is VBR-coded at an average bit rate excluding the secured bandwidth. By multiplexing each transport stream, multi-angle video can be transmitted without increasing the bandwidth.

In the case where encoding is performed a plurality of times, the bit assignment of all the images including the main image and the multi-angle image can be performed using the encoding difficulty information of the image. This makes it possible to configure a video encoding device in which the overall image quality is made uniform and the overall image quality is improved.

Further, at the time of decoding, a disk for recording multi-angle video and audio, and a switching signal for output video and audio in accordance with information on the start and end times of the multi-angle video read from the stream. By using the generated multi-angle setting device, it is possible to configure a decoding device capable of selecting a video on the receiving side.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a video encoding device according to a first embodiment of the present invention.

FIG. 2 is a diagram showing an example of a multi-angle video.

FIG. 3 is a diagram illustrating an example of CBR encoding of a multi-angle video.

FIG. 4 is a diagram showing an example of the bit allocation.

FIG. 5 is a diagram showing an example of data arrangement on the stream.

FIG. 6 is a block diagram showing a configuration of the encoder.

FIG. 7 is a block diagram showing a configuration of the video encoder.

FIG. 8 is a diagram showing adjustment of the occupancy of the VBV buffer;

FIG. 9 is a block diagram showing a configuration of the first system encoder.

FIG. 10 is a block diagram showing the configuration of the second system encoder;

FIG. 11 is a diagram showing an example of changing the encoding time of the multi-angle video.

FIG. 12 is a block diagram showing a configuration of the multiplexer.

FIG. 13 is a conceptual diagram showing a packet replacement method in the multiplexer.

FIG. 14 is a block diagram showing a configuration of a decoder according to the embodiment;

FIG. 15 is a block diagram showing a configuration of a video encoding device according to a second embodiment of the present invention.

FIG. 16 is a diagram showing an encoding order of the main video and the multi-angle video.

FIG. 17 is a block diagram showing a configuration of a video encoding device according to a third embodiment of the present invention.

FIG. 18 is a block diagram showing a configuration of a video encoder based on MPEG2.

FIG. 19 is a block diagram showing a configuration of an apparatus for performing two-pass encoding.

[Explanation of symbols]

6 External controller 8 Output transport stream 12 Multi-angle information 20 Encoder 24 System encoder 41 Bit allocator 42 2-pass bit allocator 43, 44 Bit allocation information 46 Encoding difficulty information 51, 52 Multiplexer 71 Disk controller 72 Disk 95 Multi-angle setting Device 101 main video 102 audio attached to main video 111 multi-angle video 112 audio attached to multi-angle video 131, 141 video 132, 142 audio attached to video 201 first encoder 202 second encoder 203 one-pass encoder 204 2-pass encoder 245 Temporary packet generator 511 Transport stream of main video 512 Transport stream of multi-angle video 5 13 Main video / audio buffer 514 Multi-angle video / audio buffer 515 PID decision unit 913 External input means

Continued on front page F-term (reference) 5C053 FA20 FA22 FA23 GB02 GB06 GB08 GB11 GB12 GB18 GB19 GB22 GB26 GB32 GB38 GB40 JA22 KA08 KA18 KA24 5C059 KK23 MA00 MA05 MA23 MC11 RB01 RB10 RC04 RC12 RC32 SS02 SS11 SS20 TA60 TB02 TC37 TC16 TC16 TC16 TC24 UA02 UA05 UA08 5C063 AB03 AB07 AC01 AC05 AC10 CA11 CA12 DA01

Claims (27)

[Claims] 1. A video encoding apparatus for encoding a main video and a sub-video accompanying the main video as a coded stream of a predetermined bit rate, comprising a time including at least a start time and an end time of the sub-video. Based on the information, a bit allocator that outputs bit allocation information indicating the bit allocation of the main video and the sub-video in the coded stream, a stream that encodes the main video and a stream that encodes the sub-video A video encoding device, comprising: a multiplexer that multiplexes based on the bit allocation information and generates the encoded stream. 2. A method according to claim 1, wherein the bit allocator allocates bits in the coded stream of the main video and the sub video based on at least coding difficulty information including a code amount generated when the video is coded. The video encoding device according to claim 1, wherein: 3. A first encoder for encoding a main image based on time information and bit allocation information output by a bit allocator, and a second encoder for encoding a sub-image based on the time information and the bit allocation information. The video encoding device according to claim 1 or 2, further comprising: 4. An encoder for encoding a main video and a sub video based on time information and bit allocation information output by a bit allocator, and a recording medium for temporarily storing the encoded stream. 3. The video encoding device according to claim 1, wherein: 5. A one-pass encoder for encoding a main image and a sub-image in order to obtain encoding difficulty information, and the main image and the sub-image based on time information and bit allocation information output from a bit allocator. And a recording medium for temporarily storing a stream encoded by the one-pass encoder and the two-pass encoder.
The video encoding device according to Claim 1. 6. A first encoder, encoder or 2
The path encoder, when encoding the main video and the sub video, at all the start time and all the end time of the sub video, the buffer occupancy at the time of decoding the stream obtained by encoding the main video and the sub video 3. An encoding control circuit for performing control for matching
The video encoding device according to claim 4. 7. An apparatus according to claim 3, further comprising an audio encoder for simultaneously encoding audio accompanying the main video or the sub video when encoding the main video or the sub video. The video encoding device according to claim 4, claim 5, or claim 6. 8. A system encoder according to claim 3, further comprising a system encoder for packetizing the encoded main video data and multiplexing the data into a transport stream for transmission. Item 5. The video encoding device according to claim 6. 9. A system encoder for packetizing encoded main video data and audio data encoded along with said main video and multiplexing them into a transport stream for transmission. The video encoding device according to claim 7, further comprising: 10. A tentative packet for generating a tentative packet at a position where a multiplexed stream of a sub-picture is inserted by a multiplexer based on bit allocation information output from a bit allocator in a coded stream of a main picture. The video encoding device according to claim 8, further comprising a generator. 11. A multiplexer, comprising: a first buffer for temporarily storing a transport stream; and a stream encoding a sub-picture or a stream encoding the sub-picture and audio accompanying the sub-picture. A second buffer for temporarily storing data, and data rewriting means for replacing a temporary packet existing in the first buffer with a stream existing in the second buffer. The video encoding device according to claim 10. 12. A stream obtained by encoding a sub-picture or a stream obtained by encoding the sub-picture and audio accompanying the sub-picture is data obtained by encoding the sub-picture,
12. The video encoding device according to claim 11, wherein data obtained by encoding the sub-video and audio accompanying the sub-video are packetized. 13. A video encoding method for encoding a main video and a sub video accompanying the main video with a coded stream of a predetermined bit rate, wherein the time includes at least a start time and an end time of the sub video. A bit allocation step of performing bit allocation of the main video and the sub video in the coded stream based on the information, and multiplexing the main video coded stream and the sub video coded stream. A video multiplexing step. 14. The bit allocating step performs bit allocation in the coded stream of the main video and the sub-video based on at least coding difficulty information including a code amount generated when the video is coded. 14. The video encoding method according to claim 13, wherein: 15. A first encoding step for encoding a main video based on time information and bit allocation information output from the bit allocation step, and encoding a sub-video based on the time information and the bit allocation information. 15. The video encoding method according to claim 13, further comprising a second encoding step. 16. The video code according to claim 13, further comprising an encoding step of encoding the main image and the sub-image based on the time information and the bit allocation information output by the bit allocation step. Device 17. The main video based on a one-pass encoding step of encoding a main video or a sub-video to obtain encoding difficulty information, and time information and bit allocation information output by a bit allocation step. The video encoding method according to claim 14, further comprising a two-pass encoding step of encoding the sub-image. 18. The first encoding step, the encoding step or the two-pass encoding step, when encoding a main video and a sub-video, at all start times and all end times of the sub-video. 18. The coding method according to claim 15, further comprising an encoding control step of performing control for matching the buffer occupancy at the time of decoding of the stream obtained by encoding the main image and the sub-image. Video encoding method. 19. The method according to claim 15, further comprising an audio encoding step for simultaneously encoding audio accompanying the main video or the sub video when encoding the main video or the sub video. Claim 16, claim 17,
The video encoding method according to claim 18. 20. The system according to claim 15, further comprising a system encoding step for packetizing the encoded main video data and multiplexing the data into a transport stream for transmission. , Claim 17,
The video encoding method according to claim 18. 21. System encoding for packetizing encoded main video data and audio data encoded along with the main video and multiplexing them into a transport stream for transmission. The video encoding method according to claim 19, further comprising a step. 22. A system encoding step includes, in a coded stream of a main video, a provisional packet at a position where a coded stream of a sub-video is inserted by a multiplexing step, based on bit allocation information output by the bit allocation step. 22. The video encoding method according to claim 20, further comprising a provisional packet generating step of generating a temporary packet. 23. The multiplexing step includes a data rewriting step for replacing a temporary packet with a stream obtained by encoding a sub-picture or a stream obtained by encoding the sub-picture and audio accompanying the sub-picture. 23. The video encoding method according to claim 22, wherein: 24. A stream obtained by encoding a sub-picture or a stream obtained by encoding the sub-picture and audio accompanying the sub-picture is composed of data obtained by encoding the sub-picture or the sub-picture and the sub-picture. 24. The video encoding method according to claim 23, wherein the data obtained by encoding the attached audio is packetized. 25. A recording medium on which a stream generated by using the video encoding device and method according to claim 1 is recorded. 26. A decoding device for decoding a stream generated by using the video encoding device and method according to claim 1, wherein the stream is obtained by encoding a sub-video. Or a recording device for temporarily storing a stream obtained by encoding the sub-video and audio accompanying the sub-video. 27. An external input means for externally setting whether or not to select a sub-picture, and reading time information of a start time and an end time of the sub-picture from an encoded stream and selecting the sub-picture by the external input means. 27. The decoding apparatus according to claim 26, further comprising: a sub-picture setting device that generates a switching signal for switching to an output picture and an output sound according to the time information when the picture is a sub-picture.