JP2006279550A - Video transmitting device and receiving and reproducing device therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video transmitting device which has more transmission lines in a video transmission environment wherein one or more transmission lines are present and realizes video transmission so that a user can view video of higher quality. <P>SOLUTION: An object constitution part 11 divides video into Obj1 to ObjM. An encoding part 12 generates a basic encoded stream B, an extended encoded stream E, and rate distortion characteristics. A layering part 13 layers encoded data of respective pictures as to (object, quality, and time). As an example, the objects are layered into Obj1 to ObjM, the quality is layered into a basic layer and an extended layer (further fractionized into K layers using set encoding quantities T (=T[1], T[2], ..., T[K])), and the time is layered into I, P, and B pictures. A transmission part 14 distributes and transmits encoded units through transmission lines 1, 2, ..., N in the decreasing order of priority. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a video transmission apparatus and a reception / playback apparatus for the same, and particularly in a video transmission environment in which one or two or more transmission paths exist, a user having more transmission paths can view higher quality video. The present invention relates to a video transmission apparatus and a reception / reproduction apparatus thereof.

  In recent years, digital video distribution has become easier due to improved computer performance and wider network bandwidth. On the other hand, due to the diversification of networks, the types of transmission paths possessed by video receivers are different. From the viewpoint of broadcasting, it is important that all video receivers can view the same video regardless of differences in the image receiving environment even when various image receiving environments are mixed.

  As a technique for solving this problem, there are a hierarchical encoding technique and a video transmission system using the same. In the hierarchical encoding technique, an encoded stream divided into two layers, a basic layer and an extended layer, is created. The basic layer includes minimum information for reproducing a video, and the extended layer includes information added to the basic layer to improve the quality of a reproduced image.

  In the following Patent Document 1 which is a patent application by the present applicant, the code of the encoded stream to be transmitted is used in accordance with the bandwidth of the transmission path possessed by the video receiver and the processing capability of the display device using the hierarchical encoding technology. There has been proposed a video transmission apparatus in which the amount is controlled so that the highest quality video can always be viewed.

In this apparatus, the encoding / accumulating means hierarchically encodes each object image into a basic hierarchy and an extension hierarchy, and creates and stores each stream. At this time, bit-plane encoding is performed in the enhancement layer. Further, at the time of encoding, a rate distortion characteristic indicating a relationship between a code amount and encoding distortion when the codeword of the enhancement layer encoded stream is sequentially decoded for each object image is generated and held. The transmission means allocates the code amount for each object image based on the rate distortion characteristic and the bandwidth of the communication channel, and cuts out and transmits the extended encoded stream for each object according to the allocation.
Japanese Patent Application No. 2003-387817 Image transmission device

  The conventional video transmission apparatus assumes video transmission via a single transmission line, and in such an environment, the amount of code to be transmitted is changed to reflect the state of the transmission line. It was not supposed to be an environment where there was.

  Therefore, when video transmission using a conventional video transmission device is performed in an environment where a plurality of transmission paths exist, the second and subsequent transmission paths remain empty, and a plurality of transmission paths are used at the same time. Efficient video transmission was not possible.

  Therefore, since the user can only view the same quality video as the user who has only a single transmission path despite having a plurality of transmission paths, the video transmission reflecting the user's reception environment cannot be performed. There wasn't.

  The present invention has been made in view of the above-described conventional technology, and its purpose is higher for a user having more transmission paths in a video transmission environment in which one or more transmission paths exist. An object of the present invention is to provide a video transmission apparatus and a reception / reproduction apparatus for realizing video transmission that enables viewing of quality video.

  In order to achieve the above-mentioned object, the present invention extracts a significant object from an input video, divides the input video into a plurality of objects, and each object divided by the objectification unit. On the other hand, an encoding unit that generates a basic encoded stream, an extended encoded stream, and a rate distortion characteristic, and encoded data (hereinafter referred to as code) that is hierarchized in terms of object, quality, and time for each object. The first part is that it includes a hierarchizing unit that creates a transmission unit, and a transmission unit that distributes and transmits the coding units with priorities to a plurality of transmission paths based on the priorities. There are features.

  Also, the present invention integrates the encoding unit with a receiving unit that receives the encoding unit transmitted by the video transmission apparatus and synchronizes the encoding unit so that it can be decoded without time lag. An integration unit; a decoding unit that decodes the integrated encoded data to obtain a decoded object video; and an object integration unit that integrates a plurality of decoded object videos decoded by the decoding unit to obtain one decoded video. There is a second feature in that

  According to the present invention, transmission according to the bandwidth of a plurality of lines is possible, and it is possible to transmit video of optimum quality according to the user's reception situation.

  That is, a user having a larger number of transmission paths can view higher quality video. In addition, the user first receives and decodes only the encoding units in the order of priority, grasps the outline of the video, and if it is an interesting video, the remaining encoding units are also included as necessary. By receiving and decoding, the whole picture can be viewed with higher quality. In other words, step-by-step video viewing becomes possible.

  Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a basic configuration of a video transmission apparatus according to the present invention. Hereinafter, a case where video is distributed to a video receiver will be described as an example.

  First, an input video is input to the objectification unit 11. The input video is divided into a plurality of object videos Obj1, Obj2,..., ObjM (M is a positive integer) in which the part other than the object is set to the “0” level, for example. This is obtained by dividing the video area of the object from the frame video.

  A known technique can be used to divide the object video from the frame video. For example, the object extraction described in Japanese Patent Laid-Open No. 2003-44860 “Video Object Tracking Device”, Japanese Patent Laid-Open No. 2002-358526 “Video Object Detection / Tracking Device”, and Japanese Patent Laid-Open No. 2001-236512 “Moving Object Extraction Device” Technology can be used.

  Next, each object video is input to the encoding unit 12 to generate a basic encoded stream B, an extended encoded stream E, and rate distortion characteristics. Details of the rate distortion characteristics will be described later. Note that bit-plane coding is performed in the enhancement layer.

  Next, the hierarchizing unit 13 hierarchizes the encoded data of each object video with respect to (object / quality / time). The encoded data of each hierarchical object video is hereinafter referred to as “encoding unit”. At this time, rate distortion characteristics are used for hierarchization related to quality (for example, video quality). The encoding unit is given priority in consideration of human visual characteristics.

  Next, the transmission unit 14 distributes the encoding units to the transmission path 1, the transmission path 2,..., The transmission path N in order from the higher priority, and transmits the transmission unit 19 to the transmission path 19. At this time, the transmission path index numbers are assumed to be in ascending order in the order of transmission paths with the largest number of users. Also, using the syntax of MPEG-2 TS (Transport Stream), the time is described in the packet header and packetized to be transmitted.

  Next, the receiving unit 15 receives an MPEG-2 TS format packet, looks at time information described in the packet header, and synchronizes the encoding unit.

  Next, the integration unit 16 integrates the encoding units and obtains an extended encoded stream before hierarchization regarding quality.

  Next, the decoding unit 17 decodes the encoded data of each object to obtain a decoded object video.

  Finally, the object integration unit 18 integrates a plurality of decoded video objects to obtain a decoded video. As a result, the frame video can be viewed.

  A specific example of the processing of each block in FIG. 1 will be described below taking as an example the case where an MPEG-4 based FGS (Fine Granularity Scalability) encoding technique is used as the hierarchical encoding technique.

The FGS coding technique is well known as described in the following document. The FGS encoding technique is a hierarchical encoding technique that has a particularly high scalability (degree of freedom in decoding), and realizes this high scalability by a bit-plane encoding process.
Weiping Li, “Overviw of Fine Granularity Scalability in MPEG-4 Video Standard”, IEEE Transaction on circuit and systems for video technology, Vol.11, No.3, Mar.2001

Objectification unit 11
The input video is divided into a plurality of object videos Obj1, Obj2,..., ObjM (M is a positive integer) in which the part other than the object is set to “0” level, for example. The above-described well-known technique can be used for the objectification unit that divides the video into the object video.

  In the example of FIG. 1, the object video Obj1 is a car, Obj2 is a road, Obj3 is a row of trees, and the part other than the object is at a "0" level, for example.

Encoding unit 12
FIG. 2 is a block diagram showing details of the encoding unit 12. FIG. 2 shows a configuration for processing one of the object videos Obj 1, Obj 2,... A plurality of object videos Obj 1, Obj 2,..., Obj M are processed by providing a plurality of such configurations in parallel or using one configuration in time division. In the following description, it is assumed that the configuration of FIG. 2 processes an object video Obj that is representative of each object video.

The input object video Obj is sequentially input to the subtracter 31, the subtractor 32, and the motion detector 33 as an input object image Obj _Img frame by frame. The base layer encoding process is performed at a portion subsequent to the subtracter 31, and the enhancement layer encoding process is performed at a portion subsequent to the subtractor 32.

In the base layer encoding process, first, the prediction image obtained by the motion compensation prediction unit 34 is subtracted from the input object image Obj _Img by the subtractor 31 to generate a prediction error image. This prediction error image is converted into a DCT coefficient by a DCT (orthogonal transformation) unit 35, for example, in units of blocks of 8 vertical pixels and 8 horizontal pixels. The quantization unit 36 quantizes the DCT coefficient according to the quantization parameter. The quantized DCT coefficients are variable length encoded by a VLC (variable length encoding) unit 37 and multiplexed with the motion vector MV to create a basic encoded stream B _Obj .

  The quantized DCT coefficients from the quantization unit 36 are inversely quantized by the inverse quantization unit 38, the prediction error image is reproduced by the inverse DCT unit 39, and the predicted image from the motion compensated prediction unit 34 by the adder 40. Is added. The decoded image obtained by the adder 40 is stored in the frame memory 41.

The motion detection unit 33 detects the motion of the object video Obj based on the input object image Obj _Img sequentially input frame by frame from the object video Obj and the decoded image from the frame memory 41, and outputs a motion vector MV. The motion vector MV is given to the motion compensation prediction unit 34 and the variable length coding unit 37. The motion of the object video Obj is detected in units of macroblocks having 16 pixels vertically and 16 pixels horizontally, for example. The motion compensation prediction unit 34 generates a prediction image based on the locally decoded image from the frame memory 41 and the motion vector MV from the motion detection unit 33 and outputs the prediction image to the subtracter 31 and the adder 40.

What is to be encoded in the encoding process of the enhancement layer is a residual image, which is the difference between the output of the subtractor 32, that is, the input object image Obj _Img and the decoded image stored in the frame memory 41. This residual image is converted into DCT coefficients by the DCT unit 42. As a result, the generated DCT coefficients are encoded in bit plane units by the bit plane encoding unit 43, and further variable length encoded by the variable length encoding unit 44 to create an extended encoded stream E _Obj . In addition, the rate distortion characteristic RD _Obj is generated by the bit plane encoding unit 43 during this encoding.

  FIGS. 3A to 3D are explanatory diagrams of the encoding process procedure of the enhancement layer of the FGS encoding technique performed by the bit plane encoding unit 43. FIG. In the extended layer of the FGS encoding technique, as shown in FIG. 3A, DCT is performed on the residual image in units of blocks of 8 vertical images and 8 horizontal pixels to obtain DCT coefficients. Next, as shown in FIG. 5B, the DCT coefficients of each block are zigzag scanned to rearrange the DCT coefficients into a primary series. Further, as shown in (c) of FIG. Thereafter, as shown in FIG. 9D, the binary information symbols of each bit plane are encoded by binary zero-run length encoding processing (RUN, EOP) to generate a code word. Here, (x, 0) (x = 0, 1, 2,...) Code word indicates that the number of 0s up to 1 is x, and (x, 1) is the last 1 Indicates a code word (EOP).

  According to the above encoding processing procedure, each codeword is sequentially stored in the extended encoded stream in one pass from a codeword having a high degree of contribution to quality when decoded. That is, in FIG. 3D, (1, 1) (0, 0) (0, 0) (1, 0) (0, 1) (0, 0) (1, 0) (2, 0).・ ・ In order.

  In this way, in the hierarchical encoding technique in which each codeword is stored in the encoded stream in the order of one pass from the one that contributes to the quality when decoded, the code amount R of the codeword to be decoded and The coding distortion D can be measured. Therefore, the encoding distortion D with respect to the code amount R can be known in codeword units.

  Here, the coding distortion D indicates the difference between the decoded image and the original original image when the decoding process is terminated after the coding, and the coding distortion D decreases as more codewords are decoded. For example, in the example of the FGS encoding technique, when the image quality is objectively defined, a mean square error MSE (Mean Squared Error) with the original image can be used as the encoding distortion D.

  In the present invention, in order to perform video transmission that maximizes the quality of the decoded video, the rate distortion characteristic RD that is the relationship between the code amount R of the decoded codeword and the coding distortion D is measured for each object video Obj. To do.

That is, when the encoding of the object video Obj is terminated at the j-th codeword, the total code amount is R _j, and when the difference between the decoded object image and the original object image is D _j , the object video In the hierarchical encoding process of Obj, the code amount R _j and the encoding distortion D _j are measured every time one code word is created.

This is plotted in a graph in which the horizontal axis is the code amount R and the vertical axis is the coding distortion D as shown in FIG. This is performed for each object video Obj 1, Obj 2,..., Obj M, and the respective rate distortion characteristics RD _{Obj 1} , RD _{Obj 2 ,.}

In the graph of the rate distortion characteristics RD _{Obj 1} , RD _{Obj 2} ,..., RD _{Obj M} , a combination in which the coding distortion D monotonously decreases as the code amount R increases is selected and plotted on the graph. As described above, the encoding unit 12 performs basic encoding streams B _{Obj 1} , B _{Obj 2} ,..., B _{Obj M on} each object video Obj 1, Obj 2 _,. E _{Obj 1} , E _{Obj 2} ,..., E _{Obj M} are generated, and rate distortion characteristics RD _{Obj 1} , RD _{Obj 2 ,.}

Hierarchy unit 13
The method of realizing the hierarchization for each of (object / quality / time) is as follows.

(1) Object Create one hierarchy for each object. That is, the object can be hierarchized by the same number as the number of object videos.

(2) Quality The extended encoded stream can be further divided into hierarchies. for that reason,
The rate distortion characteristic RD _Obj is used to determine the distribution of the code amount between the objects so as not to exceed a given code amount and to maximize the quality of the decoded video obtained by integrating a plurality of decoded object videos. .

  The extended encoded stream is cut based on the code amount assigned to each object. Hereinafter, description will be given assuming that the extended encoded stream is hierarchized into the K layer.

(3) Time Three types of layers are generated for time. For example, different layers are used for I, P, and B frame types.

  As described above, the hierarchizing unit 13 hierarchizes the encoded data into (M · K · 3) types with respect to (object / quality / time). That is, the encoding unit is encoded data hierarchized into a total of M × K × 3 types. At this time, rate distortion characteristics are used for hierarchization related to quality.

Hereinafter, the hierarchization regarding the quality using the rate distortion characteristic will be described in detail.
FIG. 5 is a block diagram showing a configuration example of the hierarchizing unit 13a regarding quality using rate distortion characteristics in the hierarchizing unit 13. As shown in FIG. The hierarchization unit 13a includes a code amount distribution unit and a hierarchization unit (hereinafter referred to as a code amount distribution / hierarchization unit 61). The code amount distribution / hierarchization unit 61 is generated by the encoding unit 12. Extended encoded streams E _Obj (= E _{Obj 1} , E _{Obj 2} ,..., E _{Obj M} ), rate distortion characteristics RD _Obj (= RD _{Obj 1} , RD _{Obj 2} ,..., RD _{Obj M} ) and Setting code amount for further layering the extended encoded stream into a plurality of hierarchies (hereinafter, setting code amount for subdividing the K layer into sub-layers) T (= T [1], T [2]... , T [K]).

The code amount distribution / hierarchization unit 61 sets the total code amount assigned to each object video Obj 1, Obj 2,..., Obj M to the set code amount T ( = T [1], T [2]..., T [K]) rate distortion characteristics RD _{Obj 1} , RD _{Obj 2 ,. Obj 1} , E _{Obj 2} ,..., E _{Obj M} are further subdivided into K hierarchies, and the divided first extended encoded stream E [1] (= E [1] _{Obj 1} , E [1] _{Obj 2} ,..., E [1] _{Obj M} ), the second extended encoded stream E [2] (= E [2] _{Obj 1} , E [2] _{Obj 2} ,..., E [2] _{Obj M} ),..., Divided Kth extension code Stream E [K] (= E [K] _{Obj 1} , E [K] _{Obj 2} ,..., E [K] _ObjM ) is output.

The extended encoded streams E [1], E [2],..., E [K] layered in the K layer are the basic encoded streams B (= B _{Obj 1} , B _{Obj 2,.} , B _{Obj M} ) and output to the transmitter 14.

As will be described in detail below, the code amount distribution / hierarchization unit 61 transmits a plurality of transmission paths 19 between the video receivers, that is, transmission paths 1, transmission paths 2,... A code amount distribution that maximizes the quality of decoded video is determined within a range of possible code amounts. Specifically, rate distortion characteristics RD (= RD _{Obj 1} , RD _{Obj 2} ,..., RD _{Obj M} ) and enhancement layer set code amount T (= T [1], T [2]... T [K]) is used to calculate a code amount to be assigned to each object video Obj 1, Obj 2,..., Obj M, for example, by the algorithm shown below.

(Process 1) In order to maximize the quality of the decoded object video Obj ′ 1, Obj ′ 2,..., Obj ′ M, the rate of each object video Obj 1, Obj 2,. Using the distortion characteristic RD _Obj (= RD _{Obj 1} , RD _{Obj 2} ,..., RD _{Obj M} ), R that minimizes the next evaluation function F needs to be determined by the following equation (1). .

Here, s _Obj indicates the index number of the sth code word of the object video Obj. That is, R _Obj [s _Obj ] means the amount of code when it is cut off at the sth code word and adopted so far, and D _Obj [s _Obj ] means coding distortion.

λ is a constant, and with respect to a certain constant λ, the code amount R (λ) (= R _{Obj 1} [s _{Obj 1} (λ)], R _{Obj 2} [s _{Obj 2} (λ )],..., R _{Obj M} [s _{Obj M} (λ)]) and encoding distortion D (λ) (= D _{Obj 1} [s _{Obj 1} (λ)], D _{Obj 2} [s _{Obj 2} (λ) )],..., D _{Obj M} [s _{Obj M} (λ)]) is uniquely determined.

The constant λ corresponds to the tangential slope dD _Obj [s _Obj (λ)] / dR _Obj [s _Obj (λ)] in the rate distortion characteristic curve. For a certain value of λ, minimizing the evaluation function F results in the same slope λ for the rate distortion characteristic curves for all object images Obj 1, Obj 2,..., Obj M. Equivalent to finding the tangent it has.

When tangent lines having the same slope λ are obtained with respect to the curves of the coding distortion characteristics for all object videos Obj 1, Obj 2,..., Obj M, the code word at the contact is s _Obj (λ) -th. If the code amount is R (λ) (= R _Obj1 [s _{Obj 1} (λ)], R _{Obj 2} [s _{Obj 2} (λ)],..., R _{Obj M} [ s _{Obj M} (λ)]) and coding distortion D (λ) (= D _{Obj 1} [s _{Obj 1} (λ)], D _{Obj 2} [s _{Obj 2} (λ)],..., D _{Obj M} [ s _{Obj M} (λ)]) is a combination that minimizes the evaluation function F.

In this way, for all the object videos Obj 1, Obj 2,..., Obj M, the code amount R (λ) (= R _Obj1 [s _Obj ) that minimizes the evaluation function F with respect to a certain value of λ. ₁ (λ)], R _{Obj 2} [s _{Obj 2} (λ)],..., R _{Obj M} [s _{Obj M} (λ)]) and coding distortion D (λ) (= D _{Obj 1} [s _{Obj 1} (λ)], D _{Obj 2} [s _{Obj 2} (λ)],..., D _{Obj M} [s _{Obj M} (λ)]).

  (Process 2) Since a plurality of candidates are obtained for λ, the total code amount allocated to each object video Obj 1, Obj 2,..., Obj M is allocated to the extended encoded stream in the entire frame screen. In order not to exceed the set code amount T (= T [1], T [2]..., T [K]), a desired set code amount T (= T [1], T [2]..., T [K]) are selected from constants λ [1], λ [2],.

.., Obj M divided first extended encoded stream E [1] (= E [1] _{Obj 1} , E [1] _{Obj 2} ,..., E [ 1] Code amount R (λ [1]) assigned to _{Obj M} ) (= R _Obj1 [s _{Obj 1} (λ [1])], R _{Obj 2} [s _{Obj 2} (λ [1])], If the total amount of R _{Obj M} [s _{Obj M} (λ [1])]) is R [1], the following equation (2) is satisfied and the code amount of the entire enhancement layer is as much as the set code amount A constant λ [1] is determined so as to be close.

  At this time, if the extended encoded stream is terminated by the s (λ) th codeword corresponding to the constant λ, the video quality of the entire decoded object video Obj ′ 1, Obj ′ 2,..., Obj ′ M is maximized. can do.

  Schematizing the above algorithm can be represented as described in FIGS.

  In step S1, candidate points for λ are obtained from the rate distortion information. This process corresponds to process 1 described above. In step S2, the placement number j representing the hierarchy is set to 1, and in step S3, the constraint condition is imposed, and the corresponding tangent slope λ [1] is determined from the set code amount T [1]. In step S4, it is determined whether or not j <K is satisfied. If j is smaller than K, the process proceeds to step S5 and 1 is added to j. In step S3, the constraint condition is imposed, and the set code amount is set. The slope λ [2] of the corresponding tangent is determined from T [2]. The above process is repeated until the determination in step S4 is negative. When the determination in step S4 is negative, tangential slopes λ [1] to λ [K] obtained by imposing the constraint conditions are obtained.

In step S6, the end positions S _{Obj1 (λ [1])} , S _{Obj2 (λ [1]) of} the coding path corresponding to λ [1], λ [2] _,. S _{ObjM (λ [1]} , S _{Obj1 (λ [2])} , S _{Obj2 (λ [2])} , ..., S _{ObjM (λ [2]} , ..., S _{Obj1 (λ [ K]), S Obj2 (λ} [K]), ···, separated by _{S ObjM (λ [K],} the extended coded stream _{E Obj 1, E Obj 2,} ···, E Obj M hierarchy The process of steps S2 to S6 corresponds to the process 2.

FIG. 8 shows a diagram of the extension layer encoded stream of the object videos Obj 1, Obj 2,..., Obj M obtained by the above processing. That is, the divided first encoded stream E [1] (= E [1] _{Obj 1} , E of each object video Obj 1, Obj 2,..., Obj M with respect to the set code amount T [1]. [1] _{Obj 2} ,..., E [1] _{Obj M} ), and the divided second of each object video Obj 1, Obj 2,..., Obj M with respect to the set code amount T [2] Extended encoded streams E [2] (= E [2] _{Obj 1} , E [2] _{Obj 2} ,..., E [2] _{Obj M} ) are allocated. Also, the divided K-th extended encoded stream E [K] (= E [K] _{Obj 1} , E of each object video Obj 1, Obj 2,..., Obj M with respect to the set code amount T [K]. [K] _{Obj 2} ,..., E [K] _{Obj M} ). B indicates a basic encoded stream.

FIGS. 9A and 9B are explanatory diagrams showing the concept of the code amount distribution process when there are two object videos. As shown in FIG. 5A, tangent lines having the same slope λ are obtained for the encoding distortion characteristics RD _{Obj 1} and RD _{Obj 2} for the object videos Obj 1 and Obj 2, and the code amount R _{Obj at the} contact point is obtained. ₁ [s _{Obj 1} (λ)] and R _{Obj 2} [s _{Obj 1} (λ)] are obtained. The constant λ is changed so that the total code amount R _{Obj 1} [s _{Obj 1} (λ)] + R _{Obj 2} [s _{Obj 2} (λ)] is smaller than and close to the set code amount T [1]. The code amounts R _{Obj 1} [s _{Obj 1} (λ)] and R _{Obj 2} [s _{Obj 2} (λ)] at that time are obtained. It is assumed that they are s _{Obj 1} (λ [1]) and s _{Obj 2} (λ [1]) as shown in FIG.

When the extended encoded stream is hierarchized in terms of quality and an encoding unit is created, the stream cutter 63 (1) converts the extended encoded stream E _{Obj 1} of the object video _{Obj 1} to the s _{Obj 1} (λ [1]) th _Is cut off after _signing to obtain E [1] _{Obj 1} . Further, the stream cutter 63 (2) _cuts the extended encoded stream E _{Obj 2} of the object video E _{Obj 2} after the s _{Obj 2} (λ [1])-th code to obtain E [1] _{Obj 2} . In this way, the first divided encoded stream E [1] (= E [1] _{Obj 1} , E [1] _{Obj 2} ) is obtained.

  In summary, as shown in FIG. 10, the encoding unit, that is, the hierarchically encoded data, includes three objects for automobiles, roads, and tree-lined objects, and the basic hierarchy and extension for quality. The layer and the extension layer are formed from data divided into the K layer according to the set code amount and the time divided into three layers of I, P and B pictures.

  Next, the hierarchizing unit 13 gives priority to the encoding unit in consideration of human visual characteristics. This priority assignment will be described with reference to FIG.

  FIG. 11 shows a test video Stefan recorded with a tennis game divided into three object videos of (1) player (2) ball (3) background audience. Regarding the quality, the hierarchization of the extension hierarchy is not performed, the single hierarchy K = 1, and only the basic and extension two hierarchies. Time was divided into three levels of I, P, and B.

  Therefore, 3 × 2 × 2 = 18 coding units are created with respect to (object / quality / time). When priorities are given to the encoding units in consideration of human visual characteristics, priorities such as the encoding units (1) to (18) in FIG. 11 are given as an example.

  This priority expresses the upper axis among three orthogonal axes as quality, the horizontal axis as time, and the depth axis as an object. High priority from the viewpoint of visual characteristics. From the viewpoint of the visual characteristics, if an object is a player> ball> background audience, a time is I> P> B, and a quality is basic> extended, the encoding unit (1) having a high priority is ( (Player, basic hierarchy, I), (2) (player, basic hierarchy, P), (3) (ball, basic hierarchy, I), (4) (player, basic hierarchy, B), ... (18) becomes (background audience, extended hierarchy, B).

Transmitter 14
The transmission unit 14 assigns the encoding units to the transmission path 1, the transmission path 2,..., And the transmission path N in order from the encoding unit with the highest priority, and transmits them to the transmission path. At this time, the transmission path index numbers are assumed to be in ascending order in the order of transmission paths with the largest number of users.

In addition, using the syntax of MPEG-2 TS, the time is described in the packet header and packetized to be transmitted. Bandwidth C _1, C 2 of the transmission path _19, ..., so as not to exceed the respective C _N, a combination of code units transmitted over a respective transmission path is determined in order of priority, and transmits.

In the test video Stefan shown in FIG. 11, for example, a user who has only a single transmission line 1 and can only receive the encoding unit (1) sends a player object frame-by-frame with a low quality (only basic gradation) (I Because only the frame can be obtained). In addition, a user who has a plurality of transmission paths and can receive the encoding units (1) to (9) includes all players, balls, and background audiences, and has a smooth frame rate composed of I, P, and B pictures. Low quality (basic gradation only) video can be played.
Furthermore, the user who has more transmission paths and can receive the encoding units (1) to (18) has a smooth frame rate including I / P / B pictures including all players, balls, and background audiences. High quality (basic gradation, extended hierarchy) video can be played.

Receiver 15
The receiving unit 15 receives an MPEG-2 TS format packet, looks at time information described in the packet header, and synchronizes the encoding unit. Also, the packetized data is recombined and returned to the encoding unit format.

Integration unit 16
The integration unit 16 integrates a plurality of encoding units that are hierarchized with respect to quality, and obtains an extended encoded stream that is not hierarchized with respect to quality.

Decoding unit 17
FIG. 12 is a block diagram illustrating a configuration example of the decoding unit 17. FIG. 12 shows a configuration for processing one of the decrypted object videos Obj ′ 1, Obj ′ 2,..., Obj ′ M. A plurality of decoding object videos Obj ′ 1, Obj ′ 2,..., Obj ′ M are decoded in parallel by providing a plurality of such configurations, or using one configuration in time division. In the following description, it is assumed that the configuration of FIG. 12 processes the object video Obj ′.

  The basic encoded stream B ′ is subjected to variable length decoding by a VLD (variable length decoding) unit 81, and quantized DCT coefficients are output. At the same time, a motion vector MV is output. The quantized DCT coefficient is inversely quantized by the inverse quantization unit 82 and inversely converted into an error image by the inverse DCT unit 83. The adder 84 adds the error image and the prediction image from the motion compensation prediction unit 85, and outputs a base layer decoded object image. The base layer decoded object image is output to the adder 90 and stored in the frame memory 86. The decoded image stored in the frame memory 86 is given to the motion compensation prediction unit 85.

The extension encoded stream E ′ is variable-length decoded by the variable-length decoding unit 87 and decoded by the bit-plane decoding unit 88 into DCT coefficients of the residual image. The DCT coefficient of this residual image is inversely converted into a residual image by the inverse DCT unit 89. The adder 90 adds the decoded object image of the residual image and the base layer, and outputs the decoded object images Obj _'Img. The decoded object image Obj ′ _Img is sequentially output frame by frame to obtain a decoded object video.

Object integration unit 18
For a plurality of decoded object videos Obj 1, Obj 2,... Obj M, when a part other than the object is defined as a brightness level “0”, for example, only areas having other brightness levels are integrated. Decoded video can be obtained.

  As is apparent from the above description, according to the present invention, a user having a larger number of transmission paths can view a higher quality video.

It is a block diagram which shows the basic composition of the video transmission apparatus which concerns on this invention. It is a block diagram which shows the structural example of the encoding part of FIG. It is explanatory drawing of the encoding process procedure of the extended hierarchy of FGS encoding technique. It is explanatory drawing of a rate distortion characteristic. It is a block diagram which shows the structural example of the hierarchization process regarding the quality of the hierarchization part of FIG. It is a flowchart which shows the code amount distribution process. It is a flowchart which shows the process of hierarchization. It is explanatory drawing of the hierarchization process regarding quality. It is explanatory drawing which shows the concept of the code amount allocation process between object videos. It is explanatory drawing of an encoding unit. It is explanatory drawing of the process example which provides the priority which considered the human visual characteristic to the encoding unit. FIG. 2 is a block diagram illustrating a configuration example of a decoding unit (using MPEG-4 based FGS encoding technology) in FIG. 1.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 ... Objectification part, 12 ... Encoding part, 13 ... Hierarchization part, 14 ... Transmission part, 15 ... Reception part, 16 ... Integration part, 17 ... Decoding , 18 ... Object integration part, 19 ... Transmission path, 43 ... Bit plane encoding part, 61 ... Code amount distribution / hierarchization part.

Claims (7)

An objectification unit that extracts a significant object from the input video and divides the input video into a plurality of objects;
An encoding unit that generates a basic encoded stream, an extended encoded stream, and a rate distortion characteristic for each object divided by the objectifying unit;
For each object, a hierarchization unit for creating encoded data (hereinafter referred to as an encoding unit) hierarchized with respect to the object, quality, and time;
A video transmission apparatus comprising: a transmission unit that distributes and transmits a coding unit with a priority to a plurality of transmission paths based on the priority. The video transmission apparatus according to claim 1,
The object converting section sets a video portion other than the significant object to 0 in luminance level. The video transmission device according to claim 1 or 2,
The hierarchization unit allocates a code amount among objects so as to optimize the rate distortion characteristic with respect to a set code amount for hierarchization related to quality for the hierarchization related to the quality, and assigns the code to each object. A video transmission device. The video transmission apparatus according to any one of claims 1 to 3,
The video transmission apparatus according to claim 1, wherein the hierarchization relating to the quality is performed on the extended encoded stream. The video transmission apparatus according to any one of claims 1 to 4,
The video transmission apparatus according to claim 1, wherein the transmission unit distributes and transmits encoding units with priorities in consideration of visual characteristics to a plurality of transmission paths based on the priorities. A receiving unit that receives the encoding unit transmitted by the video transmission device according to any one of claims 1 to 5 and synchronizes the encoding unit so that it can be decoded without time lag,
An integration unit for integrating the encoding units;
A decoding unit for decoding the integrated encoded data and obtaining a decoded object video;
A receiving and reproducing apparatus comprising: an object integration unit that integrates a plurality of decoded object videos decoded by the decoding unit to obtain one decoded video. The reception / playback apparatus according to claim 6,
The reception / playback apparatus, wherein the integration unit integrates an extended encoded stream of an object layered into a plurality of layers with respect to quality into one extended encoded stream.

Images