ES2347385T3 - MOVING IMAGE FLOW GENERATION DEVICE, MOVING IMAGE CODING DEVICE, MOVING IMAGE MULTIPLEXING DEVICE AND MOVING IMAGE DECODING DEVICE. - Google Patents

Abstract

To provide a moving picture stream generation apparatus and the like that generates a moving picture stream that can be played back in trick-play such as variable-speed playback and reverse playback even in a coding format such as the MPEG-4 AVC in which flexible prediction structures are allowed. A moving picture stream generation apparatus includes: a trick-play information generation unit TricPlay for generating supplemental information on a random access unit basis, the supplemental information being referred to at the time of playback of each random access unit and each random access unit including one or more pictures; and a variable length coding unit VLC for generating a stream including the generated supplemental information and the one or more pictures by adding the supplemental information to each corresponding random access unit. At the top of each random access unit, an intra coded picture that can be decoded without depending on any picture is placed, and the supplemental information includes information for specifying pictures to be decoded at the time when pictures included in each random access unit are played back in trick-play.

Description

Image flow generation device in movement, motion picture coding apparatus, multiplexing apparatus for moving images and apparatus for decoding of moving images.

Technical field

The present invention relates to an apparatus, and similar, which generates a flow of moving images encoded, especially to an apparatus, and the like, which generates a flow over which trick play can be performed such as skip playback, variable speed playback, reverse reproduction and the like.

Prior art

Recently, the era of the multimedia in which sound, images and other values of pixels are integrated into a medium, and the information media conventional, such as communication tools such as Newspapers, magazines, television, radio and telephone are considered as multimedia objects. Generally, multimedia is a simultaneous representation not only of characters but also of graphics, sound and especially images. With the purpose of manage conventional media previously described as multimedia, it is a requirement to represent the information digitally.

However, it is not realistic to process directly a lot of information digitally using conventional media previously described because, when calculating the amount of data for each information medium described above as amount of data digital, the amount of data per character is 1 to 2 octets, while the sound per second is not less than 65 kbits (telephone voice quality) and that of moving images by second is not less than 100 Mbits (current reception quality of television). For example, a commercially available telephone with television, thanks to the Digital Services Network Integrated (ISDN) with a transmission rate of 64 kbps to 1.5 Mbps, but it is impossible to transmit mobile camera images from television when using ISDN.

For this reason the technique of information compression. For example, a technical standard of Compression of moving images of H. 261 or H. 263, recommended for the International Union of Telecommunications-Sector Standardization of Telecommunications (ITU-T) is used for telephones with television Likewise, with the compression technique of the MPEG-1 standard information, it is possible to store  image information, along with sound information, on a CD (compact disc) normal for music.

Here, the Group of Image Experts in Movement (MPEG) is an international standard for compressing digitally motion picture signals, and has been standardized by ISO / IEC (the International Organization of Standardization / International Engineering Consortium). MPEG-1 is the norm for compressing signals from motion pictures at 1.5 Mbps, that is, to compress television signal information in approximately one hundredth Similarly, the quality that meets the standard MPEG-1 is the average level that can be performed at a transmission speed of approximately 1.5 Mbps. Of this MPEG-2 mode is normalized to meet the need for higher image quality, and compress the signals of moving images from 2 to 15 Mbps. Currently, the group of work (ISO / IEC JTC1 / SC29 / WG11), which normalized MPEG-1 and MPEG-2, has normalized MPEG-4 with higher compression speed. The MPEG-4 (i) standard achieves a speed of Compression higher than the MPEG-1 standard and the standard MPEG-2, (ii) allows coding, decoding and perform operations object by object, and (iii) perform new necessary functions in this era of multimedia. The initial object of the MPEG-4 standard is to standardize a procedure encoding images with low bit rates, but the object extends to a purpose coding procedure general interlaced images with high bit rates. After this, ISO / IEC and ITU / T, in combination, have normalized the AVC (advanced video encoding) of MPEG-4 as an image coding procedure of the next generation of images with a high compression speed. Be expected to be used for disc-related devices next generation optics, or in broadcast for terminals mobile phones

Generally, in the coding of images mobile, the amount of information is compressed reducing temporal and spatial redundancies. In the coding of inter-image prediction, aimed at reducing redundancies temporal, image generation estimation and prediction of movement is carried out block by block with reference to a previous image or a subsequent image, and the coding takes out on the differential value between the prediction image obtained and the image to be encoded. In this memory, the term "Image" used is a term that represents an image. In a progressive image, an image means a plot, but in a interlaced image, means a plot or a field. The term "interlaced image" described here means a plot composed of two fields with a slight time delay. In the coding and decoding procedures intertwined, it is possible to process a frame as it is, as two fields, or frame by frame or field by field of each block in a plot.

The image to carry out the coding intra-prediction without reference to any reference image is called Image Intra-encoded (image I). Likewise, the image to carry out inter-prediction coding which refers to only one image is called predictive image encoded (image P). Likewise, the image to carry out the inter-prediction coding that refers to two reference images simultaneously called Image Bipredictive encoded (image B). An image B can refer to two images selected as an arbitrary combination of a previous image and a subsequent image at display time. Such two reference images can be specified on block per block, the block being a basic coding unit and decoding. These reference images distinguish ones of the others as follows: the reference image described above in the coded bit stream is called first reference image, and the other reference image, described later, it is called second reference image. Note that such reference images must have been encoded or previously decoded to encode or decode P images and B images.

Coding inter-prediction of motion compensation is use to encode P images and B images. Coding intra-prediction of motion compensation is a intra-prediction coding procedure in which motion compensation applies. Compensation of movement is a procedure to improve an accuracy of prediction and reduce the amount of data by evaluating the amount of movement (hereinafter referred to as motion vector) of each block of an image and carrying out the coding of prediction that considers the motion vector. For example, the amount of data is reduced by evaluating the motion vectors of the images to be encoded and encoding each prediction residue between each prediction value that is shifted by the amount of each motion vector and each current image to be encoded. In the case of this procedure, since the vector information of movement is necessary in decoding, the vectors of Motion is also encoded, and recorded or transmitted.

Motion vectors are evaluated macroblock by macroblock. More specifically, the motion vectors by setting the macroblock of an image to encode, moving the macroblock of a reference image within the search range, and finding the location of the reference block that is closer to the standard block.

Fig. 1A and 1B are, respectively, diagrams Structural flows of conventional MPEG-2 flows. How shown in Fig. 1B, an MPEG-2 stream has a hierarchical structure like the one that will be described hereinafter. A flow consists of a Group of Images (called GOP in what successive). The use of a GOP as a basic unit in processing Encoding allows you to edit a moving image or carry Make a random access. A GOP consists of images I, images P and B images. A flow, a GOP and an image also include a synchronous signal (sync) indicating a border of units and a heading indicating the common data in the units, being the units here a flow, a GOP and an image, respectively.

Fig. 2A and 2B show respectively examples that indicate how to perform the inter-image prediction coding used in MPEG-2. The diagonally shaded images in the figure are the images to which other images are to refer. As shown in Fig. 2A, in the prediction encoding in MPEG-2, images P (P0, P6, P9, P12 and P15) can only refer to a single image selected as an image I or an image P immediately later in the display time. Similarly, images B (B1, B2, B4, B5, B7, B8, B10, B11, B13, B14, B16, B17, B19 and B20) can refer to two selected images as a combination of an image I or image P immediately preceding and an image I or image P immediately preceding. In addition, the order of the images to be placed in a flow is determined. The images I and an image P are placed in the order of the display time, and each image B is placed immediately after an image I to be displayed immediately after the image B or immediately after an image P. As a structural example of a GOP, as shown in Fig. 2B, images 13 through B14 are grouped into a single
GOP

Fig. 3A is a structural diagram of a AVC flow of MPEG-4. There is no concept equivalent to a GOP in the MPEG-4 AVC. Without However, since it is possible to build an accessible unit randomly equivalent to a GOP, segmenting data on the basis of a special image that can be decoded without relying on of other images, the unit will be referred to hereinafter as RAU (Random Access Unit). In other words, a unit of random access RAU is a group of encoded images that starts with an intracoded image that can be decoded without depend on any image.

Next, the access unit which is a basic unit in the management of a flow (simply called AU hereinafter). An AU is the unit for storing encoded data equivalent to an image, and includes a PS set of parameters, segment data and Similar. There are two types of PS sets of parameters. One of they are a PPS set of image parameters (named in hereinafter simply PPS) which are data equivalent to heading of each image. The other is an SPS set of sequence parameters (hereinafter simply called SPS) which is equivalent to the heading included in a unit of a GOP or more on the MPEG-2. An SPS includes the number Maximum reference images, one image size and the like. On the other hand, a PPS includes a type of length coding variable, an initial value of the quantization stage, the number of reference images and the like. Each image is assigned a identifier that indicates which of the PPS and SPS above described reference is made. Similarly, a frame FN number, which is the identification number to identify an image included in segment data. Note that a sequence begins with a special image in which all the necessary states to decode they restart as described later, and which is made up of a group of images that starts with an image special and ends with an image that is placed immediately before the next special image.

There are two types of I images in the AVC of MPEG-4 There is a Decoder Refreshment Instant (IDR) and the rest. An IDR image is the image I that you can decode all the images placed after the IDR image in a decoding order, without referring to the images placed before the IDR image in the order of decoding; in other words, it is the image I in which Restart the necessary states for decoding. A IDR image corresponds to the upper image I of a closed GOP of MPEG-2 A sequence in the AVC of MPEG-4 starts with an IDR image. In the case of a image I that is not an IDR image, an image placed after image I in the decoding order can refer to a image placed before image I in the order of decoding. The respective image types will be defined in hereafter. An IDR image and an image I are the images that are they consist only of segments I. An image P is the image that it can be composed of segments P and segments I. An image B is the image that can be composed of segments B, segments P and segments I. Note that the segments of an IDR image are stored in a NAL unit whose type is different from that of the unit NAL where the segments of an image are stored non-IDR. Here, a NAL unit is a unit of subimage.

In an AU in the MPEG-4 AVC, not only can the necessary data be included for decode but also the supplementary information and the border information. Such supplementary information is called Supplemental Enhancer Information (SEI) and is unnecessary for Decode segment data. All data such as a PS set of parameters, segment data and an SEI, it stored in a unit Network Abstraction Layer (NAL), that is, the NALU. A NAL unit consists of a header and a load Useful. A header includes a field that indicates the type of data to be stored (hereinafter referred to as the NAL unit type). The values of the NAL unit types are defined respectively for data types such as a segment or an SEI. The reference to such value of a type of NAL unit allows to identify the type of data to be stored in the NAL unit. The heading of A NAL unit includes a field called nal_ref_idc. It defines that a nal_ref_idc field is a 2-bit field and adopts a value of 0, 1 or more, depending on the types of NAL units. For example, the unit NAL of an SPS or PPS adopts the value 1 or more. In the case of NAL unit of a segment, a segment to which the others refer segments adopts the value 1 or more, while the segment to which no reference is made adopts the value 0. Similarly, the NAL unit of an SEI always takes the value 0.

One or more SEI messages can be stored in the NAL unit of an SEI. An SEI message consists of a header and payload, and the type of information to store in the payload is identified by the type of a SEI message indicated in the header. The decoding of a AU means decoding segment data in an AU, and visualizing an AU means displaying the result of decoding of segment data in the AU in what follow.

Here, since a NAL unit does not include information to identify a NAL unit border, it is possible add border information to the top of each unit NAL at the time of storing a NAL unit as an AU. In the management of an MPEG-4 AVC flow in a Flow of MPEG-2 Transport (TS) or a Program Flow (PS) of MPEG-2, a start code prefix, shown as 3 octets of 0x000001, is added to the top of a NAL unit. It is also defined that a NAL unit that indicates a border of AU must be inserted at the top of an AU on a TS or PS of MPEG-2, such as an AU called Access Unit Delimiter.

Conventionally, several have been proposed types of related techniques, like this one, with the coding of moving images (For example, refer to the Document of Patent 1).

Patent Document 1: Japanese Publication of Patent Open to Public Inspection Nº 2003-18549.

Fig. 4 is a block diagram of a conventional apparatus for encoding moving images.

The image coding apparatus 1 in movement is a device that produces a coded flow Str obtained by converting, through compression coding, a signal InputV of input video, to be introduced in a stream of bits of a coded stream of variable length, or the like. He motion picture coding apparatus includes a PTYPE prediction structure determination unit, a ME unit of motion vector estimation, an MC unit of motion compensation, one unit Subtract from subtraction, one unit Orthogonal transform T, a quantization unit Q, a IQ unit of inverse quantization, an IT unit of transform inverse orthogonal, a sum sum unit, an image memory MemImag, a switch and a VLC unit for length coding variable.

The V input signal of the input video is enter in the subtraction unit Subtract and the estimation unit ME of motion vector. The Subtraction Subtraction unit calculates the value differential between the input signal V input video input and the prediction image, and sends it to the transform unit orthogonal. The orthogonal transform unit T converts the value differential in a frequency coefficient, and sends it to the Q unit of quantization. The quantization unit Q carries out the quantization on the frequency coefficient introduced, and sends a quantization Qcoef value to the coding unit of variable length

The inverse quantization IQ unit performs the inverse quantization over the quantification Qcoef value for rebuild the frequency coefficient, and send it to the IT unit of reverse orthogonal transform. The transformed IT unit inverse orthogonal performs the inverse frequency transform to transform the frequency coefficient into a value differential of pixels, and sends it to the unit Sum of sum. The Sum sum unit adds the differential value of pixels to the prediction image, to be sent from the MC compensation unit motion, to make a decoded image. The commutator SW turns on when image storage is instructed decoded, and the decoded image is stored in memory Image MemImag.

On the other hand, the ME unit for estimating motion vector, in which a V input signal is input of macroblock input video by macroblock, look for the image decoded stored in MemImag image memory, and estimates the image area that is closest to the image signal of input, and consequently determines the motion vector MV that Indicates the position. The motion vector estimate is carried out block by block, the block being a segmented part of a macroblock Since various images can be used as reference images at this time, the numbers of identification to specify images to refer to (relative indexes) are necessary block by block. it's possible specify reference images by calculating image numbers indicated by relative indices, such image numbers being assigned to the respective images in an image memory MemImag

The MC motion compensation unit select the image area that is optimal as the prediction image  from decoded images stored in memory Image MemImag.

The PTYPE structure determination unit of prediction instructs the unit ME of vector estimation of movement and to the MC motion compensation unit for carry out intra-image coding on the objective image, as a randomly accessible special image using its Type Type of image, in the case where an image of Start of random access unit, RARA Input, indicates that the RAU random access unit starts with the current image, and instructs the variable length coding VLC unit to Encode the Type I type of image.

The VLC unit of length coding variable performs variable length coding on the Qcoef quantization value, relative index Index, type TypeI of image and motion vector MV to form a flow Coded Flow.

Fig. 5 is a block diagram of a conventional apparatus 2 for decoding moving images.  This decoding device 2 for moving images includes a variable length decoding VLC unit, a MemImag image memory, an MC compensation unit movement, a sum unit of sum, an IT unit of transform Inverse orthogonal and an inverse quantization IQ unit. Observe yourself  that, in the figure, to these processing units that perform the same operations as the processing units in a conventional motion image coding apparatus, as shown in the block diagram of Fig. 4, they are assign the same reference numbers, and the descriptions about them.

The length decoding VLD unit variable decodes a stream encoded Flow, and sends the value Qcoef quantization, the relative index Index, the TypeI type of MV image and motion vector. The Qcoef value of quantization, the relative index Index and the motion vector MV The MC unit of the motion compensation and the IQ unit of inverse quantization, respectively, and then the processing is carried out decoding on them. Such operations of a conventional motion image coding device already described using the block diagram of Fig. 4.

A random access RAU unit shows that decoding can be carried out starting with AU upper in the random access unit. However, since A conventional AVC flow of MPEG-4 allows very flexible prediction structures, an apparatus of storage with an optical disk or a hard disk cannot obtain information to determine the AUs to decode or display at the time of variable speed playback or reverse reproduction.

Figs. 6A and 6B are examples of structures. of prediction of the AU. Here, an image is stored in each AU. Fig. 6A is the prediction structure of the AUs used in a MPEG-2 flow. The diagonally shaded figures in the figure are reference images for other AUs. At MPEG-2 the AU of images P (P4 and P7) can carry out prediction coding by referring only to a single AU selected as the AU of an I image or P image immediately later in the display time. Equally, the AU of images B (B1, B2, B3, B5 and B6) can carry out prediction coding referring to two selected AUs as a combination of the AUs of an image I or image P immediately after and an image I or image P immediately previous in the display time. In addition, the order of Images to be placed in a stream are predetermined as follows way: the AUs of an image I and the images P are placed in the order of display time; and each of the AU images B is placed immediately after the AU of image I or a of the P images that are placed immediately after the AU of each image B. Consequently, decoding can be carried out in the following three ways: (1) all images are decode; (2) only the AUs of an image I and images P are decode and visualize; and (3) only the AU of an image I is decode and visualize. Therefore, the following three types of Reproduction can be carried out easily using: (1) normal playback, (2) medium speed playback, and (3) High speed playback.

In the MPEG-4 AVC, you can carry out the prediction where the AU of an image B refers to the AU of an image B. Fig. 6B is an example of a structure of prediction in an AVC stream of MPEG-4, and the AU of images B (B1 and B3) refer to AU (B2) of image B. In this example, the following four types can be carried out Decoding or viewing: (1) All images are decode; (2) only the AU, of an image I, images P e B images, to which reference is made, are decoded and visualize; (3) only the AUs of an image I, and the images P they are decoded and displayed; (4) only the AU of one Image I is decoded and displayed.

In addition, in the MPEG-4 AVC, the AU of an image P can refer to the AU of an image B. How to shown in Fig. 7, the AU of an image P (P7) can refer to the AU of an image B (B2). In this case, the AU of an image P (P7) can be decoded only after it has been decoded the AU of an image B (B2). Therefore all three following types of decoding or display can be carried out: (1) all images are decoded; (2) only AU, of an image I, images P and images B, to which it is made reference, they are decoded and visualized; (3) only the AU of one Image I is decoded and displayed.

In this way, since various are allowed prediction structures in the MPEG-4 AVC, it you must perform the analysis of the segment data and the evaluation of the prediction structure to know the relationship of reference between AU. This leads to a problem: AUs to decode or display cannot be determined based on a rule that is predetermined according to a playback speed in the time to skip playback, playback at variable speed and reverse playback, unlike the MPEG-2 case.

EP0756281 discloses an apparatus of flow generation of moving images in which the sector information of frames I and P, required for modalities of trick play, it is recorded in an input sector formed at the top of each GOP.

Disclosure of the invention

An object of the present invention is provide (i) an apparatus and a method of generating flow of moving images that generate an image flow in movement that can perform such a trick play such as skip playback, variable speed playback and reverse reproduction, even in the case of a procedure of encoding such as the MPEG-4 AVC, which allows flexible prediction structures, and (ii) an apparatus of decoding of moving images, and the like, which decode such a moving image stream.

The invention is an apparatus for generating moving flow according to claim 1, a method of generation of moving flow according to claim 2, a decoding device for moving images according to the claim 3, an image decoding method in movement according to claim 4, a recording medium computer readable according to claim 5, a method recording according to claim 6 and a system of decoding of moving images according to claim 7.

As described to this point, with the In the present invention, the AUs to be decoded in the trick play time, such as play at variable speed and reverse reproduction, referring to a specific NAL unit in the upper AU of an access RAU unit random. Therefore, an apparatus can be easily made decoding motion pictures with a function excellent trick play, and thus the present Invention is extremely practical.

Brief description of the drawings

These and other objects, advantages and features of the invention will become apparent from the following description, considered in conjunction with the drawings annexes, which illustrate a specific embodiment of the invention. In the drawings:

Fig. 1A and 1B are diagrams showing, respectively, MPEG-2 flow structures of a prior art;

Fig. 2A and 2B are diagrams showing, respectively, MPEG-2 GOP structures of a prior art;

Fig. 3A and 3B are diagrams showing, respectively, MPEG-4 flow structures of a prior art;

Fig. 4 is a block diagram showing the structure of a conventional coding apparatus;

Fig. 5 is a block diagram showing the structure of a conventional decoding apparatus;

Fig. 6A and 6B are diagrams showing, respectively, examples of the prediction structure in a conventional AVC flow of MPEG-4;

Fig. 7 is a diagram showing another example of the prediction structure in a conventional flow of MPEG-4 AVC;

Figs. 8A and 8B are diagrams showing, respectively, AVC flow structures of MPEG-4 of the present invention;

Fig. 9A to 9D are diagrams of a first example that shows the AUs to be decoded in a RAU drive of random access;

Fig. 10A to 10D are one second diagrams example that shows the AUs to be decoded in a RAU drive of random access;

Fig. 11A to 11C are diagrams of a third example that shows the AUs to be decoded in a RAU drive of random access;

Figs. 12A to 12F are diagrams of an example which shows the procedure to specify the AUs to decode in a random access RAU unit;

Fig. 13A is a diagram showing a example of a table syntax indicating information of variable speed reproduction, and Fig. 13B is a diagram that shows a data storage unit;

Fig. 14 is a diagram of an example of extension of a table indicating playback information to variable speed;

Fig. 15A to 15C are diagrams of an example which shows the AUs of image I and image P on a RAU drive random access in the form of playback information to variable speed;

Fig. 16A to 16C are diagrams of an example where the temporary memory detention time is used as priority indicator at the time of using the AU priorities as speed reproduction information variable.

Fig. 17A and 17B are diagrams showing respectively examples in which the AU of structure of plot and field structure AUs coexist in the RAU respective; Fig. 17C is a diagram showing the example of syntax of the first map (RAU_map1) that shows the structure of each AU in the RAU; Fig. 17D is a diagram showing the RAU_map1 of the RAU of Fig. 17B; Fig. 17E is a diagram that shows the RAU_map for the random access RAU unit of Fig. 17B; Fig. 17F is a diagram showing the example of second map syntax (RAU_map2) that shows the type of coding of each frame or each image of a couple of fields;

Fig. 18A to 18C are diagrams showing another exemplary map in the form of reproduction information;

Fig. 19 is a diagram of the procedure for indicate border information on an access RAU drive random;

Figs. 20A and 20B are diagrams showing examples of image prediction structures in a RAU drive random access;

Fig. 21 is a block diagram showing the structure of an image coding apparatus in movement of the present invention;

Fig. 22 is a flow chart of a coding procedure of moving images;

Fig. 23 is a block diagram showing the structure of an image multiplexing apparatus in movement of the present invention;

Fig. 24A and 24B are diagrams showing exemplary contents of the Support Information Help;

Fig. 25 is a diagram showing an example of a NAL unit in which the trick play information it is stored in the Help for support information;

Fig. 26 is a flow chart showing the operation of an image multiplexing apparatus in movement;

Fig. 27 is a block diagram showing the structure of an image decoding apparatus in movement of the present invention;

Fig. 28 is a flow chart of a conventional image decoding procedure;

Fig. 29 is a flow chart of determination of the AUs to decode in the procedure of decoding of moving images of the present invention;

Fig. 30 is a flow chart showing the processing performed in the case in which the AU to decode does not match the AUs to be displayed in the decoding procedure of moving images of the present invention;

Fig. 31 is a diagram showing a data hierarchy of an HD-DVD drive;

Fig. 32 is a structural diagram of the logical space on an HD-DVD drive;

Fig. 33 is a structural diagram of a information file of a video VOB object;

Fig. 34 is a diagram of a map temporary;

Fig. 35 is a structural diagram of a playlist file;

Fig. 36 is a structural diagram of a program file that corresponds to the playlist;

Fig. 37 is a structural diagram that shows a management information file of the entire disk Blue-Ray BD;

Fig. 38 is a structural diagram of a file to record a global event manager;

Fig. 39 is a block diagram showing the outline of an HD-DVD player; Y

Fig. 40A to 40C are diagrams showing a Recording media to store the program to perform the coding procedure of moving images and the decoding procedure of moving images of the present invention

Best way to carry out the invention

An embodiment of the present invention with reference to the figures.

Structure of an AVC flow

First, the structure of an AVC flow to generate by an image flow generation apparatus in movement, a motion image coding apparatus and a moving image multiplexing apparatus of the present invention, in other words, an AVC flow will be described at insert into an image decoding apparatus in movement of the present invention.

Fig. 8A and Fig. 8B show, respectively, the AVC flow structures of the present invention. Note that the border information to be added to the Top of a NAL unit is not shown in the figure. He AVC flow differs from a conventional AVC flow in that it add trick play information, indicating the trick play information the AUs to decode in the moment of trick play, such as play at jumps, variable speed playback and playback inverse The trick play information is stored in a NAL unit for storing playback information (Fig. 8A). In the MPEG-4 AVC, the relationship between the information to be stored and the type of NAL unit of a NAL unit Specific can be set per application. More specifically, values from 0 and 24 to 31 can be used, and these unit types NAL are called user-adjustable NAL unit types. In consequently, the trick play information is stored in the NAL unit that has such types of NAL units adjustable by Username. Here, in the case in which the NAL Unit types specific are reserved to store information other than trick play information, the types of NAL unit that are different to the NAL unit types are assigned to the trick play information. The NAL units of the Trick play information is stored in the upper AU of a random access RAU drive. Such a NAL unit is placed immediately after a NAL PPS unit, if present, in an AU, but can be placed in another position while the order meet the requirement of the MPEG-4 AVC or other rule. Similarly, in the case in which it is impossible to interpret the NAL unit of trick play information, data from the NAL unit can skip and restart decoding from the top of the next NAL unit. Therefore even a terminal that cannot interpret the NAL information unit of trick play can carry out the processing of decoding without failures.

Note that such a NAL unit of information of trick play can be included, not in the upper AU of a RAU random access unit, but in another AU such as the last AU. Likewise, such a NAL unit of reproduction information trick can be included in each AU that constitutes a RAU unit of random access

Figs. 9 to 11 show examples of AU a decode at the time of variable speed playback. Fig. 9A shows the display order of the AU. Here, the AU diagonally shaded are the AUs to which other AUs refer, and the arrows show images referred to. Be assign negative reference numbers to the AUs to be displayed before of 10, and positive reference numbers are assigned to AUs a display after B15. Fig. 9B shows the order of decoding of the AUs shown in Fig. 9A, and 10 to B11 they constitute a random access RAU unit. At this time, 10, -B14, P4, B2, P8, P6, P12 and B10 are decoded to lead to perform double speed playback (Fig. 9C), while 10, P4, P8 and P12 are decoded to perform a reproduction of quad speed (Fig. 9D). Fig. 9C and 9D show that the AU with a sign * must be decoded at the time of reproduction dual speed and quad speed playback, and these fragments of information are stored in the NAL unit of the trick play information. In the example of Fig. 10A a 10D, images from 10 to B11 in decoding order they constitute a random access RAU unit. Here, 10, -B13, P3, B1, P6, B4, P9, B7, P12 and B10 are decoded to perform a reproduction at a speed multiplied by 1.5 while I0, P3, P6, P9 and P12 are decoded to perform playback at triple speed. Similarly, in the example of Fig. 11A to 11C, I0, P3, P6, P9 and P12 are decoded to perform a triple speed playback.

Here the playback speeds do not they need to be exact because they are described as guidelines of the playback speeds For example, in the example of Fig. 11C, in the case in which all AUs shown as AUs a decode at the time of triple speed playback will decode, the speed is 3.2 times higher than that obtained at starting from the expression: 16: 5; In other words, it is not exactly triple speed Likewise, at the time of reproduction multiplied by M, in the case in which the lowest value over M is N between the playback speeds shown as information of trick play, it is possible to decode the necessary AU to be decoded at the time of multiplied playback by N and determine how the rest of the AU should encode, according to the implementation of the decoding apparatus. Likewise, it is possible to place high priorities on UA necessary to be decoded in the case in which the playback speed is fast, and determine the AU to decode based on priorities.

Note that some AUs, between AUs to decode at the time of variable speed playback,  They may not be displayed. For example, the Nth AU is displayed in the moment of double speed playback, but I don't know Displays the M-th AU. At this time, in the case in which there is a need to decode the M-th AU in order to decode the N-th AU, the M-th AU is decoded but not  displayed at the time of double speed playback.

Then the procedure for specify the AUs to decode at the time of playback to Variable speed will be described with reference to Fig. 12A a 12F. Figs. 12A to 12F show the examples of specification of the AUs to decode on the same random access RAU drive that that of Fig. 9. As shown in Fig. 12D, 10, -B14, P4, B2, P8, P6, P12, B10 are decoded at the time of playback at double speed These AUs are the first, second, fifth, sixth, ninth, tenth, thirteenth and fourteenth AU when counted AUs from the upper AU of the RAU access unit random. In this way, it is possible to uniquely specify the AU to decode at the time of variable speed playback showing the ordinal numbers of the AUs in a RAU unit of random access An access unit delimiter is placed surely on top of an AU at the time of multiplex an AVC stream by a transport stream (TS) of MPEG-2 When data is obtained from AU to decode at the time of variable speed playback, the delimiters of access units are searched in sequence to know the limits of the AU. This form of processing Search eliminates the need to analyze the unit payload NAL, such as segment data, and is therefore easier.

Note that it is possible to specify AUs to decode determining that the AUs to which they will refer other AU, such as the AU of an image I and P images (such AU to those that will be referred to are called reference AU), decode at the time of variable speed playback, and specifying the ordinal numbers of the reference AUs in a random access RAU drive. In the access RAU drive random of Fig. 12B, as shown in Fig. 12C, 10, -B14, P4, B2, P8, P6, P12, B10, are reference AU. Likewise, in the Dual speed playback time, 10, -B14, P4, B2, P8, P6, P12, B10 are decoded, but when these AUs are indicated on the order of reference AUs, correspond to the first, second, third, fourth, fifth, sixth, seventh and eighth reference AU, as shown in Fig. 12F. If an AU is or is not an AU of reference can be evaluated by referring to a specific field in the NAL unit heading in a segment. Plus specifically, in the case where the value of a field nal_ref_idc is not 0, the AU is a reference AU. Note that A reference AU to decode can be specified on the base of a frame number, because it is possible to identify an AU of reference based on a frame number.

In addition, it is possible to specify AUs to decode specifying the offset value equivalent to the length in octets, from the start position of the upper AU from a random access RAU drive to the starting position of the AU to decode. For example, in Figs. 12A to 12F, in the in case 10 starts with the position away from the part top of a flow in 10,000 octets, and P4 starts with the remote position of P4 in 20,000 octets, the value shifted to P4 is 10,000 octets, obtained from the expression: 20,000 - 10,000. In the case in which a multiplexed flow is used in a MPEG-2 TS, it is possible to specify a value of displacement that includes the overhead of the heading of a TS package or a PES package (Elementary flow in packages), or it is possible to specify a shifted value that includes it in the time to carry out the filling of data by application. Likewise, it is possible to specify an AU for an FN number of plot.

Note that, in the case of using a flow multiplexed in a MPEG-2 TS, it is possible specify AUs based on the number of TS packets from of (i) the TS package to store the index number and the address information to identify a TS package that includes the upper AU data to be decoded, or the data RAU unit top random access to (ii) TS package current. Here, it is possible to use the information about the Package Source to use for a disc recording format Blu-ray (BD) instead of a TS package. The package Origin is obtained by adding, to a TS package, a header 4-byte that includes temporary TS packet information, copy control information and the like.

Fig. 13A is an example of a table syntax indicating the information for variable speed playback. In the syntax, nro_imag_en_RAU shows the number of AUs that constitute a random access RAU unit, nro_velocity shows the number of playback speeds at which AUs are to be decoded, playback speed shows a playback speed, nro_imag_desc shows the number of AU to be decoded at the time of playback at a playback speed shown in playback speed, imag_desc shows the ordinal numbers of the AUs to be decoded in the case of counting the AUs from the upper AU in a random access RAU unit. Fig. 13B is an example in the case of storing information about the AUs to be decoded in a random access RAU unit shown in Figs. 9A to 9D at the time of double speed playback and quad speed playback. Note that nro_imag_en_RAU is used at the time of calculating an exact playback speed based on the number of AUs to be decoded and the total number of AUs in a random access RAU unit, or jump based on random access RAU units in sequence. However, nro_imag_en_RAU can be omitted because the same information can be obtained by searching the upper AUs of the random access RAU units. Similarly, a field indicating the size of a table can be added to the table. Note that, in the example syntax of Fig. 13A, the ordinal number of an AU to be decoded, counting from the top of a random access RAU unit, is shown directly, but whether or not it is necessary to decode each AU can display by turning on or off the bits corresponding to each AU. For example, a random access RAU unit is composed of 16 AU in the example of Figs. 9A to 9D, and 16 bits are required when 1 bit is assigned to an AU. At the time of quad speed playback, it is shown that the first, fifth, ninth and thirteenth AUs are decoded by assigning 16-bit information that is represented as 0b1000100010001000 (0b indicates a binary number). Here, the upper bit and the last bit correspond, respectively, to the upper AU and the last AU of an access RAU unit
random.

Note that the size of a table is variable in the syntax example of Fig. 13A. The maximum value The size of the table is determined in the case where they are prescribed the maximum value of the number of AUs that constitute a RAU unit of random access and the maximum value of nro_velocity. In Consequently, it is possible to set the table size to the value maximum determined and, in the case where the size of the information for variable speed playback does not reach the maximum value, It is possible to carry out the filling. Fixing the size of the table in this way makes it possible to always obtain data from a fixed size when playback information is obtained at variable speed, which allows to speed up the processing of obtaining information. Note that the table size, or the size of a NAL unit to store the table, it is shown as management information It is also possible to predetermine the size of a NAL unit to store playback information trick, and, in the case where the information cannot be stored in a single NAL unit, it is possible to store the information to Variable speed playback on various NAL units by separated. At this time the filling is carried out on the load useful of the last NAL unit, so that the unit size NAL becomes the default size. Also some prescribed values are determined as the size values of the table, and the index number indicating a prescribed value of the Table size can be displayed in the table, or use the application management information.

It is also possible to display information differential instead of listing all AUs to decode each playback speed As the information at the time of the playback at a speed multiplied by M (<N), only the AUs required to decode are displayed in addition to the AUs a decode at the time of playback at a speed multiplied by N. In the example of Fig. 13B, as the second, sixth, tenth and fourteenth AU, in addition, of the AU to decode in the moment of playback at quad speed, it decode at the time of double speed playback, it is possible to show only the second, sixth, tenth and fourteenth AU as the information for double speed playback.

Note that the AUs required to decode at the time of variable speed playback they are displayed in the above description, but also, it is possible to show the information indicating the display order of the AU necessary to decode. For example, the information at the moment of double speed playback and speed playback quadruple is shown in the example of Fig. 9A to 9D, but there is a  Playback example of this random access RAU drive to triple speed The display of a part of the AU a display at double speed playback, in addition of the AU to display at the moment the speed reproduction Quad, allows triple speed playback. Here, when considering the case where one or more AUs are displayed between I0 and P4, to be displayed at the time of Quad-speed playback, the information for the Dual speed playback shows that the candidates are -B14, B2, B6 and B10. However, the display order of these four  AU can be obtained only in the case where the heading information of a segment. Here, since the information in the display order shows that only -B14 is displayed between I0 and P4, it is possible to determine that -B14 is decoded Fig. 14 is an example of a syntax indicating the information about the display order, and is obtained by adding the information about the display order to the syntax of the Fig. 13A. Here, ind_pts_dts shows if the decoding order of the AUs to decode at the playback speed matches or not with the display order of the AUs, and only in the case where the decoding order does not match the order of display, display order information is displayed in a display_order field.

Note that, in the case of reproduction at a playback speed that is not shown by the information in variable speed playback, it is possible to determine the AU at decode and the AU to display based on the rule that is Default in the terminal. For example, in the case of reproduction at triple speed in the example of Fig. 9, it is possible display I0, B3, B9 and P12 in addition to the AUs to be displayed in the Quad speed playback time, instead of visualize a part of the AU to be displayed at the time of double speed playback. Here, as in images B, the B images in the reference AUs can preferably be decode or display.

Likewise, there is a case where trick play, such as variable speed playback, is performed by reproducing only the AU of an image I or only the AU of an image I and P images. Therefore, a list of an I image and P images as trick play information. Fig. 15A to 15C show another example. Here, images from 10 to B14 are included in a random access RAU unit as shown in Fig. 15B and, among those, the AUs of an image I and images P are 10, P3, P6, P9, P12 and P15, as shown in Fig. 15C. Therefore, the information is stored to identify 10, P3, P6, P9, P12 and P15. At this time, it is possible to add the information to distinguish the AU of an image I from the AU of an image P. Likewise, it is possible to display the information to distinguish the following images from each other, including the images: an image I, P images, B images to be used as reference (hereinafter referred to as B reference images), and B images to which no reference will be made (hereinafter referred to as B images of non-
reference).

In addition, it is possible to store the information of priority of the respective AUs as playback information trick, and decode or display the UA according to the priorities in the moment of playback at variable speed. it's possible Use image types as priority information. By For example, UA priorities can be assigned in the following order of enumeration: (i) an image I; (ii) P images; (iii) reference B images; e (iv) B images of non-reference It is also possible to establish Priority information as follows: the higher the time between the moment after decoding of an AU and the moment the AU is displayed, the higher the priority becomes. Fig. 16A to 16C show an example of priority setting according to the time of detention in temporary memory. Fig. 16A shows the prediction structure of the AU and at P3 it is done reference also by B7 and P9. At this time, in the case where the random access RAU unit is composed of the AU from I0 to B11 (Fig. 16B), the stop time in memory Temporal of each AU is as shown in Fig. 16C. Here the Detection time in temporary memory is displayed based on the number of frames For example, P3 is necessary until it is decodes P9, and the temporary memory stop time must be equivalent to six images. Therefore, decoding of the AU whose detention time in temporary memory is 3 or more means the decoding of the entire image I and all P images, and triple speed playback is performed. Here, the temporary memory stop time of P3 is longer than 10, but it is possible to add a shifted value to the AU of the image I to place the highest priority over the AU of the image I. Likewise, it is possible to place high priorities in the AU necessary to decode at the time of high playback speed and use, as priority information, N in the AU necessary to decode at the time of reproduction to velocity multiplied by N. Note that, in the case where a AU is referenced in other AUs after being decoded or displayed, it is possible to show the period of time during which Reference is made to AU.

Note that the reproduction information Trick can be stored in a SEI message (Fig. 8B). In this case, the type of SEI message is defined for the information of trick play, and trick play information is stores in the SEI message of the defined type. The SEI message for the Trick play information is stored in the NAL unit of the SEI, alone or together with other messages. Note that it is possible store trick play information in the SEI message user_data_registered_itu_t_t35 or in the SEI message user_data_unregistered, which are SEI messages to store the User defined information. At the time of using these SEI messages, it is possible to show that the playback information trick is stored, or that type of reproduction information trick in the payload part of an SEI adding information of identification of the information to be stored.

Note that it is possible to store information of trick play in AUs other than the upper AU in a random access RAU drive. It is also possible predetermine the values to identify the AU needed to decode at the time of playback at a speed of specific reproduction and add the determined values for each AU. For example, in relation to UA to decode a playback speed which is a speed multiplied by N, or lower, N is given as speed information of reproduction. It is also possible to show the following in nal_ref_idc, and the like, of the NAL unit of a segment: the structure of the image in an AU, the structure being a frame structure or a field structure and, in addition, in the case where the image has a field structure, it is possible to show the type of field, which is an upper field or a lower field. By example, as there is a need to display alternately the upper fields and upper fields in the case of the interlaced display, it is desirable that it can be evaluated easily that the field to decode next is a field upper or lower field when decoding fields skipping some fields at the time of high playback speed. In the case where the type of field can be evaluated at Starting from the heading of a NAL unit, there is no need to analyze the segment heading and the magnitude of Necessary processing for such evaluation can be reduced.

Note that the information indicating whether the AU which constitutes a random access RAU unit is a field or a frame can be stored in the upper AU of a RAU unit of random access It is also possible to easily determine the AU to decode at the time of trick play, even in the case where a field structure and a frame structure coexist storing such information in the upper AU of the random access unit. Fig. 17A and 17B are examples where AUs that have a frame structure and AUs that have a field structure coexist in the random access RAU unit, and show the order of display of the AUs and the order of decoding of UA, respectively. The following images respectively, they are coded as pairs of fields: B2 and B3; 14 and P5; B9 and B10; B11 and B12; P13 and P14; B15 and B16; B17 and B18; and P19 and P20 Similarly, the other AUs are coded as the AUs that have a plot structure. At this time, in the case of reproducing only the AU of an image I and P images, you can decode and reproduce the following in the following order listed: the pair of fields of 14 and P5; the plot of P8; The pair of P13 and P14 fields; and the pair of fields of P19 and P20. But nevertheless, the addition of such information is effective because there is a need to evaluate if each AU is one of the fields that constitute a pair of fields or if each AU is a frame at the time of determining the AU to decode.

Fig. 17C is an example of the first syntax map (RAU_map1) that indicates whether an AU in an access RAU drive Random is a plot or a field. The number of AUs that constitute a random access unit shown in nro_AU_en_RAU, and the information about each AU is shown in the Next loop in a decoding order. Here, tram_field indicator shows if the image to be stored in an AU is A plot or a field. Similarly, type_imag shows the information about the type of coding of an image. The types of encoding that can be displayed include: an image I; a IDR image; an image P; a reference image B; an image B non-reference; and the like Therefore it is possible to determine the images to be decoded at the time of Trick play referring to this map. Note that it is possible to indicate if each image I and each image P are subject to reference or not. In addition, it is possible to indicate the information for assess whether a predetermined requirement applies in terms of prediction structures

Fig. 17D shows RAU_map1, which refers to a random access RAU unit of Fig. 17B. Here, the type_imag of an image I, P images, reference B images, and B images of non-reference, are 0, 1, 2 and 3, respectively. Here, it is possible to store the information that indicates the types of image coding on the basis listed above, because the images are reproduced plot to plot, or pair of fields to pair of fields, at the time of the trick play

Fig. 17F is an example of the syntax of the second map (RAU-map2) indicating types of frame-by-frame or pair of field image coding fields Here, nro_trama_en_RAU shows the number of frames that constitute a random access RAU unit and the number of pairs of fields. Similarly, plot_indicator shows if an image is a plot or not, and in the case where it is a plot, 1 is put there. At case where 1 is placed in plot_indicator, the information on the encoding type of a frame is shown in frame_type. At case where 0 is set in frame_indicator, in other words, the image is one of a couple of fields, and the type of coding of each field that constitutes the pair of fields is shown in field_type_type.

Fig. 17E shows the RAU_map2 in terms of RAU random access unit of Fig. 17B. In Fig. 17E, the values that indicate the frame_type of an image I, images P, B reference images, and B images of non-reference, they are 0, 1, 2 and 3, respectively. Similarly, the field_par type shows the type of each field in a decoding order. The types of fields are as follows: I for an image I; P for P images; Br for B images of reference; and Bn for non-reference images. By example, it is shown as IP in the case where the first field is a image I and the second field is an image P, and is shown as BnBn in the case where the first field and the second field are images B of non-reference. Here, the values to indicate combinations of IP, PP, PI, BrBr, BnBn and the like are established previously. Note that the following information can be used as the information that indicates the type of coding of a pair of fields: information as to whether the pair of fields includes a image I or one or more images P; information as to whether the pair of fields includes one or more reference B images; and information as to whether the pair of fields includes one or more B images of non-reference.

For example, playback information trick can be the map of a random access RAU drive, as a syntax shown in Fig. 18A. This map includes structure_image, which indicates the structure of each of the images included in the random access RAU drive, and type_image, which indicates the type of image. As shown in the Fig. 18B, image structure shows the structure of each image, it is say, a field structure or a frame structure, and Similar. Likewise, as shown in Fig. 18C, image_type shows the type of image of each image, that is, an image I, a reference image B, an image B of non-reference, and an image P. Thus, the motion picture decoding apparatus that you received This map can easily identify the AUs on which Performs trick play by referring to this map. How For example, it is possible to decode and reproduce, in reproduction at high speed, only one image I and P images, or images Reference B, in addition to an image I and images P.

Note that in the case where the information which indicates the image structure, such as a downward advance 3-2, is included in an AU that constitutes a unit RAU random access, it is possible to include the information that indicates the image structure on the first or second map previously described. For example, it is possible to show if each image has display fields equivalent to three images or if each image has display fields equivalent to two images, In addition, in the case where you have display fields equivalent to three images, it is possible to display the information that indicates whether the first field is displayed repeatedly, or the information that indicates whether the first field is a higher field. Likewise, in the case where you have display fields equivalent to two images, it is possible to show the information of whether The first field is a higher field. Here, in the AVC of MPEG-4, whether or not an image has a structure of image, such as a 3-2 downward advance, can be displayed using (i) the pic_struct_present_flag of a set (SPS) of sequence parameters or (ii) the picture_to_display_conversion_flag, and the like, on timing of the AVC and the HRD descriptor defined in the MPEG-2 system. In addition, the structure of each image is displayed by a pic_struct field of a Timing SEI of Image. Therefore, it is possible to display the image structure setting an indicator only in the case where a pic_struct field has a specific value, for example, an image has fields of Display equivalent to three images. In other words, indicate the following three types of information regarding each image is effective (i) in the case where jump playback is performed in the middle of a random access RAU unit and (ii) in the moment to determine the field to display in a moment specific or the frame in which a field is stored. The same is can say in the case of determining images to be displayed during Variable speed playback. The three types of information They are:

(i)

countryside

(ii)

plot (used at the time of not using an advance descending from 3-2, or which is also used in the time to use a 3-2 downward advance. At In the latter case, the plot has display fields equivalent to two images).

(iii)

plot that has a display field equivalent to three images at the time of using a preview descending from 3-2

Note that these types of information are they can indicate in the image_ structure of a RAU map shown in Fig. 18A.

Indicate list information of types of images of the respective images that constitute a RAU of this  way makes it possible to easily determine images to decode or display at the time of making a trick play, such as variable speed playback, Skip play and reverse play. This is especially effective in the following cases:

(i)

where only one image I and P images are reproduce

(ii)

where high speed playback is performed, in which there is an image I, images P and reference images B; Y

(iii)

where the images, on which they are established requirements regarding prediction structures, it identified on the basis of image types, the images necessary to decode at the time of trick play, and the selected images are played back in trick play

In addition, it is possible to store a value by omission of trick play information in a region, which is different from the AVC flow, such as management information to application level, and include trick play information in a random access RAU unit only in the case where the trick play information is different from the information of trick play displayed by default.

The trick play information as far as to the variable speed reproduction described above,  but it is possible to use similar information as information supplementary at the time of reverse reproduction. it's possible complete decoding at a time, at the time of reverse playback in the case where all the images to display can be stored in a memory, and the load of Processing required for decoding can be reduced. Considering a case of reverse reproduction in the listed order of P12, P8, P4 and I0 in the example of Fig. 9A to 9D, provided that all decoding results of the four AUs are stored, it is possible to decode 10, P4, P8 and P12 in this order in a moment and perform reverse playback. Therefore, it is possible to evaluate whether all decoded data of the AUs may or may not be stored based on the number of AUs to decode or display at the time of playback, at a speed multiplied by N, and determine the AU to be displayed in the time of performing reverse playback based on The result of the evaluation.

Likewise, information from trick play as supplementary information at the time from jumping playback. Here, skip playback means fast forward of a moving image and perform normal playback of the start of moving images with the position determined at random. Determine advance images fast using such supplementary information, even in the Skip play time, makes it possible determine the image in which playback starts at jumps

Note that the AU to serve as a reference for each AU that constitutes a random access unit can be show directly in the trick information. In the case where there is various reference AUs, all of them are shown. Here in the case where a reference AU belongs to an access unit random other than the random access unit that includes a AU which refers to the reference AU, the AU can be indicated as the following specific way: the M-th AU of the access unit random that is placed before or after N unit numbers random access, or the AU can be indicated as follows simple way: the AU that belongs to the access unit random that is placed before or after N numbers of units of random access Note that it is possible to show the number ordinal, in the order of decoding, of the AU reference in the case of counting from the AU that refers to the AU of reference. At that time, AUs are counted on the basis of one in the following cases: all AU; the AU reference; the AU of a specific type of image such as I, P and B. It is also possible to show that each AU can refer to the AU of only up to N AU numbers before and after, in an order of decoding. Note that, in the case of referring to an AU which is not included in the AU of up to N AU numbers before and after in the order of decoding, it is possible to add the information Which indicates the fact.

Note that it is possible to use information from trick play described above in a similar way also in a multiplexed format, such as MP4, where the size of a NAL unit is used instead of using a code prefix of start as frontier information of a NAL unit.

Note that, at the time of receiving and recording an encoded stream that is packetized using a TS packet (Flow transport) of MPEG-2, or an RTP (Protocol real-time transmission), packet loss occurs. From this way, in the case of recording the data received in a environment in which a packet loss occurs, it is possible store, in a coded stream as supplementary information, or as management information, the information that indicates that the data  in a flow they have been lost due to a loss of packages. Is possible to show a loss of data due to the loss of a package by inserting the indicator information that indicates whether flow data has been lost or not, or a notification code of Special error to notify the lost part. Note that, in the case of performing error concealment processing when data is lost, it is possible to store information from identification indicating the presence / absence, or the procedure, error concealment processing.

The trick play information for determine the AU to decode or display at the time of Trick play has been described to this point. Here the data structure to allow the detection of the border of RAU random access units will be described with reference to Fig. 19.

In the upper AU of an access RAU drive random, the NAL unit of an SPS to be referenced by an AU that constitutes a random access RAU unit is stored forever. Moreover, in the MPEG-4 AVC standard, it is possible to store the NAL unit of the SPS to be subject to reference by the N-th AU in a decoding order, in a AU that is arbitrarily selected from the Nth AU or AU placed before the N-th AU in a decoding order. Such a NAL unit is stored so that the NAL unit of an SPS can be transmitted repeatedly, in preparation for the case where the NAL unit of an SPS is lost due to a loss of packets at the time of transmitting a flow in communication or diffusion. However, the following rule is effective for use. of storage applications. Only one unit SPS NAL to be referenced by all unit AUs Random access RAU is stored in the upper AU of a unit Random access RAU, and the SPS NAL drive is not stored in the following AUs in the random access unit. Doing so makes it possible to ensure that the AU is the upper AU of the RAU unit random access if it includes the NAL unit of an SPS. The beggining of the random access RAU drive can be found by looking for the NAL unit of the SPS. The management information of a flow such as a temporary map does not guarantee the provision of access information as for all random access RAU units. For the therefore, it is especially effective that the starting position of each RAU random access unit can be obtained by searching for the unit NAL of an SPS in a flow in the case of, for example, performing skip playback on the image in the middle of the unit RAU random access whose access information is not provides

Here, in the case where the upper AU of the RAU random access unit is the AU of an IDR image, the AU of the random access RAU unit does not refer to the AU in the unit RAU random access that is previously placed in an order of decoding. This type of random access RAU drive is denominated random access RAU unit of closed type. For other part, in the case where the upper AU of an access RAU drive random is the AU of an image I that is not an IDR image, the AU of the random access RAU unit can refer to the AU in the RAU random access unit that is previously placed in an order decoding This type of random access RAU drive is denominated RAU unit of random access of open type. At moment when the angles are switched during playback in a optical disk or similar, the switching is performed from a RAU random access RAU unit. Therefore, it is effective that the assessment as to whether a random access RAU drive is a random access RAU unit of open type or type closed can be done on top of the RAU unit of random access For example, it is possible to display the information of indicator to evaluate the type, that is, an open type or a type closed, in a nal_ref_idc field of the NAL unit of an SPS. How I know defines that the value of nal_ref_idc is 1 or more in the NAL unit of an SPS, the higher order bit is always set to 1 and the indicator information is shown by the order bit lower. Note that an AU in a random access RAU drive cannot refer to an AU in a random access RAU drive which is previously placed in a decoding order, even in the case where the upper AU is the AU of an image I that is not a IDR. This type of random access RAU drive can be consider as a random access RAU unit of closed type. Note that the indicator information can be displayed using a field other than nal_ref_idc.

Note that it is possible to specify the position Startup of a random access RAU drive based on the drive NAL other than an SPS, to be stored only in the upper AU of a random access RAU drive. It is also possible to show the type, that is, the open type or the closed type, of each of RAU random access units, using the nal_ref_idc field of each random access RAU unit.

Finally, Figs. 20A and 20B show examples. of AU prediction structures that constitute a unit RAU random access. Fig. 20A shows the positions of the AU in a display order, and Fig. 20B shows the AU positions in decoding order. As shown in figures, B1 and B2, shown before 13, which is AU top of a random access RAU drive, they can refer to the AU to display after I3. In the figure, B1 refers to P6, Here, in order to ensure that the AUs of I3 and the following images in the order of display can be decoded correctly, it is prohibited that the AUs of I3 and the following images in the order of display refer to the AU before I3 in the order of display.

Motion picture coding device

Fig. 21 is a block diagram of the apparatus 100 coding of moving images performed by the coding procedure of moving images of the present invention This image coding apparatus 100 in movement generates a coded flow, shown in Figs. 8 to 20, of a moving image that can be played using a trick play such as skip play, the Variable speed playback and reverse playback. He motion image coding apparatus 100 includes a TrickPlay unit for generating playback information trick, in addition to the units of an apparatus 1 coding of moving images shown in Fig. 4. Note that at processing units that perform the same operations as the processing units of a conventional apparatus of coding of moving images, shown in the diagram in blocks of Fig. 4, the same reference numbers are assigned in the figure, and their descriptions will be omitted.

The TrickPlay information generation unit Trick play is an example of a unit that generates, based on a random access unit that includes one or more images, supplementary information to be subject to reference in the playing time of the access units random. The TrickPlay information generation unit of trick play generates trick play information based on TypeI types of images, and notifies information of trick play to the VLC unit of length coding variable.

The VLC unit of length coding variable is an example of a flow generating unit that generates a flow that includes supplementary information and images, adding the additional information generated to each unit of corresponding random access. The VLC encoding unit of variable length encodes and places the NAL unit to store trick play information in the upper AU of a unit RAU random access.

Fig. 22 is a flow chart of how the motion image coding apparatus 100 (mainly the TrickPlay information generation unit of trick play) shown in Fig. 21 performs the procedure for generating a coded stream that includes trick play information.

First, in step 10, the apparatus 100 Encoding of moving images evaluates whether AU to Encode is or is not the top AU of an access RAU drive random. In the case where the upper AU is, it is passed to the Stage 11, while in the case where it is not the upper AU, it is passed to the step 12. In step 11, the coding apparatus 100 of moving images performs initial processing to generate  trick play information of the access RAU drive random, and also ensures the area to store the trick play information on the upper AU of the unit RAU random access. In Step 12, the apparatus 100 of Motion picture encoding encodes AU data, and then goes to Stage 13. In Stage 13, the apparatus 100 of motion picture encoding gets the information necessary at the time of generating reproduction information trick Such information is: the types of images of the AU, is that is, an image I, an image P, a reference image B, or a non-reference B image; or if there is a need for decode the AU at the time of playback at a speed multiplied by N. After this, the apparatus 100 of encoding of moving images goes to Stage 14. In the Step 14, the motion image coding apparatus 100 evaluates if the AU is the last AU of the access RAU unit random. In the case where it is the last AU, the apparatus 100 of Motion picture coding goes to Stage 15, while that in the case where it is not the last AU, it goes to Stage 16. In the Step 15, the motion image coding apparatus 100 determines trick play information, generates the NAL unit to store the trick play information, and store the NAL unit generated in the area insured in Stage 11. After complete the processing of Step 15, the apparatus 100 of encoding of moving images goes to Stage 16. In the Step 16, the motion image coding apparatus 100 Evaluate whether or not there is an AU to code next. If where there is an AU to code, repeat Stage 10 and the following stages, while in the case where there is no AU to code, Complete the processing. Here, in the case where the apparatus 100 of encoding of moving images evaluates that there is no AU to encode in Step 16, store playback information trick of the last random access RAU drive, and to Then complete the processing.

For example, when the apparatus 100 of Motion picture encoding generates information from trick play shown in Fig. 18A, gets the following in Stage 13: the type of image; if the image has a field structure or if the image has a frame structure; or / and the information indicating whether the display field of the image is equivalent to two images or equivalent to three images in the case where the information regarding a downward advance 3-2 is included in the coded stream. On stage 15, the motion image coding apparatus 100 set the image_ structure and image_type of all images in the random access RAU in an order of decoding.

Note that, in the case where the size of the NAL unit to store trick play information is not know when you start coding the upper AU of a random access RAU drive, processing to ensure the area to store the trick play information is will omit in Step 11. In this case, the NAL unit generated for storing trick play information is inserted into the AU superior in Stage 15.

Likewise, storing or not storing the trick play information can be switched over a coded flow base. Especially in the case where the prediction structure among the AUs that constitute a unit of random access is prescribed by application, it is possible to determine that the trick play information is not stored. By example, in the case where a coded stream has the same prediction structure that in the case of a flow of MPEG-2, there is no need to store information from trick play This is due to the fact that it is possible determine the AU needed to decode at the time of Trick play without trick play information. Note that such switching can be carried out on the basis of a random access RAU drive.

Motion picture multiplexing device

Fig. 23 is a block diagram showing the structure of the image multiplexing apparatus 108 in movement of the present invention. This apparatus 108 of multiplexed moving images enter image data in motion, encodes data from moving images to compose an AVC stream of MPEG-4, multiplex the flow with the access information to the AUs that constitute the flow and the management information that includes the information Supplementary to determine operations performed at the time of trick play, and record the multiplexed stream. He motion image multiplexing apparatus 108 includes a unit 101 for determining flow attributes, a unit 102 of coding, an information generating unit 103 of management, a multiplexing unit 106 and a unit 107 of storage. Here, the encoding unit 102 has a function to add trick play information in the motion image coding apparatus 100 shown in the Fig. 21.

The attribute determining unit 101 of flow determines the requirements concerning reproduction trick performed at the time of encoding an AVC flow MPEG-4, and sends them to unit 102 of coding and to the information generating unit 105 of Playback support as TYPE attribute information. Here, the requirements concerning trick play include Information indicating: if the requirement to constitute a unit random access applies or not to an AVC flow of MPEG-4; if the information that indicates the AU to decode or display at speed playback variable or reverse reproduction is included in the flow; or if I know establish or not a requirement on the prediction structure between the AU The flow attribute determination unit 101 sends to the general management information generation unit 104 the general management information which is the information necessary for generate management information such as a compression format or a resolution The coding unit 102 encodes the data of video introduced into the MPEG-4 AVC stream based on the TYPE attribute information, send the data encoded to multiplexing unit 106, and sends the information of access in the flow to the information generating unit 104 General management Here, in the case where the information from TYPE attribute shows that the information indicating the AUs to decode or display at speed playback variable or reverse reproduction is not included in the flow, the trick play information is not included in the flow encoded. Note that the access information indicates the information of an access unit that is the basic unit to access the flow, and includes the start address, the time of display, and the like, of the upper AU in a unit of access. The management information generation unit 104 general generates the table data to refer to at the moment access to a flow, and the table data they store attribute information, such as a compression format based in the access information, and general management information, and sends the table data to multiplexing unit 106 as the INFO management information. The 105 generation unit Playback support information generates information from HLP support that indicates if the flow has an access structure random based on the TYPE attribute information entered, and sends the HLP support information to the multiplexing unit 106. The multiplexing unit 106 generates coded data entered through the coding unit 102, the information of INFO management, and multiplexing data, multiplexing the HLP support information, and then send them to unit 107 of storage. The storage unit 107 records the data of multiplexed introduced through multiplexing unit 106 on a recording medium, such as an optical disc, a disc rigid and a memory. Note that the coding unit 102 you can packetize the MPEG-4 AVC stream in, by example, a TS (transport stream) of MPEG-2 or a PS (program stream) of MPEG-2, and then Send the MPEG-2 TS or PS packetized. Similarly, the encoding unit 102 can packetize the flow using a format prescribed by the application, such as a BD.

Note that the information content of management is not required to depend on whether the information from Trick play is stored in the coded stream or not. In At this time, HLP support information can be omitted. Likewise, the motion image multiplexing apparatus 108 can have the structure without a unit 105 generating Playback support information.

Figs. 24A and 24B show examples of the information displayed by the HLP support information. The HLP support information includes the procedure that indicates directly the information of a flow as shown in Fig. 24A, and the procedure that indicates whether the flow satisfies the requirement prescribed by a specific application standard, as shown in Fig. 24B.

Fig. 24A shows the following as information concerning a flow: information as to whether the flow has a random access structure; information in as to whether there is a requirement on the prediction structure between images stored in an AU; and information as to whether there is information that indicates the AU to decode or display in the Trick play time.

Here the information related to the AU a decode or display at the time of trick play can directly indicate the AUs to decode or display, or indicate priorities at the time of decoding or viewing. For example, you can indicate that the information that indicates that AU to decode or display on the basis of the units of Random access is stored in a NAL unit that has a type of Special NAL unit prescribed by application, an SEI message or Similar. Note that it is possible to indicate if there is information that indicates the prediction structure between the AUs that constitute a random access unit. Likewise, the information related to the AUs to decode or display at the time of playback Trick can be added based on one or more units of random access, or to each of the AUs that constitute the random access unit.

In addition, in the case where the information that indicates the AU to decode or display is stored in the unit NAL that has a special type, it is possible to show the type of unit NAL of the NAL unit. In the example of Fig. 25, in the HLP support information, information related to AUs to decode or display at the time of trick play is includes in the NAL unit whose NAL unit type is 0. In that At this time, it is possible to obtain information regarding the trick play demultiplexing the NAL unit whose type of NAL unit is 0 from the flow AU data. If where information related to trick play is stored using an SEI message, it is possible to indicate the information for Identify the SEI message.

Similarly, regarding the requirements on prediction structures, it is possible to indicate if one or more Default requirements are met or not, or it is possible to indicate that the following respective requirements are met independently:

(i)

as for the AU of an image I and P images, the decoding order should match the order of display;

(ii)

the AU of an image P cannot refer to the AU of an image B;

(iii)

the AU after the upper AU in an order of display in a random access unit can refer only to AUs included in the random access unit; Y

(iv)

each AU can only refer to the AU placed up to N numbers before and after in the order of decoding. In this case, all AUs are fully counted. or the AUs are counted on the basis of the reference AUs, and the N value can be displayed in support information HLP

Note that, in the AVC of MPEG-4, it is possible to use, as images of reference, images on which processing is performed filtering (unlocking) to eliminate block distortion after decoding, to improve image quality, and it is possible to use, as images for display, images before unlocking In this case, the decoding apparatus of moving images you need to keep image data before and after unlocking. Therefore, it is possible to store, in the HLP support information, the information that indicates whether there is a need to keep the images before unlocking for the use of visualization The MPEG-4 AVC standard defines the maximum size of a temporary memory (DPB: Memory Temporary decoded images) required to store reference images or images to be displayed as the decoding results. Therefore, with a memory temporary DPB that has the maximum size or a temporary memory that It has the maximum prescribed size per application, it is possible to indicate if decoding processing can be done without failures even in the case of storing images for viewing The reference images. Note that, in order to store the images before unlocking the reference images, is possible to indicate the size of the temporary memory to be secured, in addition of the necessary size as a DPB, using the number of octets or the number of frames Here, whether or not the unlocking is performed on each image can be known from the information in the flow, or the information outside the flow such as management information. In the case of obtaining the information in the flow, for example, You can get it from an SEI. In addition, in the case of decoding of an MPEG-4 AVC stream, is possible to evaluate if the images, before unlocking the reference images, can be used to visualize or not, based on the size of temporary memory that can be used in the decoding unit and the information above described, and then it is possible to determine how to visualize the images.

Note that all information, or a part of the information, can be included as supporting information HLP It is also possible to include necessary information based on a predetermined condition, for example, include information in Regarding the presence or absence of reproduction information trick only in the case where there is no relative requirement to the prediction structure. Likewise, the different information of the information described above can be included in the HLP support information.

Fig. 24B does not directly indicate the information related to the structure of a flow, but indicates if a flow satisfies the requirements on flow structures prescribed by the Blu-ray disc standard (BD-ROM) or High Definition DVD (HD) standard, which is the norm for storing high definition images in a DVD. Similarly, in the case where different modes are defined as the requirements of a flow in an application standard such as the standard BD-ROM or similar, the information that indicates the Applied mode can be stored. For example, the following modalities: mode 1, which indicates that there is no requirement; mode 2, which indicates that the flow has a random access structure and includes information for specify the AU to decode at the time of playback trick and the like Note that it is possible to indicate whether the flow satisfies the requirements prescribed in the communication service, such as discharge or emission of flow, or a standard of diffusion.

Note that it is possible to indicate both the information shown in Fig. 24A as the information shown in Fig. 24B. Likewise, in the case where it is known that the flow satisfies the requirements in a specific application standard, it is possible to store the requirements in the application standard converting the flow structure to the format for the description direct, as shown in Fig. 24A, instead of indicating whether the flow satisfies the application standard.

Note that it is possible to store the information indicating the AUs to decode or display in the Trick play time as management information. Likewise, in the case where the information content of HLP support is switched in a flow, you can indicate the information HLP support section by section.

Fig. 26 is a flow chart showing the operations of the image multiplexing apparatus 108 in movement. In Step 51, the unit 101 for determining flow attributes determines the TYPE based attribute information in the values set by the user or the conditions predetermined In Step 52, the encoding unit 102 encodes a flow based on the TYPE attribute information. In the Step 53, the support information generation unit 105 of Playback generates HLP support information based on the TYPE attribute information. Consequently, in Stage 54, the encoding unit 102 generates access information about the base of a coded flow access unit, and the unit 104 general management information generation generates the INFO management information by adding the access information to the other necessary information (general management information). In the Step 55, the multiplexing unit 106 multiplexes a flow, the HLP support information and INFO management information. In the Step 56, the storage unit 107 records the data multiplexed Note that Step 53 can be performed before Stage 52, or after Stage 54.

Note that the coding unit 102 can store the information shown in the support information  HLP in a flow. In this case, the information shown in the HLP support information is stored in the NAL unit for Store trick play. For example, in the case where P images do not refer to B images, it is possible to decode only one image I and P images at the time of playback  at variable speed. Therefore, information from indicator that indicates if only one image I and P images are They can decode and display. Similarly, there is a case where some AU to decode at the time of playback to variable speed cannot get an SPS or a PPS from AU to which should refer to the respective AU. It is the case where the PPS referenced by an image P is stored only in the AU of an image B in the case of decoding only one image and Q images. In this case, there is a need to obtain the PPS necessary to decode the P image from the AU of a image B. Therefore, it is possible to include indicator information which indicates whether the SPS or PPS to which each AU refers to decode at the time of variable speed playback will you can get safely from one of the other AU to decode at the time of variable speed playback. Doing it this way makes it possible to perform the operation as detect an SPS or a PPS also from the AU of an image not intended to be decoded at the time of playback to variable speed only in the case where a indicator. Likewise, at the time it is shown that only an image I and images P can be decoded and displayed, it is possible to adjust the playback speed by also decoding B images, especially reference B images to the ones they do Reference other images.

Likewise, it is possible to store the information of indicator in the heading of another NAL unit such as a SPS, a PPS or a segment, instead of using any NAL unit to store trick play. For example, in the case where an SPS referenced by an AU that constitutes a RAU unit Random access is stored in the upper AU on the RAU drive random access, the nal_ref_idc field of the NAL unit of a SPS can indicate the indicator information. How is it defined that the nal_ref_idc value is 1 or more in the NAL unit of an SPS, it is It is possible to always set the higher order bit to 1 and indicate the indicator information using the lower order bit.

Note that the information content of HLP support can be stored in a stream or in information from management, or both. For example, the content can be displayed in management information in the case where the content of the HLP support information is fixed in a stream, while the content can be displayed in a flow in the case where the Content is variable. It is also possible to store the indicator information indicating whether support information HLP is fixed or not in the management information. Likewise, in the case where HLP support information is predetermined in a application standard such as a BD-ROM or a RAM, or in the case where HLP support information is provides separately by communication or dissemination, the HLP support information cannot be stored.

Decoding apparatus for moving images

Fig. 27 is a block diagram of the apparatus 200 decoding of moving images performed by the decoding procedure of moving images of the present invention This image decoding apparatus 200 in motion reproduces a coded stream shown in Fig. 8A and 8B to 20. You can perform not only normal playback, but also trick play, such as jump play, Variable speed playback and reverse playback. He motion picture decoding apparatus 200 includes, in addition, an EXT flow extraction unit and an AUsel unit of selection of the AUs to decode, in addition to the units of a conventional decoding apparatus 2 shown in Fig. 5. Note that the processing units that perform the same operations as the respective processing units of the conventional decoding apparatus 2, shown in the block diagram of Fig. 5, they are assigned the same numbers of reference, and their descriptions will be omitted.

The AUsel unit for selecting AUs to decode determines the AUs required to decode based in the GrpInf information of decoded trick play in the VLD unit of variable length decoding, according to a trick play instruction introduced from outside. Here, the instruction indicating trick play is introduced from the AUsel unit to select the AUs to decode. In addition, the AUsel unit for selecting the AUs to decode notifies the EXT flow extraction unit about DecAU, which is the information indicating the AU determined as the necessary AU to decode. EXT flow extraction unit extracts only the flow corresponding to the AUs that are evaluated as the AU needed to decode by the AUsel unit for selecting the AUs to decode, and then transmit the stream to the VLD unit of variable length decoding.

Fig. 28 is a flow chart of the way in which the motion decoding apparatus 200 (mainly the AUsel unit of selection of the AU to decoding) shown in Fig. 27 performs the procedure of decoding a stream that includes playback information trick at the time of trick play.

First, in stage 20, the AUsel unit selection of the AU to decode evaluates if the AU is the AU top of a random access RAU unit detecting an SPS, or similar, in the flow. In the case where the AU is the upper AU, it goes on to Stage 21, while in the case where the AU is not the AU upper, go to Stage 22. Here, the starting position of the RAU random access unit can be obtained from management information, such as temporary map. Especially in the case where the playback start position is determined in the moment of skip playback, or only the top image of the random access RAU drive and performed High speed playback on the top image selected, it is possible to determine the starting position of the RAU random access unit that refers to the temporary map. In Stage 21, the AUsel unit for selecting the AUs to decode get the trick play information from the AU data, analyze the AU data and determine the AU to decode before moving on to Stage 22. In Stage 22, the AUsel unit to select the AU to be decoded evaluates whether the AU is the AU that is determined in Stage 21 as the AU a decode. In the case where the AU is determined, the device 200 motion picture decoding decodes the AU in Stage 23, while in the case where AU is not determined, it goes to Stage 24. In Stage 24, the apparatus 200 of decoding of moving images evaluates if any AU remains to decode. In the case where there is an AU, the apparatus 200 of decoding of moving images repeats the processing of Stage 20, and the following stages, while in the case where there is no AU, the processing is completed. Note that it is possible to skip the process of Stage 21 and Stage 22, or skip the determination processing in Step 21, and send the information indicating that all AUs are decoded in the normal playback time, where all AUs are decode and display in order.

Fig. 29 is a flow chart indicating the processing (processing by the AUsel unit for selecting the AUs to decode) in Step 21. First, the unit AUsel for selecting the AU to decode detects the position of start of a NAL unit that constitutes an AU, looking at the AU data a start code prefix, starting with the upper octet in Stage 30, and go to Stage 31. Note that You can search for a start code prefix starting, not the octet top of the AU data, but another position, such as the final position of an Access Unit Delimiter. On stage 31, the AUsel unit for selecting the AUs to be decoded obtains the type of NAL unit of a NAL unit, and goes to Step 32. In Stage 32, the AUsel unit for selecting the AUs to decode assesses whether the type of NAL unit obtained in Step 31 is the type NAL unit to store trick play information. In the case where the trick play information is stored, go to Stage 33, while in the case where the Trick play information is not stored, repeat the Processing of Stage 30 and the following stages. Here in the case where the trick play information is stored in an SEI message, the AUsel unit for selecting AUs to decode gets the NAL unit of an SEI first, and also evaluates whether the SEI message to store the information of Trick play is included or not in the NAL unit. On stage 33, the AUsel unit for selecting the AUs to be decoded obtains trick play information, and go to Stage 34. In the Step 34, the AUsel unit for selecting the AUs to be decoded determine the necessary images to decode at the time of Perform a specified trick play operation. By example, as long as double play is specified speed. In the case where the trick play information indicates that double speed playback is possible decoding and reproducing only one image I, images P e B reference images, it is determined that these three types of Images are decoded and reproduced. Note that, in the case where the trick play information is not detected in the top image of the random access RAU drive in the Processing from Stage 30 to Stage 32, the images necessary to decode in order to perform the operation of trick play are determined according to a procedure predetermined. As an example, it is possible to assess whether the image is a reference image or not referring to the field indicated by the type of image of an image in an Access Unit Delimiter, or checking the nal_ref_idc of the NAL unit header. For example, it is possible to distinguish reference B images from non-reference B images referring to both field that indicates the types of images as a nal_ref_idc.

Fig. 30 is a flow chart indicating the processing (processing by the AUsel unit of selection of AUs to decode) in the case where all AUs to Decode is not always displayed. To the stages to perform the same processing as the stages in the flow chart of the Fig. 28 are assigned the same reference numbers, and are will omit your description. In Step 41, the AUsel unit of selection of the AUs to decode obtains and analyzes information from trick play, determines the AU to decode and the AU to display in a specified trick play operation, and goes to Stage 42. In Stage 42, the AUsel unit for selecting the AU to decode evaluates if the AU to decode match completely with the AU to display. In the case where there is a complete coincidence, go to Stage 22, while in the case where there is no complete match, go to Stage 43. In Stage 43, the AUsel unit for selecting the AUs to decode sends information from the list of the AUs to be displayed, and goes to the Stage 22. The list information of the AUs sent is used in one stage (not shown in a figure) to determine the AU to display between decoded AUs.

Note that, in the AVC of MPEG-4, it is possible to use, as images of reference, images on which processing is performed filtering (unlocking) to eliminate block distortion after decoding, in order to improve the quality of image, and it is possible to use, as display images, Images before unlocking. In this case, the apparatus 200 of decoding of moving images you need to keep the Image data before and after unlocking. Here, on condition of that the motion picture decoding apparatus 200 have a memory that can store data post-decoding equivalent to four images, in the case where you store the image data before and after Unlock in memory, memory needs to store data equivalent to two images, in order to keep images before Unlock reference images. However, as has been described above, it is desirable that they can be contained in the memory as many images as possible at the time of reverse reproduction. Provided that the apparatus 200 of decoding of moving images use images After unlocking also for display use, you can contain data from four images in a memory because it is not You need to store images before unlocking. Thus, display images before unlocking, in order to improve the image quality at the time of playback in one direction normal and display images after unlocking at the moment of reverse playback, makes it possible to keep more images in a memory, and reduce the magnitude of processing in The moment of reverse playback. For example, in the example of Fig. 15A to 15C, which show a list of the AUs of a image I and P images as trick play information, all data from four images can be kept in a memory at the time of reverse playback, while the following sets of two images, which are selected arbitrarily between I0, P3, P6 and P9, they can be maintained in the memory at the same time at the time of playback in a normal address: 10 and P3; P3 and P6; and P6 and P9.

Example of a trick play recording format in a Optical disc

A trick play function is especially important in an optical disc device that reproduces  Package means Here, an example of recording of trick play information described above in a Blu-ray disc (BD) which is an optical disc of the Next Generation.

First, a format of BD-ROM recording.

Fig. 31 is a diagram indicating the BD-ROM structure, especially structures of a BD 114 disk, which is a disk carrier, and data 111, 112 and 113 stored on disk. The data stored in the BD 114 disk includes data from VA 113, management information 112 BD, such as management information related to VA data and a VA playback sequence, and a program 111 of BD playback that performs interactivity. Here, for reasons for convenience, the description of the BD disk will be focused on the VA application to play audio and visual content of movies, but you can make a similar description focused in another use.

Fig. 32 is a diagram showing the structure of a stored logical data directory file on the BD disc described above. A BD disk has an area of recording from your innermost radio to your outermost radio like, for example a DVD, a CD and the like, and has space for logical addresses to store logical data between the input by reading in the innermost radius and the output by reading in the outside radio Likewise, inside the entrance by reading, there is a special area that can only be read by a controller called Burst Cut Area (BCA). Like this area cannot be read from the application, it can be used, for example, for a property protection technique intellectual.

File system information (volume) It is stored on top of the address space Logic, and application data such as video data are They also store there. As described in the prior art, A file system is, for example, the UDF or ISO9660, and allows the reading of the stored logical data using a directory structure or a file structure as in the case of a normal personal computer.

In this embodiment, as the structure of directory and file structure on the BD disk, is placed the BDVIDEO directory immediately below a directory root (ROOT). This directory is a directory that stores data such as VA content or management information (101, 102 and 103, which are described in Fig. 32) which is managed in the BD.

Below the BDVIDEO directory, the Next seven files.

(i)

BD. INFO (the file name is fixed) which is a fragment of the "BD management information" and is a file that stores the information related to the entire BD disk. He BD player reads this file first.

(ii)

BD.PROG (the file name is fixed) which is one of the "BD playback programs" and is a file that stores playback control information relative to everything the BD disk.

(iii)

XXX PL ("XXX" is variable, and the extension "PL" is fixed) which is an element of "management information of BD "and is a file that stores the list information of reproduction, which is a script (sequence of reproduction). Every Playlist has a file.

(iv)

XXX PROG ("XXX" is variable, and the extension "PROGR" is fixed) which is one of the "playback programs  of BD "and is a file that stores the control information of prepared reproduction based on playlists. The corresponding playlist is identified on the basis of a file body name (based on the match of "XXX").

(v)

YYY.VOB ("YYY" is variable, and the extension "VOB" is fixed) which is one of the "VA data" and is a file that stores the VOB (same as the VOB described in the prior art). Each VOB has a file.

(saw)

YYY.VOBI ("YYY" is variable, and the extension "VOBI" is fixed) which is an element of the "information of BD management "and is a file that stores the information of flow management relative to the VOB that is the VA data. The list corresponding reproduction is identified based on a name file body (based on a match of "YYY").

(vii)

ZZZ.PNG ("ZZZ" is variable, and the extension "PNG" is fixed) which is one of the "VA data" and is a file that stores PNG image data (which is a format of image normalized by the W3C and called "ping") for constitute subtitles and menus. Each PNG image has a file.

BD navigation data structure (BD management information) will be described with reference to the Fig. 33 to 38.

Fig. 33 is a diagram showing the internal structure of a VOB management information file ("YYY.VOBI"). VOB management information has the VOB flow attribute (Attribute) information and a map Temporary (TMAP). The stream attribute has video attribute (Video) and audio attribute (Audio # 0 to Audio # m) separately. Especially in the case of audio stream, as a VOB has various audio streams at the same time, the presence or absence of a data field is indicated by the number (Number) of flows of Audio.

The following are video attributes (Video) stored in fields respectively, and the values that the fields Respectives may have:

(i)

Compression format (Coding): MPEG-1; MPEG-2; MPEG-4; and AVC (Advanced Video Encoding) of MPEG-4

(ii)

Resolution (Resolution): 1920 x 1080; 1440x1080; 1280x720; 720x480, and 720x565.

(iii)

aspect ratio (Aspect): 4 to 3; and 16 to 9.

(iv)

frame rate (frame rate): 60; 59.94 (60 / 1,001), 50; 30; 29.97 (30 / 1,001), 25; 24; and 23,976 (24 / 1,001).

The following are audio attributes (Audio) stored in fields respectively, and the values that respective fields may have.

(i)

Compression format (Coding): AC3; MPEG-1; MPEG-2; and LPCM.

(ii)

the number of channels (Ch): 1 to 8

(iii)

Language attribute (Language):

       \ vskip1.000000 \ baselineskip
    

The temporal map (TMAP) is a table for store information about for each VOBU, and have the number of the VOBUs that the VOB has and the respective fragments of VOBU information (VOBU # 1 to VOBU # n). The respective fragments of VOBU information include I_inicio, which is the address (the start address of an image I) of the upper TS packet of a VOBU and a direction of travel (I_fin) to the address Image end I, and playback start time (PTS) of the image I.

Fig. 34 is a diagram illustrating the VOBU information details. As it is widely known, since variable bit rate compression can be performed in the MPEG video stream, in order to record the video stream in High quality, there is no proportionality between the time of Reproduction and data size. On the other hand, as I know performs a fixed bit rate compression in AC3, which is a Audio compression standard, the relationship between time and Address can be obtained from a primary expression. Without However, in the case of MPEG video data, each frame has a fixed display time, for example, a frame has a display time of 1 / 29.97 seconds in the case of NTSC, but  the size of the data after compressing each frame changes in large extent, depending on the characteristic of the image, or the type of image used in compression, such as an image I, an image P or an image B. Therefore, in the case of an MPEG video stream, it is impossible to represent the relationship between time and direction using a primary expression.

As you can expect, it is impossible to represent the relationship between time and data size using a primary expression in an MPEG system flow where data from MPEG video is multiplexed, that is, a VOB. Therefore a map Temporary (TMAP) associates time with the address in a VOB.

In this way, in the case where it occurs time information, the VOBU to which time belongs Search first (following the VOBU PTS in order), it jumps to the PTS immediately before that time in the VOBU that a TMAP has (the address specified by I_inicio), the decoding starts with the upper image I of the VOBU, and the visualization starts with the image corresponding to that weather.

Next, the internal structure of a Playlist information ("XXX.PL") will be described with reference to Fig. 35. The playlist information includes a list of cells (List Cells) and a list of events (Events List).

The cell list (Cell List) is a sequence of playback cells in the playlist; Y the cells are reproduced in the order of description indicated in this list. The content of the cell list (Cell List) is the number of cells (Number) and the information of each cell (Cell # 1 a Cell #n).

The cell information (Cell #) has a number VOB file (VOB Name), start time (In) and end time (Exit) in the VOB, and subtitles (TableSubtitles). Start time (In) and the final time (Sal) are represented as a frame number in each VOB. It is possible to obtain the VOB data address needed to reproduce using the time map (TMAP) previously described.

The subtitle table (Subtitle Table) is a table that stores subtitle information that is played in sync with the VOB. As in the case of audio, they are included Various languages in subtitles. The first information of the Subtitle table (Subtitle Table) includes the number of languages (Number) and the following tables (Language # 1 to Language # k) prepared On a language basis.

Each language table (Language #) includes language information (Language), the number (Number) of fragments of subtitle information of the subtitles to be displayed by separate, and subtitle information (Speech # 1 to Speech # j) of Subtitles to be displayed separately. Subtitle Information (Speech #) includes an image data file name (Name), the subtitle display start time (In), the  subtitle display end time (Exit) and a Subtitle display position (Position).

The event list (EventList) is a table which defines each event that occurs in the playlist. The event list includes the number of events (Number) and the respective events (Event # 1 to Event # m). Each event (Event #) includes an event type (Type), an event identifier (ID), a time of occurrence of the event (Time) and a duration of event (Duration).

Fig. 36 is an event management table ("XXX. PROG") that has an event manager (which is an event temporary and an event for menu selection) prepared on the basis of a playlist. The event management table includes the number of event managers / programs defined (Number) and the respective event managers / programs (Program # 1 to Program # n). The content of each manager / program of events (Program #) is the definition of the start of a manager event (event_handler tag) and the manager id of events (ID) that matches the event ID previously described, and following it, the program described in "{}" that follows the word Function. He event (Event # 1 to Event # m) stored in the event list (EventsList) of the "XXX.PL" described above is specified using an identifier (ID) of the event manager of "XXX. PROG ".

Next, the internal structure of the information regarding the entire BD disc ("BD.INFO") is will describe with reference to Fig. 37. Information regarding The entire BD disc includes a list of titles (ListTitles) and a event table for global events (EventList).

The title list (Title List) includes the number of titles of a disc (Number) and fragments of information of title (Title # 1 to Title # 3) that follow the number of titles. The respective fragments of title information (Title #) include a playlist table in the title (TablaPL) and a chapter list in the title (Chapter List). The list table Playback (TablePL) includes the number of lists of reproduction in the title (Number) and names of lists of reproduction (Name) which are the file names of lists of reproduction. The chapter list (Chapter List) includes the number of chapters included in the title (Number) and fragments of chapter information (Chapter # 1 to Chapter # n). Each fragment of  Chapter information (Chapter #) includes a table of cells (Table Cells) included in the chapter, and the table of cells (TableCells) includes the number of cells (Number) and fragments of cell input information (Cell Input # 1 to Cell Input # k). The cell input information (Cell Input #) includes the playlist name that includes the cell and a number of cells in the playlist.

The event list (EventList) includes the number of global events (number) and information fragments of global events It should be noted that the global event to be defined First, it is called the first event (First Event), and it is the event invoked first after insertion of BD disk in the player. The event information for the events global has only one type of event (Type) and a event identifier (ID).

Fig. 38 is a table ("BD.PROG") of a program of a global event manager. The content of this table is the same as the content of the event management table described in Fig. 36.

In the case of storing the information of trick play described above in the format BD-ROM described so far, it is considered that a VOBU includes one or more random access RAU units, and the Trick play information is included in the upper AU of VOBU Note that, in the MPEG-4 AVC, includes a NAL unit where information is stored trick play

Note that the reproduction information Trick can be stored in the BD management information. By example, it is possible to store trick play information prepared in VOBU to VOBU, extending the temporal map of the VOB management information. It is also possible to define a new map to store playback information trick

Likewise, it is possible to store the information of trick play either in the VOBU or in the information of BD management.

It is also possible to store only the default value of the trick play information in the BD management information, and only in the case where the trick play information, as for the VOBU, is other than the default value, it is possible to store the information of trick play in the VOBU.

It is also possible to store a set of one or more fragments of trick play information in the BD management information as the information that is common among the flows. The VOBU can refer to a piece of information of trick play between information fragments of trick play stored in the management information of the BD. In this case, the index information of the information of Trick play reference object by the VOBU is stored in the management information of a VOBU unit or in the VOBU.

Player to play optical discs

Fig. 39 is a block diagram showing in general terms the functional structure of a player that plays a BD disc shown in Fig. 31, and the like. The data on the BD 201 disc are read by an optical player 202. Read data is transmitted to an exclusive memory that It depends on the types of the respective data. The program of BD playback (the content of "BD.PROG" or "XXX.PROG") is transmitted to a program memory 203. Similarly, the BD management information ("BD.INFO", "XXX.PL" or "YYY.VOBI") is transmitted to a memory 204 of management information Likewise the AV data ("YYY.VOB" or "ZZZ.PNG") is transmitted to an AV 205 memory.

The BD playback program recorded in the program memory 203 is processed by a program processing unit 206. Similarly, the management information of the BD recorded in the management information memory 204 is processed by the management information processing unit 207. Likewise, the AV data recorded in the AV memory 205 is processed by a presentation processing unit 208
tion.

The program processing unit 206 receives the playlist information to be played by the unit 207 of management information processing and the event information, such as execution schedule of the program, and performs the program processing. Equally, it is possible to dynamically change playlists to Play by the program. This can be done by sending a playback instruction of playlists at 207 unit of management information processing. Unit Program processing 206 receives an event from a user; In other words, you receive a request through a controller remote, and in the case where there is a program that corresponds to the event user, run the program.

The information processing unit 207 of management receives an instruction from processing unit 206 of program, analyze playlists and information on management of the VOBs that correspond to the playlists, and instructs presentation processing unit 208 to Play the desired AV data. Similarly, unit 207 of Management information processing receives information from standard time from processing unit 208 of presentation, instructs the processing unit 208 presentation to stop playback of AV data based in time information. Similarly, unit 207 of Management information processing generates an event for notify program processing unit 206 of Program execution schedule.

The presentation processing unit 208 it has a decoder that can process video, audio and subtitles / images (still images), respectively. Decode and produces the AV data according to an instruction of unit 207 of management information processing. In the case of data from video, and subtitles / images, are decoded and then they represent in the respective exclusive planes, that is, the plane 210 video and image plane 209. After this, the unit 211 synthesis processing performs the processing of synthesis about the video, and sends the video to a device display such as a television.

At the time of trick play, such such as skip playback, variable speed playback and reverse reproduction, the processing unit 208 of presentation interprets the trick play operation requested by the user, and notifies unit 207 of information management information management such as playback speed The processing unit 207 management information analyzes the reproduction information trick stored in the upper AU of the VOBU and determines the AU to decode and display so that the operation of trick play specified by the user can be performed in a safe way. Note that the processing unit 207 of management information can get the playback information trick, send it to the presentation processing unit 208 and determine the AU to decode and the AU to display in the 208 presentation processing unit.

Note that an autonomous computer system you can easily execute the processing shown in this realization, recording the program to perform the procedure of coding of moving images and the procedure of decoding of motion pictures shown in this realization on a recording medium such as a disc flexible.

Fig. 40A to 40C are illustrations of the mode in that the computer system executes the coding procedure of moving images and the decoding procedure of moving images of this embodiment using a program recorded on a recording medium such as a floppy disk.

Fig. 40A shows an example of a format Physical of a floppy disk as a recording medium. Fig. 40B shows a floppy disk and the front view and the view in cross section of the aspect of the floppy disk. A disc flexible (FD) is contained in a housing F, a plurality of tracks (Tr) are formed concentrically on the surface of the disk from the outer radius to the inner radius of the disk, and Each track is divided into 16 sectors (Se) in the angular direction. Therefore, in the case of the floppy disk that stores the program described above, the program is recorded in an area assigned to it on the floppy disk (FD).

Similarly, Fig. 40C shows the structure to record and play the program on the floppy disk. At case of recording the previous program to perform the procedure Encoding of moving images and the procedure of decoding of moving images on the floppy disk FD, a computer system Cs writes the program on the disk flexible through a flexible disk drive. Likewise, in the  case of building the previous image coding apparatus in motion and the previous image decoding apparatus moving to perform the coding procedure of motion pictures and the decoding procedure of motion pictures using the program on the floppy disk, the program is read from the floppy disk through the drive floppy disk, and is transmitted to the computer system.

Note that the above description has been made using a floppy disk as a recording medium, but The program can be recorded on an optical disc. Likewise a Recording media is not limited to this, and you can use another recording media such as an IC card or a memory cassette ROM while you can record the program.

Up to this point, the apparatus for generating flow of moving images, the coding apparatus of moving images, the image multiplexing apparatus in motion and image decoding apparatus in movement of the present invention have been described based on the embodiment, but the present invention is not limited to this realization. For example, the present invention includes the following In this embodiment: (i) a flow generating apparatus of moving images; an optical disc recording apparatus that has one among a moving image coding apparatus and a decoding apparatus for moving images; a apparatus for sending moving images; an apparatus of broadcasting of digital television; a web server; a communication apparatus; a mobile information terminal; Y Similar; and (ii) a moving image receiving device which has a decoding apparatus for moving images; a digital television broadcast reception apparatus; a communication apparatus; a mobile information terminal; Y Similar.

Note that the functional blocks respective shown in Figs. 21, 23, 27 and 39 are performed typically as an LSI, which is a large integration circuit scale. Each of the functional blocks can be manufactured in a single chip, or a part of, or all, the functional blocks can be integrated into a single chip (for example, the blocks functional except a memory can be manufactured on a single chip). The integrated circuit is called LSI here, but it can be called IC, LSI system, super LSI or ultra LSI, depending on the level of integration. Likewise, the manufacturing process of themselves in an integrated circuit is not limited to the procedure of your manufacturing in an LSI, it can be done through a circuit exclusive or a generic processor. It is also possible to use (i) a reconfigurable processor where connection or configuration of circuit cells can be reconfigured or (ii) an FPGA programmable (Formation of programmable gates in the field), after manufacturing them in an LSI. In addition, in the case where the technique of manufacturing in an integrated circuit, instead of its manufacturing in an LSI, it appears when it develops, in addition, the semiconductor technique or any derivative technique appears, to in due time functional blocks can be manufactured in a integrated circuit that uses such a new technique. The application of Biotechnics is likely. Likewise, between the functional blocks respective, a storage unit (an image memory) in which stores the image data to be encoded or decode, can be configured separately instead of included on a single chip

Industrial applicability

The present invention can be applied as: a apparatus for generating movement of images in motion that generates a moving image to play in trick play; a motion image coding apparatus that generates, by encoding, a moving image to reproduce in "trick play"; an image multiplexing apparatus in motion that generates, by multiplexing packets, an image in motion to reproduce in trick play; and an apparatus of decoding of moving images that reproduces the image in motion in trick play; and especially as a apparatus for constructing the system to reproduce an AVC flow of MPEG-4 that uses a trick play mode, such such as variable speed playback and playback conversely, such apparatus being, for example, an apparatus related to  an optical disc on which its function of trick play

Claims (7)

1. A flow generating apparatus of moving images that generate a flow that includes images that they constitute a moving image, said apparatus comprising Flow generation of moving images:

a first coding unit usable to encode a first supplementary information included in an access unit random, including the random access unit a group of images whose first image is an image I, (i) including the first supplementary information plural fragments of information  of image types, which indicates types of images included in the group, and (ii) being used when the images included in the random access unit are played in trick play, and the plural fragments of image type information being placed in an order that corresponds to a decoding order of the group of images;

A second coding unit usable to encode a second supplementary information included in the access unit randomized, (i) including the second supplementary information plural fragments of image structure information, which indicates field structures or frames of the images included in the group, and (ii) being used when the images included in the random access unit are played in trick play, and being the plural fragments of information on structures of image placed in the order that corresponds to the order of decoding of the group of images; Y

a unit of usable generation to generate a flow of images in movement, adding coded supplementary information first and second to a coded image corresponding to the first image I which is an initial image of the access unit random.

2. A flow generation procedure of moving images to generate a flow that includes images which constitute a moving image, said saying Procedure of generating flow of moving images:

a first coding stage of the first supplementary information included in a random access unit, including the unit of random access a group of images whose first image is a image I, (i) including the first supplementary information plural fragments of image type information, which indicates types of images included in the group, and (ii) being used when the images included in the random access unit are reproduce in trick play, and the fragments being plurals of information of image types placed in an order corresponding to a decoding order of the group of images;

A second coding stage to encode the second information supplementary included in the random access unit, (i) including the second supplementary information plural fragments of image structure information, which indicates structures of fields or frames of the images included in the group, and (ii) being used when the images included in the access unit randomly reproduced in trick play, and being plural fragments of image structure information placed in the order that corresponds to the decoding order of the group of images; Y

a stage of generation to generate a flow of moving images by adding the first and second coded supplementary information to a coded image corresponding to the first image I, which is a Initial image of the random access unit.

3. An image decoding apparatus in movement that decodes a stream that includes images encoded that constitute a moving image and reproduces the decoded stream, said decoding apparatus comprising of moving images:

a unit of obtaining instructions usable to obtain an instruction which indicates that reproduction should take place trick

a unit of usable analysis to analyze a first information supplementary and a second supplementary information for the random access unit, demultiplexing the information supplementary first and second of a first image I, performing the analysis for each random access unit, the random access unit including a group of images whose first image is an image I;

a unit of playback image specification usable for specify images, among the images included in the unit random access, which are necessary for trick play indicated by the instruction obtained by said obtaining unit of instructions, based on a result of the analysis performed by said unit of analysis; Y

a unit of decoding usable to decode images specified by said image specification unit of Playback and play decoded images,

in which the first supplementary information in the random access unit includes plural fragments of image type information, which indicates types of images included in the access unit randomized, and plural fragments of information types of image are placed in an order that corresponds to an order of decoding of the group of images; and the second information supplementary included in the random access unit includes plural fragments of image structure information that indicates field structures or frames of the images included in the random access unit, and the plural fragments of the Image structure information is placed in the order that corresponds to the decoding order of the group of images, Y

information supplementary first and second are added to an encoded image corresponding to the first image I which is the initial image of the random access unit, and are used when images included in the random access unit are played in trick play

4. A decoding procedure of motion pictures to decode a stream that includes encoded images that constitute a moving image and reproduce the decoded stream, comprising said decoding procedure of moving images:

a stage of obtaining instructions to obtain an instruction that indicates that trick play should be carried out;

a stage of analysis to analyze a first supplementary information and a second supplementary information for the access unit randomized, demultiplexing the supplementary information first and second of a first image I, performing the analysis for each random access unit, including the access unit random a group of images whose first image is an image I;

a stage of Specification of playback images, to specify images, among the images included in the access unit random, which are necessary for trick play indicated for the instruction obtained in said stage of obtaining instructions, based on a result of the analysis performed on said stage of analysis; Y

a stage of decoding to decode the images specified in said stage of specifying reproduction images and play decoded images,

in which the first supplementary information in the random access unit includes plural fragments of image type information, which indicates types of images included in the access unit randomized, and plural fragments of information types of image are placed in an order that corresponds to an order of decoding of the group of images; and the second information supplementary included in the random access unit includes plural fragments of image structure information that indicates field structures or frames of the images included in the random access unit, and the plural fragments of the Image structure information is placed in the order that corresponds to the decoding order of the group of images, Y

information supplementary first and second are added to an encoded image corresponding to the first image I which is the initial image of the random access unit, and are used when images included in the random access unit are played in trick play

5. A computer readable recording medium over which a flow is recorded, including the flow images and first and second supplementary information,

in which the flow is structured so that the first information supplementary and the second supplementary information are added to an encoded image corresponding to a first image I, which is a first image of a random access unit, and are used when the images included in each random access unit are play in trick play, including the access unit random a group of images whose first image is an image I, being the first supplementary information included in the unit of  random access plural fragments of information types of image, included in the random access unit, and being the plural fragments of the information of placed image types in an order that corresponds to a group decoding order of images, and the second supplementary information being included in the random access unit plural fragments of information of image structures, which indicates field or frame structures of the images included in the random access unit, and the plural fragments of the information of types of image placed in the order that corresponds to the order of decoding of the group of images.

6. A recording procedure to record a flow that includes images that constitute a moving image on a computer-readable recording medium, comprising said recording procedure

a stage of recording to record the generated flow using the procedure of flow generation of moving images according to the claim 2.

7. An image decoding system in movement comprising the recording medium according to the claim 5, and the image decoding apparatus in movement according to claim 3, which reads a flow that includes encoded images that constitute a moving image of the Support recording, decode and play the stream.