JP2008067364A - Image decoding apparatus and its control method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To avoid a fatal error in the decoding process even when the size of a reference picture buffer is less than the necessary capacity. <P>SOLUTION: An access unit to be decoded is selected from data stored in a coded picture buffer (CPB) 56. When the access unit is an IDR (Instantaneous Decoder Refresh) picture, it is checked whether a decoded picture buffer DPB 60 has a free area enough to store the access unit. When the DPB 60 has a sufficient free area, a decoder 58 stores the decoding result in the DPB 60, and selects the next access unit from the CPB 56. When the DPB 60 does not have a sufficient free area and the earliest stored picture has already been displayed, the buffer area of the displayed picture is deallocated. When the earliest stored picture has not been displayed, a warning is displayed to the user. All memory areas in the DPB 60 except for the area of a currently displayed picture are deallocated to wait for an IDR picture. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image decoding apparatus. The present invention relates to an image decoding apparatus for decoding compressed image data compressed by inter-picture predictive encoding such as H.264 and a control method thereof.

  A video recording / reproducing apparatus such as a digital video camera for recording a still image and a moving image on a large-capacity optical disk instead of a DVD (Digital Versatile Disc) as a recording medium has been studied. In addition to increasing the recording capacity of media, the next-generation optical disc standard also incorporates the use of a moving image compression encoding technique with a higher compression rate. H. One of them is H.264. H. H.264 is also referred to as MPEG (Moving Picture Experts Group) -4 Part 10: AVC.

  H. Because of the high compression in H.264, it is allowed to have more reference picture memory used for inter-picture prediction than in conventional techniques such as MPEG2, and the degree of freedom of rearrangement of decoded pictures is high.

  For example, in MPEG2, a P picture is always used as a reference picture, and a B picture cannot be used as a reference picture. Therefore, the MPEG2 decoder can determine whether the decoded picture needs to be stored in the reference picture memory by simply checking the picture type (I, P, or B). On the other hand, H.H. In H.264, even a P picture may not be used as a reference picture, and even a B picture may be used as a reference picture.

  H. In H.264, the standard allows a specific reference picture to be used for decoding or encoding over a long period of time. That is, in MPEG2, FIFO (First In First Out) is used as a reference picture memory, and the pictures that can be referred to are limited. In contrast, H.C. In H.264, it can be said that even a picture separated in time that cannot be referred to in MPEG2 can be used as a reference picture.

  Such H. H.264 flexible inter-screen prediction is an important technique for improving encoding efficiency. On the other hand, in order to implement this function, In H.264, compared to MPEG2, more reference pictures must be stored in the reference picture memory, and more memory capacity and computation are required to rearrange them in the display order.

In order to solve such a problem, a configuration has been proposed in which the number of images to be stored in the reference picture memory is reduced by controlling the reference relationship (for example, Patent Document 1).
JP-A-2005-260588

  H. H.264 long-referenced pictures, so-called long-term reference pictures, are advantageous for compression coding scenes with little motion, such as landscape photography. However, it is not suitable for a digital video camera that encodes a subject that moves a lot in real time, such as athletic meet shooting. Further, the method described in Patent Document 1 is also not suitable for compressing and coding a video obtained by shooting a subject that moves rapidly, such as an athletic meet. When a B picture is used as a reference picture in a digital video camera, it is considered that the reference relationship becomes complicated, the calculation processing becomes heavy, and battery consumption increases.

  However, H.H. performs real-time encoding used in consumer digital video cameras. In the H.264 codec, specifications such as not using a reference picture for a long time and not using a B picture as a reference picture are conceivable. That is, H.C. For functions that are permitted in the H.264 standard profile, setting a certain amount of usage restrictions at the time of encoding can also reduce the amount of memory, reduce the computational load, reduce hardware costs, extend battery life, etc. It is considered effective.

  In such a small codec decoding, there is no problem in reproducing a self-recording stream. However, for example, the H.D. When a 264-bit stream is input, there is a risk that decoding may fail due to a lack of a reference picture storage buffer.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image decoding apparatus and a control method thereof capable of decoding without failure even when a coded bit storm having a complicated reference relationship is input. And

  In order to achieve the above object, an image decoding apparatus according to the present invention is an image decoding apparatus that decodes compressed image data compressed using inter-picture predictive coding, wherein the compressed image data is A first picture buffer to be stored, a second picture buffer to store decoded image data, and image data stored in the second picture buffer are selectively referred to, and the first picture buffer Decoding means for decoding the compressed image data stored in the buffer and storing the decoded image data in the second picture buffer; detection means for detecting a free capacity of the second picture buffer; Control means for interrupting decoding by the decoding means when the free space is less than a predetermined value according to a detection result of the detection means.

  An image decoding apparatus according to the present invention is an image decoding apparatus for decoding compressed image data compressed using inter-picture predictive coding and intra-picture coding, and stores the compressed image data. A first picture buffer, a second picture buffer for storing decoded image data, and image data stored in the second picture buffer are selectively referred to and stored in the first picture buffer. Decoding means for decoding the stored compressed image data and storing the decoded image data in the second picture buffer, the compressed image data by the inter-picture predictive encoding and the intra-picture encoding An all-decoding processing mode for decoding both of the compressed image data based on the image and a selective decoding processing mode for decoding only the compressed image data based on the intra-picture coding. According to the detection result of the detection means, the detection means for detecting the free capacity of the second picture buffer, and when the free capacity is less than a predetermined value, Control means for instructing the decoding means to perform decoding in the selective decoding processing mode.

  The image decoding apparatus control method according to the present invention is an image decoding apparatus that decodes compressed image data compressed using inter-picture predictive coding, and stores the compressed image data. And a second picture buffer for storing decoded image data, wherein the image data stored in the second picture buffer is selectively selected. Referring to the decoding step of decoding the compressed image data stored in the first picture buffer and storing the decoded image data in the second picture buffer; and A detection step for detecting free space, and control for interrupting decoding in the decoding step when the free space is less than a predetermined value according to the detection result of the detection step Characterized in that it comprises a degree, the.

The control method of the image decoding apparatus according to the present invention is an image decoding apparatus that decodes compressed image data compressed using inter-picture predictive coding and intra-picture coding, and the compressed image data A method for controlling an image decoding device comprising a first picture buffer for storing image data and a second picture buffer for storing decoded image data, the image being stored in the second picture buffer A decoding step of selectively referring to data, decoding the compressed image data stored in the first picture buffer, and storing the decoded image data in the second picture buffer; An all-decoding processing mode for decoding both the compressed image data by the inter-picture prediction encoding and the compressed image data by the intra-picture encoding; A decoding step capable of selectively executing a selective decoding processing mode for decoding only compressed image data by encoding, a detection step for detecting a free capacity of the second picture buffer, and a detection result of the detection step And when the free space is less than a predetermined value, a control step for instructing the decoding step to perform decoding in the selective decoding processing mode;
It is characterized by providing.

  According to the present invention, even if the compressed image data to be decoded has only a reference picture buffer having a buffer size relatively smaller than that required, the decoding process is continued while avoiding a fatal failure. can do.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(Example 1)
FIG. 1 is a schematic block diagram of an image pickup apparatus which is an embodiment of an image decoding apparatus according to the present invention, and FIG. 2 is a block diagram of a schematic structure of a buffer of this embodiment. The imaging apparatus 10 is, for example, H.264. H.264 (AVC) digital camera or digital video camera. In FIG. 1, a solid line indicates the flow of image data or audio data, and a broken line indicates a control signal.

  The imaging device 10 includes a photographic lens 12, an imaging element 14, a signal processing unit 16, a buffer 18, an H.P. H.264 includes a video encoding / decoding unit 20 and a flash memory 22 for encoding / decoding video data according to H.264. Further, a disk control unit 24 for writing and reading data on a recording medium 26 such as a DVD disk (hereinafter referred to as DVD), a display unit 28, and a multiplexer (MUX) / demultiplexer (DEMUX) 30 are also provided. Furthermore, the audio | voice encoding / decoding unit 36 which encodes / decodes audio | voice data according to the microphone 32, the speaker 34, and AC3, the control unit 38, the operation unit 40, and the bus | bath 42 are also provided.

  The taking lens 12 is an optical element that forms an optical image from a subject on the image pickup surface of the image pickup element 14, and includes an aperture mechanism and a shutter mechanism. The imaging apparatus 10 also includes an autofocus mechanism that automatically adjusts the focus of the photographic lens to the subject.

  The imaging element 14 is an optical element that converts an optical image from the photographing lens 12 into an electrical signal. The image sensor 14 is composed of, for example, a CCD (Charge Coupled Devices) type image sensor or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor. The signal processing unit 16 converts an analog image signal from the image sensor 14 into a digital signal, performs known signal processing (for example, gamma correction, color balance adjustment, luminance / color separation), etc., in a predetermined video format. Output video data.

  The buffer 18 temporarily stores data input / output between devices connected to the bus 42. Specific functions of the buffer 18 will be described later.

  The video encoding / decoding unit 20 is connected to the H.264 standard. The digital video signal is compressed and encoded by a video compression method such as H.264 to generate compressed video data. Further, the video encoding / decoding unit 20 expands the compressed video data from the disk control unit 24. H. H.264 is also referred to as MPEG-4 Part 10: AVC (Advanced Video Coding).

  The flash memory 22 is a non-volatile recording medium, and stores a program related to the operation of the imaging device 10.

  The disk control unit 24 writes compressed video data, compressed audio data, file management information, etc. to the recording medium 26 in accordance with instructions from the control unit 38, and stores compressed video data, compressed audio data, file management data, etc. from the recording medium 26. read out. The recording medium 26 includes, for example, a writable optical disk, a magnetic disk, a semiconductor memory (memory card), or the like such as a DVD-R.

  The display unit 28 includes an electronic viewfinder, an LCD (liquid crystal panel), or the like. In the imaging mode, an image related to the video data from the signal processing unit 16 is displayed, and in the reproduction mode, an image related to the compressed video data reproduced from the recording medium 26 by the disk control unit 24 is displayed.

  The multiplexer / demultiplexer 30 multiplexes the compressed video data by the video encoding / decoding unit 20 and the compressed audio data by the audio encoding / decoding unit 36, and separates the multiplexed compressed video data and compressed audio data. Further, the multiplexer / demultiplexer 30 adds a 4-byte header to the source packet of compressed video data or compressed audio data. Compressed video data and compressed video data are managed using this header information.

  The microphone 32 includes an AGC (Automatic Gain Control) and an A / D converter, and inputs and amplifies an external sound of the imaging apparatus 10 to generate a digital sound signal. The speaker 34 includes an amplifier and outputs a digital audio signal as sound to the outside of the imaging device 10.

  The audio encoding / decoding unit 36 compresses and encodes the digital audio signal from the microphone 32 using an AC (Audio Code number) 3 audio compression method to generate compressed audio data. The audio encoding / decoding unit 36 decompresses the compressed audio data from the disk control unit 24 and applies the decompressed audio data to the speaker 34.

  The control unit 38 includes a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and controls the entire imaging apparatus 10. The detection unit 38 a detects the entire capacity value of the buffer 18. Further, the detection unit 38a sends an enable signal to a CPB (Coded Picture Buffer) or a DPB (Decoded Picture Buffer) described later. The bus 42 is used to transfer various data between the above-described units.

  The operation unit 40 includes a moving image shooting switch, a release switch, a playback switch, and a stop switch. The user uses the operation unit 40 to instruct the control unit 38 to capture moving images, still images, and reproduce image data or audio data. Further, the operation unit 40 includes a mode dial, a menu key, a selection key, a determination key, and the like, so that the user can switch operation modes such as an imaging mode and a playback mode, display a setting screen, and display on the screen. The control unit 38 is instructed to make various selections and various decisions on the screen.

  The control unit 38 of the imaging apparatus 10 controls each unit according to the operation mode specified by the user. As a result, in the imaging mode, an image captured by the imaging element 14 is displayed on the display unit 28. In the recording mode for recording captured images, video data and audio data are compression encoded based on the video aspect ratio, compression encoding format / resolution, and the number of audio channels, etc., and multiplexed in a predetermined format. The data is recorded on the recording medium 26 in a predetermined digital moving image format. In the playback mode, the compressed video data and compressed audio data multiplexed from the recording medium 26 are read out and sequentially decoded based on the playback management information (index.bdmv, MovieObject, PlayList, etc.) and user instructions, Playback output.

  With reference to FIGS. 2 to 4, the characteristic operation of the present embodiment relating to the decoding of the compressed data and the management of the buffer 18 will be described. FIG. 2 is a schematic diagram showing processing procedures of the buffer 18 and the video encoding / decoding unit 20 during reproduction. FIG. 3 shows packet formats of the compressed video data of the video encoding / decoding unit 20 and the compressed audio data of the audio encoding / decoding unit 36. The source packet source_packet () includes a 4-byte header TP_Extra_header () and a 188-byte transport packet Transport_packet (). As shown in FIG. 4, the header TP_Extra_header () includes a 2-bit copy permission indicator copy_permission_indicator and a 30-bit arrival time stamp arrival_time_stamp. The copy permission indicator copy_permission_indicator stores information related to content protection. The arrival time stamp arrival_time_stamp stores the time when the transport packet arrives at the transport buffer 54 with the accuracy of 27 MHz. A detailed description of the transport packet Transport_packet () is described in ISO / IEC13818-1, and is omitted here.

  When receiving a reproduction instruction from the user, the imaging device 10 reads the compressed video data and the compressed audio data from the recording medium 26 and stores them in a read buffer (RB) 50. The packet separation unit 52 deletes the header TP_Extra_header () and writes the source packet that has reached the arrival time stamp (arrival_time_stamp) among the source packets stored in the read buffer 50 in the TB 54. The data stored in the transport buffer (TB) 54 is a constant rate or H.264. In accordance with the H.264 / AVC virtual decoder model, it is decomposed into elementary streams. Then, it is stored in a coded picture buffer (CPB) 56. The coded picture buffer (CPB) 56 corresponds to a first picture buffer recited in the claims.

  The decoding unit 58 of the video encoding / decoding unit 20 decodes each access unit of the elementary stream stored in the CPB 56 that stores the compressed image data. Specifically, decoding is performed based on a buffer period SEI (Buffering Period SEI) and a picture timing SEI (Picture Timing SEI). Then, the restored video data is stored in a decoded picture buffer (DPB) 60. The decoded picture buffer (DPB) 60 corresponds to a second picture buffer described in the claims. The buffer period SEI and the picture period SEI are additional information included in the elementary stream. The buffer period SEI indicates a delay time until the access unit is decoded, and the picture timing SEI indicates a decoding time and a display time for each access unit. The details of the buffer period SEI and the picture timing SEI are described in ITU-T Rec. H. Since this is described in H.264 etc., further explanation is omitted.

  The decoded video data stored in the DPB 60 is used for display and is also used as a reference picture at the time of decoding of a picture encoded using inter-frame prediction.

  There are two types of methods for managing the reference pictures stored in the DPB 60. The moving frame memory management method is a method of releasing the buffer area of the earliest stored picture among the reference pictures when there is not enough memory to store the picture in the reference picture newly stored in the DPB. On the other hand, the adaptive memory management method is a method for specifically controlling the reference picture and its operation. The reference pictures include a long-term reference picture that is referred to in the stream for a long period of time and a short-term reference picture that is referred to only for a short period of time. Depending on the occurrence of these reference pictures, the moving frame memory management method and adaptive memory management The method is switched and used.

  FIG. 5 shows an operation flowchart in the case of reproducing compressed video / audio data in which the capacity of the DPB 60 is insufficient. For example, there is a case where a recording medium 26 on which video and audio are recorded by another recording apparatus is loaded, and video and audio are reproduced from the recording medium 26.

  The decoding unit 58 selects an access unit to be decoded from the stored data of the CPB 56 (S1), and determines whether or not the access unit is an IDR (Instantaneous Decoder Refresh) picture (S2). If it is not an IDR picture (NO in S2), the decoding process cannot be started, so the access unit is abandoned and the process returns to step S1.

  If it is an IDR picture (YES in S2), it is checked whether there is sufficient free space for storing the access unit in the decoded picture buffer (DPB) 60 (S3). If there is sufficient space for storing (YES in S3), the decoding unit 58 stores the decoding result in the DPB 60 (S4), selects the next decoding target access unit from the CPB 56 (S5), Return to step S3.

  On the other hand, if there is no free space in the DPB 60, that is, if the free space is less than the predetermined value (NO in S3), it is checked whether or not the picture stored earliest has been displayed on the display unit 28 (S6). If already displayed (YES in S6), the buffer area of the displayed picture is released (S7), and the process proceeds to step S4. If not displayed (NO in S6), a warning message is displayed to the user by the display unit 28 (S8), the entire memory area on the DPB 60 is released except for the picture area currently being displayed (S9), and step S1. The process returns to the IDR picture waiting state. In this IDR picture waiting state, the picture other than the picture being displayed is released, and the picture of the only picture remaining in the DPB 60 is repeatedly displayed.

  Thus, in this embodiment, it is detected whether a sufficient buffer area for storing the reference picture can be secured in the decoded picture buffer (DPB) 60. Then, according to the detection result, when the free area becomes less than a predetermined value, the decoding is interrupted. While decoding is interrupted, the currently displayed image is repeatedly displayed and the next IDR picture is waited, so that a fatal failure of the decoding process can be avoided.

  In step S8, a warning message is issued when decoding is interrupted, but may be omitted if unnecessary. In step S9, the buffer area other than the picture being displayed is released. However, if the memory area necessary for the next decoding process can be finally secured in the decoded picture buffer (DPB) 60, this is not the only case. Not a thing. For example, undisplayed pictures on the DPB 60 may be released after being displayed sequentially, or the display of the reproduced image is abandoned and, for example, a blue background is displayed on the display unit 28, and the entire memory area of the DPB 60 is displayed at once. You may release.

  FIG. 6 shows an operation flowchart in the case of the adaptive memory management method. Steps S51 to S59 correspond to steps S1 to S9 in FIG. Steps S56 and 57 are different from FIG. In FIG. 6, when there is not enough free space in the decoded picture buffer (DPB) 60 (when the free space is less than the predetermined value) (NO in S53), it is checked whether there is a releasable picture in the DPB 60. (S56). If there is a releasable picture (YES in S56), the picture is deleted from the DPB 60, the buffer area is released (S57), and the process proceeds to step S54. On the other hand, if there is no releasable picture (NO in S56), a warning message is displayed to the user by the display unit 28 (S58), and the entire memory area on the DPB 60 is released except for the picture area currently being displayed. (S59), the process returns to step S51. Then, an IDR picture wait state is entered.

  Also in the operation shown in FIG. 6, it is detected whether a sufficient buffer area for storing the reference picture can be secured in the decoded picture buffer (DPB) 60. Then, according to the detection result, when the free area becomes less than a predetermined value, the decoding is interrupted. While decoding is interrupted, the currently displayed image is repeatedly displayed and the next IDR picture is waited, so that a fatal failure of the decoding process can be avoided.

  In S58, a warning message is issued when decoding is interrupted, but may be omitted if unnecessary. In S59, the buffer area other than the picture being displayed is released. However, if the memory area necessary for the next decoding process can be secured in the decoded picture buffer (DPB) 60 in the end, it is limited to this. It is not a thing. For example, undisplayed pictures on the DPB 60 may be released after being displayed sequentially, or the display of the reproduced image is abandoned and, for example, a blue background is displayed on the display unit 28, and the entire memory area of the DPB 60 is released at once. May be.

(Example 2)
The operation of Embodiment 2 of the present invention will be described. In the first embodiment, the display of the picture on the decoded picture buffer (DPB) is repeated until the decoding process with the IDR picture is resumed. However, in the second embodiment, as shown in FIG. 7, before the decoding with the IDR picture is resumed. The display image is updated at any time by the I picture. In the second embodiment, only the operations of the control unit 38 and the decoding unit 58 are different, and the operation of the other components is the same as that of the first embodiment. In addition, the decoding unit 58 according to the second embodiment selectively encodes intra-picture encoded image data (I) in addition to a mode (all decoding processing mode) for sequentially decoding I, P, and B pictures, which is a normal operation. This is a configuration capable of executing a mode (selective decoding processing mode) for decoding only (picture).

  Steps S201 to S209 in FIG. 7 are the same as steps S1 to S9 in FIG. That is, step S210 and subsequent steps in FIG. 7 are added to FIG. After the entire memory area on the decoded picture buffer (DPB) 60 is released except for the picture area currently being displayed (S209), the access to be decoded from the coded picture buffer (CPB) 56 is the same as in step S201. A unit is selected (S210). It is determined whether or not the access unit is an IDR picture (S211). If the selected access unit is an IDR picture (YES in S211), the process proceeds to step S203, and the subsequent IDR picture is similarly decoded.

  On the other hand, if the selected access unit is not an IDR picture (NO in S211), it is determined whether or not the access unit is an I picture (S212). If it is an I picture (YES in S212), the I picture is decoded and the decoding result is stored in the DPB 60 (S213). The decoded I picture is displayed on the display unit 28 at a predetermined timing, for example, a timing defined by the picture timing SEI, and the display images are sequentially switched. On the other hand, if it is not an I picture (NO in S212), the process returns to step S210, and the next access unit to be decoded is selected (S210).

  That is, in steps S210 to S213, the decoding unit 58 executes the selective decoding processing mode in which only the I picture is detected and decoded under the instruction of the control unit 38 until the next IDR picture is detected. Note that, by detecting the IDR picture, the decoding unit 58 switches the operation from the selective decoding processing mode to the normal all decoding processing mode. In the all decoding processing mode, both compressed image data (I picture) by intra-picture encoding and compressed image data (P picture, B picture) by inter-picture predictive encoding are decoded.

  By doing this, when a sufficient buffer area for storing the reference picture cannot be secured in the decoded picture buffer (DPB) 60, the next IDR picture is waited while repeatedly displaying the displayed image. A fatal failure of the decryption process can be avoided. Furthermore, since the I picture is decoded and the display screen is updated until the decoding process is resumed, the uncomfortable feeling of the playback screen can be reduced.

  FIG. 8 shows an operation flowchart in the case of the adaptive memory management method. Steps S251 to S263 correspond to steps S201 to S213 in FIG. Steps S256 and 257 are different from FIG. In FIG. 8, when there is not enough free space in the decoded picture buffer (DPB) 60 (NO in S253), it is checked whether there is a releasable picture in the DPB 60 (S256). If there is a releasable picture (YES in S256), the picture is deleted from the DPB 60, the buffer area is released (S257), and the process proceeds to step S254. On the other hand, if there is no releasable picture (NO in S256), the same processing as in FIG. 7 (steps S258 to S263) is executed.

(Example 3)
Information on the capacity of the decoded picture buffer (DPB) 60 necessary for the decoding process is recorded on the recording medium 26 together with the compressed data at the time of the encoding process, and this information is referred to at the time of decoding. Can be prevented.

  FIG. 9 is a flowchart for acquiring the number of pictures that need to be held in the decoded picture buffer (DPB) 60 during the decoding process of each picture. It is determined whether the encoding target is an IDR picture (S301). If it is an IDR picture (YES in S301), the number of pictures required on the DPB 60 is initialized to 1 (S302).

  If it is not an IDR picture (NO in S301), the required number of pictures is incremented by 1 (S303). Then, after decoding the picture, it is determined whether or not the coding is such that an arbitrary picture held in the DPB 60 may be released (S304). In the case of encoding in which an arbitrary picture held in the DPB 60 may be released after decoding the picture (YES in S304), the necessary number of pictures is decremented by 1 (S305), and the process returns to step S301. On the other hand, after decoding the picture, if the encoding is not such that any picture held in the DPB 60 may be released (NO in S304), the process proceeds to step S301.

  By repeating such processing, the required number of pictures stored in the decoded picture buffer (DPB) 60 can be calculated at the time of decoding of each picture. The calculated necessary number of pictures may be stored in the stream data as follows, for example. FIG. 10 shows a structural example in which the required number of pictures is stored in the stream data. FIG. 11 shows an example of the structure of the management file zzzzz.clpi recorded on a one-to-one basis with the stream file. FIG. 12 is a schematic diagram for explaining the details of the DPB information DbpInfo ().

  In response to the message SEI_massage () added to the access unit encoded in FIG. 10, user data user_data_unregisterd for storing the number of pictures necessary for the DPB 60 in the variable DPB_PIC_NUM at the time of decoding is stored. The message SEI_massage () and user data user_data_unregisterd are stored in ITU-T Rec. H. The detailed description thereof is omitted.

  This information may be added in units of access units, or only in the access units that are IDR pictures, or in other cases. When adding only to an access unit that is an IDR picture, the maximum value of the number of pictures required for the DPB 60 generated from the IDR picture to the next IDR picture may be stored in the user data user_data_unregisterd of the IDR picture.

  In FIG. 11, DPB information start address DPB_info_start_address indicates an offset value obtained by counting the start address of DPB information DbpInfo () in byte order from the top of management file zzzzz.clpi. The DPB information DPBInfo () is an entity of an area for storing the capacity of the DPB 60 necessary for decoding. Details regarding other areas are described in Blu-ray Disc Read-Only Format (Blu-ray is a trademark).

  In FIG. 12, in the information length Length, a value obtained by counting the entire size of the DPB information DPBInfo () in byte order is recorded. The number of entries number_of_entries stores the number of DPB_Info and PTS described later. DPB_Info stores the number of pictures stored in the DPB 60 during the decoding process of each picture calculated in FIG. The PTS stores the display timing of the access unit for which DPB_Info is valid with an accuracy of 90 kHz.

  In this way, the number of pictures that need to be stored in the decoded picture buffer (DPB) 60 at the time of decoding can be calculated in advance at the time of encoding and recorded in the stream data or the management file.

  In this embodiment, the information on the number of pictures may be stored in units of access units in the stream file managed by the management file or in units of IDR pictures. Alternatively, the number may be one for the entire stream file managed by the management file, or may be other than that.

  In the case of the IDR picture unit, the maximum value of the number of pictures stored in the decoded picture buffer (DPB) 60 from the IDR picture to the next IDR picture is stored, and the display of the IDR picture is displayed in the PTS. Stores the time. When there is one for the entire stream file managed by the management file, the maximum value of the number of pictures that need to be stored in the DPB 60 in the encoding process of the stream file managed by the management file is stored. Stores the display start time of stream data.

  Further, although the capacity of the DPB 60 has been described in terms of the number of pictures, it may be recorded in a size such as a byte unit. In the case of recording in units of bytes, in steps S302, S303, and S305 in FIG.

Example 4
A method of determining the capacity of the shock proof buffer (read buffer 50) when reproducing compressed video data in which information on the number of reference pictures necessary for decoding is recorded together as in the third embodiment will be described. To do. Note that the capacity of the decoded picture buffer (DPB) 60 necessary for decoding is recorded in the management file. Furthermore, it is assumed that the number of entries number_of_entries in the DPB information DPBInfo () is 1, and the maximum value of the number of pictures that need to be stored in the DPB 60 in the stream file managed by the management file is recorded in the DPB_Info. FIG. 13 is a flowchart for determining the sizes of the decoded picture buffer (DPB) 60 and the shock proof buffer (read buffer 50) in the decoding process.

  First, the buffer size required for decoding is initialized to 0 (S401). The size of the DPB 60 necessary for reproduction of all stream data is acquired (S402 to 404). It is calculated how much the size needs to be increased with respect to the system default value (initial value) of the DPB buffer (S405). The necessary increase is added to the buffer size of the DPB 60 (S406). The DPB buffer increment calculated in step S405 is subtracted from the buffer size of the shock proof buffer (read buffer 50) (S407).

  By controlling the buffer size as described above, it is possible to obtain an optimal memory map that achieves both durability against disturbance such as vibration and stream reproduction.

  In this embodiment, it is assumed that the capacity of the decoded picture buffer (DPB) 60 necessary for decoding is recorded in the management file. Furthermore, it is assumed that the number of entries number_of_entries of the DPB information DPBInfo () is 1, and the maximum value of the number of pictures that need to be stored in the DPB 60 in the stream file managed by the management file is recorded in the DPB_Info. However, the present invention is not limited to this.

(Example 5)
Another method for determining the capacity of the shock proof buffer (read buffer 50) will be described. It is assumed that the capacity of the decoded picture buffer (DPB) 60 necessary for decoding is recorded in the management file. Further, it is assumed that the number of entries number_of_entries of the DPB information DPBInfo () is the number of IDR pictures included in the stream file managed by the management file. Furthermore, it is assumed that the maximum value of the number of pictures that need to be stored in the DPB 60 between the IDR picture and the next IDR picture is recorded in each DPB_Info.

  FIG. 14 is an operation flowchart at the time of decoding. In FIG. 14, an access unit to be decoded is selected from the coded picture buffer (CPB) 56 (S501). It is determined whether or not the selected access unit is an IDR picture (S502). If the selected access unit is not an IDR picture (NO in S502), the process returns to step S501. If it is an IDR picture (YES in S502), the size of the DPB 60 required for decoding from the IDR picture to the next IDR picture is acquired according to the PTS from the management information, and the buffer size of the DPB 60 is greater than or equal to the required amount Is determined (S503). If the buffer capacity of the DPB 60 is insufficient (YES in S503), the IDR picture is decoded and stored in the DPB 60 (S504), and a program in the system issues a warning to the user at the display timing of the IDR picture. A reservation is made (S505). Then, the process returns to step S501.

  On the other hand, if the buffer capacity of the DPB 60 is greater than the required amount (NO in S503), the access units are sequentially decoded and stored in the DPB 60 until an IDR picture is detected (S506 to S508). If the access unit becomes an IDR picture (YES in S508), the process returns to step S503.

  Since the operation in the case where a display image is lost in the decoded picture buffer (DPB) 60 by the processing after step S504 is the same as that in the first embodiment, a description thereof will be omitted.

  As described above, in this embodiment, even when the DPB buffer size necessary for decoding cannot be secured in the decoded picture buffer (DPB) 60 at the time of decoding, the fatal failure of the decoding process is avoided and the IDR picture decoding is triggered. The decoding process can be resumed.

(Example 6)
An embodiment in which the display of a reproduced image is updated before decoding with an IDR picture is resumed will be described. FIG. 15 is a flowchart of the decoding operation of this embodiment. The operations in steps S601 to S605 in FIG. 15 are the same as those in steps S501 to S505 in FIG. 14, and the operations in steps S610 to S612 are the same as those in steps S506 to 508 in FIG.

  In step S605, after the occurrence of a user warning is reserved, the next decoding target is acquired from the coded picture buffer (CPB) 56 (S606). It is determined whether or not the access unit is an IDR picture (S607). If the access unit is an IDR picture (YES in S607), the process returns to step S603. If the access unit is not an IDR picture (NO in S607), it is determined whether or not the access unit is an I picture (S608). If it is an I picture (YES in S608), the I picture is decoded and the decoding result is stored in the DPB 60 (S609). If it is not an I picture (NO in S608), the process returns to step S606.

  In step S604 and subsequent steps, the operation when the display image of the DPB 60 is missing is the same as that in the first embodiment, and thus the description is omitted.

  Thus, in this embodiment, even when a sufficient buffer area for storing the decoded picture cannot be secured in the decoded picture buffer (DPB) at the time of decoding, a fatal failure of the decoding process is avoided, and the IDR picture The decoding process can be resumed with the decoding of. In addition, until the decoding process is resumed, the display picture can be updated by decoding the I picture.

It is a schematic block diagram of one Example of this invention. It is a schematic block diagram which shows the buffer structure at the time of the decoding of a present Example. It is a structural diagram of the source packet of a present Example. It is a structural diagram of the header of the present embodiment. It is a flowchart which shows the decoding operation | movement of a present Example. It is a flowchart which shows the decoding operation | movement by another buffer management method of a present Example. 10 is a flowchart illustrating a decoding operation according to the second embodiment. 12 is a flowchart illustrating a decoding operation in another buffer management method according to the second embodiment. It is a flowchart which acquires the required buffer capacity of DPB. It is an example of the structure of the user data which stores the required buffer capacity of DPB. It is an example of the structure of a management file. It is a structural example of DPB information. It is a flowchart which determines the buffer size of DPB and shockproof. 10 is a flowchart illustrating a decoding operation according to the fifth embodiment. 18 is a flowchart illustrating a decoding operation according to the sixth embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image pick-up device 12 Shooting lens 14 Image pick-up element 16 Signal processing unit 18 Buffer 20 Video coding / decoding unit 22 Flash memory 24 Disk control unit 26 Recording medium 28 Display unit 30 Multiplexer / demultiplexer 32 Microphone 34 Speaker 36 Audio coding / decoding Unit 38 Control unit 38a Detection unit 40 Operation unit 42 Bus 50 Read buffer (RB)
52 Packet Separation Unit 54 Transport Buffer (TB)
56 Coded picture buffer (CPB)
58 Decoding unit 60 Decoded picture buffer (DPB)

Claims (16)

An image decoding apparatus for decoding compressed image data compressed using inter-picture prediction encoding,
A first picture buffer for storing the compressed image data;
A second picture buffer for storing the decoded image data;
The image data stored in the second picture buffer is selectively referred to, the compressed image data stored in the first picture buffer is decoded, and the decoded image data is converted into the second picture. Decoding means for storing in a buffer;
Detecting means for detecting free space in the second picture buffer;
Control means for interrupting decoding by the decoding means when the free space is less than a predetermined value according to the detection result of the detection means;
An image decoding apparatus comprising:   2. The image decoding apparatus according to claim 1, wherein the control unit repeats reading predetermined image data stored in the second picture buffer when the free space is less than a predetermined value. 3.   The control means includes means for detecting an IDR (Instantaneous Decoder Refresh) picture stored in the first picture buffer, and when the IDR picture is detected, the control means restarts decoding of the decoding means. The image decoding apparatus according to claim 1 or 2.   The image decoding apparatus according to claim 1, further comprising a warning unit that issues a message indicating interruption of decoding by the decoding unit. An image decoding apparatus for decoding compressed image data compressed using inter-picture predictive coding and intra-picture coding,
A first picture buffer for storing the compressed image data;
A second picture buffer for storing the decoded image data;
The image data stored in the second picture buffer is selectively referred to, the compressed image data stored in the first picture buffer is decoded, and the decoded image data is converted to the second picture. Decoding means for storing in a buffer, all decoding processing modes for decoding both compressed image data by the inter-picture predictive coding and compressed image data by the intra-picture coding, and compression by the intra-picture coding Decoding means capable of selectively executing a selective decoding processing mode for decoding only image data;
Detecting means for detecting free space in the second picture buffer;
Control means for instructing the decoding means to perform decoding in the selective decoding processing mode when the free space is less than a predetermined value according to the detection result of the detecting means;
An image decoding apparatus comprising:   The control means repeatedly reads out the image data of the second picture buffer for a predetermined period when no image data is stored in the second picture buffer for a predetermined period in the selective decoding processing mode. The image decoding apparatus according to claim 5.   When the control means detects an IDR (Instantaneous Decoder Refresh) picture stored in the first picture buffer, the control means instructs the decoding means to switch from the selective decoding processing mode to the full decoding processing mode. The image decoding device according to claim 5, wherein the image decoding device is characterized in that   8. The image decoding apparatus according to claim 5, further comprising warning means for issuing a message indicating switching of the decoding means to the selective decoding processing mode. An image decoding apparatus for decoding compressed image data compressed using inter-picture predictive encoding, and a first picture buffer for storing the compressed image data, and a first picture buffer for storing the decoded image data A method for controlling an image decoding device comprising two picture buffers,
The image data stored in the second picture buffer is selectively referred to, the compressed image data stored in the first picture buffer is decoded, and the decoded image data is converted into the second picture. A decoding step to store in a buffer;
A detecting step of detecting a free capacity of the second picture buffer;
According to the detection result of the detection step, when the free space is less than a predetermined value, a control step of interrupting decoding in the decoding step;
An image decoding apparatus control method comprising:   10. The image decoding apparatus according to claim 9, wherein in the control step, when the free space is less than a predetermined value, reading of predetermined image data stored in the second picture buffer is repeated. Control method.   The control step includes a step of detecting an IDR (Instantaneous Decoder Refresh) picture stored in the first picture buffer, and when the IDR picture is detected, the decoding of the decoding step is resumed. The control method of the image decoding apparatus of Claim 9 or 10.   The method of controlling an image decoding apparatus according to claim 9, further comprising a warning step of issuing a message indicating interruption of decoding by the decoding step. An image decoding apparatus for decoding compressed image data compressed using inter-picture predictive coding and intra-picture coding, and a first picture buffer for storing the compressed image data, and a decoded image A method for controlling an image decoding device comprising a second picture buffer for storing data, comprising:
The image data stored in the second picture buffer is selectively referred to, the compressed image data stored in the first picture buffer is decoded, and the decoded image data is converted to the second picture. A decoding process stored in a buffer, the decoding process storing both the compressed image data by the inter-picture prediction encoding and the compressed image data by the intra-picture encoding, and the compression by the intra-picture encoding A decoding step capable of selectively executing a selective decoding processing mode for decoding only image data;
A detecting step of detecting a free capacity of the second picture buffer;
According to the detection result of the detection step, when the free space is less than a predetermined value, a control step for instructing the decoding step to perform decoding in the selective decoding processing mode;
An image decoding apparatus control method comprising:   In the control step, when no image data is stored in the second picture buffer for a predetermined period in the selective decoding processing mode, the image data in the second picture buffer is repeatedly read out for a predetermined period. The method of controlling an image decoding device according to claim 13.   In the control step, when an IDR (Instantaneous Decoder Refresh) picture stored in the first picture buffer is detected, the decoding step is instructed to switch from the selective decoding processing mode to the full decoding processing mode. The method for controlling an image decoding apparatus according to claim 13 or 14, characterized in that:   16. The method of controlling an image decoding device according to claim 13, further comprising a warning step of issuing a message indicating switching of the decoding step to the selective decoding processing mode.

Images