JP6399189B2 - Video coding method - Google Patents

Description

  The present invention relates to a moving image encoding method capable of editing encoded moving image data without decoding, for example.

  The moving image data generally has a very large amount of data. Therefore, a device that handles moving image data compresses the moving image data by encoding it when transmitting the moving image data to another device or when storing the moving image data in the storage device. . As a typical video coding standard, Moving Picture Experts Group phase 2 (MPEG-2), MPEG-4, or H.264 MPEG-4 established by the International Standardization Organization / International Electrotechnical Commission (ISO / IEC) Advanced Video Coding (MPEG-4 AVC / H.264) is widely used.

  In such an encoding standard, only the information of the encoding target picture and the inter encoding method for encoding the encoding target picture using the encoding target picture and the information of the pictures before and after the encoding target picture are used. An intra-encoding method that uses and encodes is employed. In addition, the inter-coding method includes intra-coded pictures (I pictures), forward-predicted pictures (P pictures) that generally use past pictures as predicted pictures, and generally past pictures and future pictures. There are three types of pictures, bi-predictive pictures (B pictures), both of which are predictive pictures.

In general, the code amount of a picture or block encoded by the inter encoding method is smaller than the code amount of a picture or block encoded by the intra encoding method. In this way, the coding amount of the picture is biased in the sequence depending on the selected coding mode. Similarly, the coding amount selected causes a deviation in the code amount of the block in the picture.
Therefore, even if the code amount fluctuates with time, a transmission buffer for the data stream is prepared in the transmission source device so that a data stream including a moving image encoded at a constant transmission rate can be transmitted. In addition, a reception buffer for the data stream is prepared in the transmission destination device.

  In MPEG-2 and MPEG-4 AVC / H.264, the operation of a reception buffer in an ideal moving picture decoding apparatus called Video Buffering Verifier (VBV) and Coded Picture Buffer (CPB) is defined. In the following, for convenience, an ideal video decoding device is simply referred to as an ideal decoding device. The ideal decoding device is defined to perform instantaneous decoding in which the time required for the decoding process is zero. For example, Patent Document 1 discloses a method for controlling a moving picture coding apparatus related to VBV.

The moving picture coding apparatus ensures that the data of the picture is stored in the reception buffer at the time when the picture is decoded by the ideal decoding apparatus so that the reception buffer of the ideal decoding apparatus does not overflow and underflow. The code amount is controlled.
The underflow of the reception buffer is to decode a picture by a time when the moving image decoding apparatus should decode and display when the moving image encoding apparatus transmits a moving image data stream encoded at a constant transmission rate. This occurs when the transmission of data necessary for data transfer is not completed. That is, the underflow of the reception buffer means that data necessary for decoding a picture does not exist in the reception buffer of the video decoding device. In this case, since the moving picture decoding apparatus cannot perform the decoding process, frame skip occurs.

The moving image decoding apparatus displays the picture after delaying the stream by a predetermined time from the reception time so that the decoding process can be performed without causing an underflow of the reception buffer.
As described above, in the ideal decoding device, it is defined that the decoding process is instantaneously completed at the processing time 0. Therefore, if the input time of the i-th picture to the video encoding device is t (i) and the decoding time of the i-th picture in the ideal decoding device is tr (i), the earliest possible display of that picture Time is equal to tr (i). Since the picture display periods {t (i + 1) -t (i)} and {tr (i + 1) -tr (i)} are equal in all pictures, the decoding time tr (i) is the input time A time {tr (i) = t (i) + dly} delayed by a fixed time dly from t (i). Therefore, the moving picture coding apparatus must complete transmission of data necessary for decoding to the reception buffer of the moving picture decoding apparatus by time tr (i).

The state of the reception buffer will be described with reference to FIG. In FIG. 1, the horizontal axis represents time, and the vertical axis represents the buffer occupation amount of the reception buffer. A solid line graph 100 represents the buffer occupancy at each time.
In the reception buffer, the buffer occupancy is restored at a predetermined transmission rate, and data used for decoding the picture is extracted from the buffer at the decoding time of each picture. The i-th picture data starts to be input to the reception buffer at time at (i), and the last data of the i-th picture is input at time ft (i). The ideal decoding device completes decoding of the i-th picture at time tr (i), and can display the i-th picture at time tr (i). However, when a B picture is included, picture reordering (reordering of coding order) has occurred, so the actual display time for the i-th picture may be later than tr (i).

  Details of the decoding time and display time description method for each picture in MPEG-4 AVC / H.264 will be described below.

  In MPEG-4 AVC / H.264, supplementary information that is not directly related to pixel decoding processing is described in an SEI (Supplemental enhancement information) message. Dozens of types of SEI are defined, and the type is identified by the payloadType parameter. This SEI is added for each picture.

BPSEI (Buffering Period SEI), which is one of SEI, is added to a picture that can be redrawn, that is, a picture that can be decoded without a past picture (generally an I picture). In BPSEI, a parameter InitialCpbRemovalDelay is described. The parameter InitialCpbRemovalDelay represents a difference value between the arrival time of the first bit of the picture to which the BPSEI is added and the decoding time of the picture to which the BPSEI is added. The accuracy of this difference value is 90 kHz.
The decoding time tr (0) of the first picture is the time when the first bit of the encoded moving image data reaches the moving image decoding device (assumed to be 0), that is, the time delayed by InitialCpbRemovalDelay ÷ 90,000 [sec] from at (0) become.

  In addition, generally, PTSEI (Picture Timing SEI), which is one of SEI, is added to each picture. In PTSEI, parameters CpbRemovalDelay and DpbOutputDelay are described. The parameter CpbRemovalDelay represents the difference value between the decoding time of the previous BPSEI-added picture and the decoding time of the picture to which PTSEI is added. The parameter DpbOutputDelay represents a difference value between the decoding time of the picture to which PTSEI is added and the display time of the picture. The accuracy of these difference values is one field picture interval. Therefore, when the picture is a frame, the parameters CpbRemovalDelay and DpbOutputDelay are multiples of 2.

  The decoding time tr (i) of the second and subsequent pictures is delayed by tc * CpbRemovalDelay (i) [sec] from the decoding time tr (0) of the first picture. CpbRemovalDelay (i) is CpbRemovalDelay added to the i-th picture. tc is a time interval [sec] between pictures. For example, in the case of a 29.97 Hz progressive image, tc is 1001/60000.

  The display time of each picture including the picture to which BPSEI is added is a time delayed by tc * DpbOutputDelay (i) from tr (i). DpbOutputDelay (i) is a DpbOutputDelay added to the i-th picture. That is, after time tr (0), picture decoding and display are performed in time units that are integral multiples of tc.

  Depending on the use of the moving image data, the encoded moving image may be edited. The editing operation of the encoded moving image is to subdivide the encoded moving image data and connect them to generate another encoded moving image data. For example, processing (splicing) for inserting another program (for example, CM) in the middle of an encoded moving image being broadcast is one of editing operations.

  In editing a motion picture that has been subjected to inter-frame predictive coding, especially an inter-coded picture cannot be normally decoded by the coded picture alone. Therefore, in order to combine two encoded moving image data at an arbitrary picture position, an apparatus for editing the encoded moving image data decodes the two encoded moving image data once combined, After combining in units, the combined moving image data is encoded again.

  However, since the re-encoding process is complicated, it is common to directly edit the encoded moving image data without performing the re-encoding process by limiting the coupling position particularly in the real-time process such as splicing. When editing the encoded moving image data without performing re-encoding processing, the first picture of the encoded moving image data that is combined backward in time among the two encoded moving image data to be combined is the I picture And In addition, in the encoded video data that is temporally coupled to the back side, the so-called closed GOP structure, that is, all pictures following the top I picture do not refer to past pictures in time than the top I picture. Limited. As a result, all the pictures after the first I picture of the encoded video data on the rear side of the two encoded video data combined at a predetermined connection point by editing can be normally decoded.

  Note that the closed GOP structure has a lower encoding efficiency than the non-closed GOP structure, and therefore the non-closed GOP structure may be adopted. In this case, some pictures immediately after the first I picture after the connection point are not decoded normally, but since these pictures are displayed before the first I picture in display time, they cannot be displayed. Also good. Therefore, in general, a moving image decoding apparatus normally displays a final picture of temporally preceding encoded moving image data among two combined encoded moving image data and then freezes the display. Mask the display of pictures that could not be decoded.

In the prior art, even when the video data that has been inter-frame predictively encoded is edited without being re-encoded, the header information is also included so that no wrinkles occur between the two encoded video data that have been combined. Will be corrected.
For example, in MPEG-4 AVC / H.264, POC (Picture Order Count) and FrameNum are added to a slice header in order to specify a temporal relationship between pictures and to specify a reference picture. POC represents the display order of relative pictures. FrameNum is a value that increases by 1 each time a reference picture appears in the encoded video. Since the POC and FrameNum need to be continuous between the two encoded video data combined, all the POC and FrameNum of the encoded video data on the back side of the two encoded video data to be combined It is necessary to rewrite.

  On the other hand, in the method disclosed in Non-Patent Document 1, FrameNum is abolished because a new reference picture specifying method is introduced. Of the two encoded moving image data to be combined, the POC value of the first picture of the backward encoded moving image data need not have continuity with the encoded image data of the front side, so the slice header No change is required. In the method disclosed in Non-Patent Document 1, as a picture type, in addition to IDR (Instantaneous Decoding Refresh) picture defined by MPEG-4 AVC / H.264, CRA (Clean Random Access) picture, BLA (Broken Link Access) ) Picture, TFD (Tagged For Discard) picture, DLP (Decodable Leading Picture) picture, and TP (Trailing Picture) picture are newly introduced.

  Among these, the CRA picture and the BLA picture are both pictures that become re-drawing points. A picture that serves as a redrawing point is a picture that does not refer to other pictures in decoding the picture. When the moving picture decoding apparatus starts the decoding operation from the CRA picture and the BLA picture, it is possible to normally decode other than the TFD picture immediately following the CRA picture or the BLA picture.

  A TFD picture is a picture that appears immediately after a CRA picture or a BLA picture and refers to a picture preceding the CRA picture or the BLA picture in terms of time and decoding order. In the case of a non-closed GOP structure compliant with MPEG-2, a plurality of B pictures immediately after the I picture at the head of the GOP correspond to TFD pictures.

  A BLA picture is generated by an editing operation of encoded moving image data. Of the two encoded moving image data to be combined, the first picture of the encoded image data on the back side is generally a CRA picture, but this CRA picture is located in the middle of the combined encoded moving image data. In this case, the type is changed from the CRA picture to the BLA picture. In the method disclosed in Non-Patent Document 1, when a BLA picture appears, the POC value is allowed to be discontinuous. Further, the TFD picture immediately after the BLA picture cannot be normally decoded no matter where it is decoded from the encoded moving picture data because the picture to be referred to is lost in the combined encoded moving picture data. For this reason, the moving picture coding apparatus may delete the TFD picture that follows the first BLA picture of the back coded moving picture data from the coded moving picture data, of the two coded moving picture data to be combined. .

  A DLP picture is a picture that appears immediately after a CRA picture or a BLA picture, like a TFD picture. Unlike the TDF picture, the DLP picture does not refer to the previous picture in terms of time and decoding order than the CRA picture or the BLA picture. Therefore, the DLP picture can be normally decoded even when decoding is started from the CRA picture or the BLA picture.

  The TP picture appears later in the decoding order than the CRA picture or BLA picture, TFD picture, and DLP picture, and is a picture temporally later than the CRA picture or BLA picture. Therefore, the TP picture can be normally decoded even when decoding is started from the CRA picture or the BLA picture.

JP 2003-179938 A

JCTVC-J1003, "High-Efficiency Video Coding (HEVC) text specification Draft 8", Joint Collaborative Team on Video Coding of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11, July 2012

  In the method disclosed in Non-Patent Document 1, the decoding time and display time of each encoded picture are determined using parameters InitialCpbRemovalDelay, CpbRemovalDelay, and DpbOutputDelay, as in MPEG-4 AVC / H.264. When two encoded video data are combined, in order to continuously perform video decoding processing and display at the connection point, CpbRemovalDelay and DpbOutputDelay of the pictures after the connection point are corrected to appropriate values. Is required.

Specifically, the moving image encoding device or the moving image decoding device calculates the CpbRemovalDelay of the first CRA picture of the encoded image data on the rear side of the two encoded moving image data to be combined with the encoded image on the front side. It is necessary to correct based on the number of pictures following the final BPSEI additional picture in the image data. Further, in order to guarantee the continuity of the CPB buffer, the moving image encoding device or moving image decoding device increases the value of CpbRemovalDelay. Also, when removing the TFD picture from the encoded video data on the back side, the video encoding device or video decoding device uses the CpbRemovalDelay of the picture to be decoded after the removed TFD picture, and the leading CRA picture at the connection point In addition, it is necessary to change the DpbOutputDelay of the picture to be decoded before the removed TFD picture.
As described above, in the method disclosed in Non-Patent Document 1, it is still necessary to change the contents of PTSEI in an editing operation for combining two encoded moving image data.

  Therefore, in this specification, when two moving image data subjected to interframe prediction encoding are combined, continuous decoding processing and display processing are performed without changing the parameters in the header of the original encoded moving image data. It is an object of the present invention to provide a moving picture encoding method that enables this.

  According to one embodiment, the moving image encoding that generates the combined moving image data encoded by combining the first moving image data and the second moving image data encoded by the inter-frame prediction method. A method is provided. In this moving image encoding method, in the moving image decoding apparatus, the first code combined after the encoded first moving image data among the pictures included in the encoded second moving image data. Even when one or more pictures whose encoding order is later than the first encoded picture are removed, the pictures after the first encoded picture in the encoded second moving image data are also deleted. Determining correction information for decoding delay and display delay to enable continuous decoding and display, and adding the correction information to the combined moving image data, the correction information for each picture to be removed The decoding order for the picture to be removed is calculated based on the decoding interval between the previous picture.

The objects and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.
It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention as claimed.

  The moving image encoding method disclosed in the present specification can be performed without combining the parameters in the header of the original encoded moving image data when combining two pieces of moving image data subjected to inter-frame predictive encoding. Enables continuous decoding and display processing.

It is a figure which shows the relationship between the transition of the buffer occupation amount of a receiving buffer, and display time. It is a figure which shows the relationship between the display order and decoding order of each picture contained in moving image data, and a decoding delay and a display delay. It is explanatory drawing of the decoding delay and display delay of the picture after the coupling | bonding point when two coding moving image data are couple | bonded. It is explanatory drawing of the data structure of one picture of the encoding moving image by 1st Embodiment. It is a schematic block diagram of the moving image encoder by 1st Embodiment. It is an operation | movement flowchart of the moving image encoding process by 1st Embodiment. It is an operation | movement flowchart of the moving image edit process by 1st Embodiment. It is a schematic block diagram of the moving image decoding apparatus by 1st Embodiment. It is an operation | movement flowchart of the moving image decoding process by 1st Embodiment. It is explanatory drawing of the decoding delay and display delay of the picture after the coupling | bonding point at the time of two coding moving image data being couple | bonded by 2nd Embodiment. It is explanatory drawing of the data structure of one picture of the encoding moving image by 2nd Embodiment. The block diagram of the computer which operate | moves as a moving image encoding apparatus or a moving image decoding apparatus by the computer program which implement | achieves the function of each part of the moving image encoding apparatus or moving image decoding apparatus by each embodiment or its modification It is.

Hereinafter, a video encoding device and a video decoding device according to various embodiments will be described with reference to the drawings. This moving image encoding apparatus uses values used for correcting parameters representing decoding time and display time for pictures after the combining point when combining two encoded moving image data without decoding. And the value is added to the header information of the pictures after the coupling point. As a result, the moving image encoding apparatus does not require rewriting of the parameters in the header of the original encoded moving image data even when two encoded moving image data are combined.

  In the present embodiment, the picture is a frame. However, the picture is not limited to a frame and may be a field. The frame is one still image in the moving image data, while the field is a still image obtained by extracting only odd-numbered data or even-numbered data from the frame. The encoded moving image may be a color moving image or a monochrome moving image.

With reference to FIG. 2, first, the picture decoding delay CpbRemovalDelay and the display delay DpbOutputDelay in the first embodiment will be described by taking one picture coding structure as an example.
In FIG. 2, a picture coding structure 201, which is an example of a picture coding structure, includes a plurality of pictures. One block in the picture coding structure 201 represents one picture. Of the two characters shown in the block corresponding to each picture, the alphabet of the left one character represents the encoding mode applied to that picture. I, P, and B mean an I picture, a P picture, and a B picture, respectively. Also, the number on the right of the two characters shown inside the block is the order of input to the video encoding device. Note that the input order matches the order output from the video decoding device. An arrow shown on the upper side of the picture coding structure 201 represents a reference picture to which a picture to be coded in forward frame prediction refers. For example, the picture P4 refers to a picture I0 located before the picture P4. Similarly, an arrow shown below the picture coding structure 201 represents a reference picture that is referenced by a picture to be coded in backward frame prediction. For example, picture B2 refers to picture P4 located after picture B2.

Below the picture coding structure 201, a decoding order 202 of each picture included in the picture coding structure 201 in the video decoding device is shown. Each block shown in the decoding order 202 represents one picture, and the characters in each block are input to the encoding mode and the moving image encoding apparatus in the same manner as the picture encoding structure 201. Represents the order. Note that this decoding order 202 matches the order of encoding by the moving picture encoding apparatus. The arrows shown above and below the decoding order 202 of each picture are a reference picture referenced by a picture encoded in forward frame prediction and a reference referenced by a picture encoded in backward frame prediction, respectively. Represents a picture.
In the decoding order 202, BPSEI is added to a picture in which “BPSEI” is written on the lower side. In this example, BPSEI is added to all I pictures. That is, the parameter InitialCpbRemovalDelay indicating the difference value between the arrival time of the first bit of the I picture at the reception buffer and the decoding time of the I picture is described in all I pictures.

  A block string 203 shown below the decoding order 202 represents the values of CpbRemovalDelay and DpbOutputDelay included in PTSEI added to each picture. Each block on the upper side of the block sequence 203 indicates the value of CpbRemovalDelay of a picture in decoding order 202 located above that block. On the other hand, each lower block of the block sequence 203 indicates the value of DpbOutputDelay of a picture in decoding order 202 located above that block. CpbRemovalDelay corresponds to the coding order from the previous picture in the coding order among the pictures to which BPSEI is added. For example, the picture P8 is fifth from the picture I0 in the coding order. In this embodiment, each picture is a frame, and the time interval tc between pictures is a field unit value, so the CpbRemovalDelay of the picture P8 is 10 (= 5 * 2).

  On the other hand, DpbOutputDelay is a display delay for continuously outputting each picture in the correct order in the video decoding device. For example, DpbOutputDelay is 10 for picture P4. This is a delay necessary for correctly displaying the picture B1 having the largest difference between the input order and the encoding order in the moving picture encoding apparatus. That is, since the picture B1 is decoded two picture times after the decoding time of the picture P4, the display time of the picture P4 is three pictures from the earliest time when the picture B1 can be displayed, that is, the decoding time of the picture B1. It should be displayed after hours. The difference between the decoding time and the display time of the picture P4 is 5 picture times, and DpbOutputDelay is 10 because tc is a field unit.

  Next, referring to FIG. 3, when two encoded moving image data are combined, in order to ensure that there is no contradiction in decoding delay and display delay before and after the combination point of the two encoded moving image data. Next, the values to be taken for the decoding delay CpbRemovalDelay and the display delay DpbOutputDelay in the picture of the encoded video data behind the coupling point will be described.

  Each block shown in the first moving image encoded data 301 combined before the combining point represents one picture, and the characters in each block are encoded as in FIG. Represents the order of input to the video coding mode and the moving picture coding apparatus. In this example, the encoding structure of the first moving image encoded data 301 is the same as the encoding structure 201 shown in FIG.

  In this example, the second moving image encoded data 302 is combined immediately after the last picture B15 of the first moving image encoded data. Also in the second moving image encoded data 302, each block represents one picture, and the characters in each block represent the encoding mode and the order of input to the moving image encoding device. The arrows shown on the upper side of the second moving image encoded data 302 represent reference pictures that the pictures B70, B69, and B71 to be encoded refer to in the forward frame prediction, respectively. In addition, the arrows shown below the second moving image encoded data 302 represent reference pictures that the pictures B70, B69, and B71 to be encoded refer to in backward frame prediction, respectively. The encoding structure of the second moving image encoded data 302 is the same as the encoding structure 201 shown in FIG. 2 except for pictures B70, B69, and B71. The encoding order of the pictures B70, B69, and B71 is the same as the encoding order of the bi-predictive pictures included in the encoding structure 201 shown in FIG. However, the reference pictures of the pictures B70, B69, and B71 are different from the reference pictures of the bi-predictive pictures included in the coding structure 201. Pictures B70 and B71 refer only to pictures whose display time is later, that is, picture I72. On the other hand, the picture B69 refers to only the picture with the previous display time, that is, I68. Such a situation occurs, for example, when there is a scene change between the picture B69 and the picture B70. Because the image changes greatly at the scene change boundary, bi-predictive pictures located near the scene change boundary should refer only to pictures on the same side as the scene change boundary, which is more predictive. Become. In this example, B69 is a TFD picture, and B70 and B71 are DLP pictures. In this example, of the second moving image encoded data 302, the pictures after the picture I72 are combined after the picture B15 of the first moving image encoded data. In the method of Non-Patent Document 1, there are restrictions that the display time of the TFD picture is earlier than the display time of the DLP picture, and that the DLP picture is not referenced from the TP picture.

  The block sequence 303 shown below the second moving image encoded data 302 is a value of a decoding delay CpbRemovalDelay and a display delay DpbOutputDelay included in PTSEI added to each picture of the second moving image encoded data 302. Represents. Each block on the upper side of the block sequence 303 indicates the value of the decoding delay CpbRemovalDelay of the picture of the second moving image encoded data 302 located above the block. On the other hand, each lower block in the block sequence 303 indicates the value of the display delay DpbOutputDelay of the picture of the second moving image encoded data 302 located above the block.

  Below the block sequence 303, combined moving image encoded data 304 in which the first moving image encoded data 301 and the second moving image encoded data 302 are combined is shown. In this example, the combined moving image encoded data 304 does not include the picture B67 of the second moving image encoded data 302 and the encoded picture preceding the picture B67 in the encoding order. Also, the picture B69 is a TFD picture that refers to a picture I68 that is an encoded picture preceding the picture I72 in the encoding order. Therefore, when combined with the picture I72, the picture B69 cannot be normally reproduced. Therefore, picture B69 is removed at the time of combination. However, the picture B69 may be left in the combined moving image encoded data without being removed. Pictures B70 and B71 are DLP pictures that do not refer to an encoded picture prior to picture I72 in the encoding order, and can be normally reproduced. Since the pictures B70 and B71 are not referenced from the pictures after P76, even if the pictures B70 and B71 are removed at the same time as the TFD picture 69, the reproduction of the pictures after P76 is not affected.

  A block string 305 indicates values of a decoding delay CpbRemovalDelay and a display delay DpbOutputDelay that the pictures I72, B70, B71, P76, B74, B73, and B75 should have in the combined moving image encoded data 304. Each block on the upper side of the block sequence 305 indicates the value of the decoding delay CpbRemovalDelay of the picture of the combined moving image encoded data 304 located above the block. On the other hand, each lower block in the block sequence 305 indicates the value of the display delay DpbOutputDelay of the picture of the combined moving image encoded data 304 positioned above that block.

  The decoding delay CpbRemovalDelay of the picture I72 needs to be matched with the encoded picture interval with the picture I12 that is the picture having the immediately preceding BPSEI after the combination. In this example, since the picture I72 is the eighth from the picture I12 according to the coding order, the decoding delay CpbRemovalDelay is 16 (= 8 * 2). Also, the display delay DpbOutputDelay of the picture I72 needs to be corrected in order to correctly display the picture B73 decoded after the picture I72. The value of the display delay DpbOutputDelay of the picture I72 is different before and after the picture B69 is removed. The value of the display delay DpbOutputDelay after removing the picture B69 decreases by the decoding interval that is the difference in decoding time between the removed picture and the previous picture in the decoding order for the removed picture after I72 in the decoding order. . In this example, picture B69 corresponds to the picture to be removed, and the decoding interval of B69 (the difference between the decoding time of B69 and the decoding time of B70 that is the previous picture in the decoding order) is 2, so The value of the display delay DpbOutputDelay is 2. Similarly, the display delay DpbOutputDelay of the picture B70 is decreased to 2 by the decoding interval of the removed pictures after B70 in the decoding order, that is, by 2.

  Furthermore, the decoding delays CpbRemovalDelay of the pictures B71, P76, B74, B73, and B75 are different before and after the picture B69 is removed. The values of the decoding delays CpbRemovalDelay of the pictures B71, P76, B74, B73, and B75 after the removal of the picture B69 are the removed pictures before the respective pictures I72 in the decoding order from the values of the decoding delays CpbRemovalDelay before the removal. Decrease by the decoding interval. In this example, the values of the decoding delays CpbRemovalDelay of the pictures B71, P76, B74, B73, and B75 are respectively 4, 6, 8, and the value of the original decoding delay CpbRemovalDelay minus the decoding interval 2 of the TFD picture B69. 10 and 12. For DLP picture B70, since there is no removed picture prior to B70 in the decoding order, the value of CpbRemovalDelay is unchanged even if picture B69 is removed. Further, for the pictures P76, B74, B73, and B75, the value of the display delay DpbOutputDelay is unchanged. Furthermore, it is not necessary to modify either the decoding delay CpbRemovalDelay or the display delay DpbOutputDelay for a picture that is later in the input order than the picture that is the first CRA after combining.

  As described above, by combining the two encoded moving image data, for some pictures included in the encoded moving image data after the connection point, the decoding delay CpbRemovalDelay and the display are displayed at the time of decoding. The delay DpbOutputDelay needs to be corrected. Therefore, in the present embodiment, the moving picture decoding apparatus does not need to modify the values of the decoding delay CpbRemovalDelay and the display delay DpbOutputDelay of the pictures included in the original moving picture encoded data before combining. When decoding the combined moving image encoded data, parameters that can be used to rewrite the values of the decoding delay CpbRemovalDelay and the display delay DpbOutputDelay to appropriate values are added to the header of the moving image encoded data.

With reference to FIG. 4, the structure of encoded moving image data including parameters that can be used to rewrite the values of the decoding delay CpbRemovalDelay and the display delay DpbOutputDelay to appropriate values in the first embodiment will be described.
As shown in FIG. 4, the data structure 400 of one picture includes six types of Network Abstraction Layer (NAL) units 410 to 415. These NAL units 410 to 415 all comply with the method of Non-Patent Document 1 and the NAL unit of MPEG-4 AVC / H.264. A header NUH420 is added to each NAL unit. The header NUH 420 includes a NalUnitType field indicating the type of each NAL unit. When NalUnitType is '1' or '2', it represents a TP picture. When NalUnitType is “7”, it represents that the TFD picture and the DLP picture are likely to appear immediately and are BLA pictures that can be redrawn. When NalUnitType is “8”, this indicates that the DLP picture may appear immediately after and is a redrawable BLA picture. When NalUnitType is “9”, it represents a redrawable BLA picture in which a TFD picture and a DLP picture do not appear immediately after. When NalUnitType is '12', it represents a redrawable CRA picture. When NalUnitType is “13”, it represents a DLP picture. When NalUnitType is “14”, it represents a TFD picture.
Note that the value of NalUnitType of each picture may be set to other than the above values.

Hereinafter, each NAL unit will be described.
The NAL unit 410 is a delimiter (DELIM) NAL unit and indicates a picture boundary.
The NAL unit 411 is a sequence parameter set (SPS) NAL unit, and the SPS NAL unit 411 describes parameters common to the entire sequence of encoded moving images. The SPS NAL unit 411 is added to a picture that can be redrawn.
The NAL unit 412 is a picture parameter set (PPS) NAL unit, and the PPS NAL unit 412 describes parameters common to a plurality of coded pictures. The PPS NAL unit 412 may be added to other pictures in addition to the redrawable picture.
The NAL unit 413 is a BPSEI NAL unit, and the BPSEI NAL unit 413 is added only to a picture that can be redrawn. In the present embodiment, parameters used for correcting the decoding delay and display delay of pictures after the coupling point in the video decoding device are added to the BPSEI NAL unit 413.
The NAL unit 414 is a PTSEI NAL unit, and the PTSEI NAL unit 414 is added to each picture.
The NAL unit 415 is a slice (SLICE) NAL unit and is an entity of a coded picture.

  The BPSEI NAL unit 413 according to this embodiment includes a set of (N + 1) (where N is an integer equal to or greater than 0) InitialCpbRemovalDelay field and InitialCpbRemovalDelayOffset field. The definition of these fields may be equivalent to the method of Non-Patent Document 1 and MPEG-4 AVC / H.264.

  Note that there are a plurality of InitialCpbRemovalDelay fields and InitialCpbRemovalDelayOffset fields in order to describe InitialCpbRemovalDelay and InitialCpbRemovalDelayOffset that are suitable when the encoded bitstream is transmitted at (N + 1) types of bit rates. Note that InitialCpbRemovalDelayOffset represents the difference between the encoding completion time of the first picture in the video encoding device and the time when transmission of encoded picture data to the video decoding device is started.

  PTSEI NAL unit 414 includes a decoding delay CpbRemovalDelay field, a display delay DpbOutputDelay field, and a NumRemovedTfds field. The NumRemovedTfds field is an example of correction information used for correcting decoding delay and display delay. The NumRemovedTfds field describes the sum of the decoding intervals of the removed pictures from the picture to which the PTSEI is added to the next BPSEI-added picture in decoding order. The decoding interval of a picture is defined as a value obtained by subtracting the CpbRemovalDelay field value of PTSEI added to the previous picture in the decoding order from the CpbRemovalDelay field value of PTSEI added to this picture. When the immediately preceding picture in decoding order is a BLA picture, the CpbRemovalDelay field value of the PTSEI added to the BLA picture is treated as 0. At the stage where the encoded bitstream is generated, the NumRemovedTfds field value is set to 0.

  FIG. 5 is a schematic configuration diagram of a moving image encoding apparatus according to the first embodiment. The moving image encoding apparatus 1 includes a control unit 11, an encoding control unit 12, a picture encoding unit 13, a connection point identification information processing unit 14, and a data combining unit 15. Each of these units included in the video encoding device 1 is mounted on the video encoding device 1 as a separate circuit. Alternatively, these units included in the video encoding device 1 may be mounted on the video encoding device 1 as one integrated circuit in which circuits that realize the functions of the units are integrated. Alternatively, each of these units included in the moving image encoding device 1 may be a functional module realized by a computer program executed on a processor included in the moving image encoding device 1.

  The control unit 11 controls the processing of each unit of the video encoding device 1 when encoding video data or when performing an editing operation on the encoded video data. For example, the control unit 11 applies to the moving image data to be encoded based on the nature of the moving image data such as the position of the scene change, the required reproduction image quality of the encoded moving image data, the compression rate, and the like. Determine the GOP structure to be performed. Then, the control unit 11 notifies the encoding control unit 12 of the GOP structure and the like.

First, a moving image encoding process for encoding moving image data will be described. In the moving image encoding process, the encoding control unit 12 and the picture encoding unit 13 are used.
According to the GOP structure notified from the control unit 11, the encoding control unit 12 encodes each picture in coding order, coding mode (for example, among intra coding mode, forward prediction mode, and bidirectional prediction mode). Or any of these). The encoding control unit 12 determines the CRA picture insertion interval, the number of reordering pictures at the time of encoding, and the maximum display delay according to the encoding mode of each picture, the position in the GOP structure, and the like. In the example shown in FIG. 2, the insertion interval of CRA pictures is 12, the number of reordering pictures is 2, and the maximum display delay is 5. Further, the encoding control unit 12 generates header information of each picture according to these values.

  For example, when the picture type is an I picture (CRA picture), that is, a picture that is coded without referring to another picture, and the picture is not the first picture of the coded moving image data, the coding control unit 12 sets NalUnitType of NUH 720 of each slice in the picture to '12'. The NUH720 NalUnitType of each slice in the first picture of the encoded moving image data is set to 10 (IDR picture). When the number of reordering pictures is 1 or more, the encoding control unit 12 sets NalUnitType to “14” (TFD picture) for a picture that refers to a picture immediately after the CRA picture and whose decoding order and display order are earlier than the CRA picture. ). Also, for a picture immediately after the CRA picture, the display time of which is before the CRA picture, and which does not refer to a picture whose decoding order and display order are earlier than the CRA picture, the encoding control unit 12 sets NalUnitType to “13” (DLP Picture). For the other pictures, the encoding control unit 12 sets NalUnitType to “1” or “2” (TP picture).

  The encoding control unit 12 notifies the picture encoding unit 13 of the NalUnitType of NUH 720 of each slice header in the picture to be encoded. Also, the encoding control unit 12 obtains and notifies the decoding delay CpbRemovalDelay and display delay DpbOutputDelay in the PTSEI of each picture from the picture prediction structure as shown in FIG.

  Further, when the NalUnitType of NUH 720 of each slice in the picture is “10” or “12”, the encoding control unit 12 adds BPSEI to the picture.

  For each picture, the coding control unit 12 notifies the picture coding unit 13 of the coding mode and header information, and instructs the picture coding.

  In accordance with an instruction from the encoding control unit 12, the picture encoding unit 13 encodes the corresponding picture in the designated encoding mode in accordance with any one of the moving image encoding methods capable of interframe predictive encoding. To do. The moving picture coding method to which the picture coding unit 13 is based can be, for example, MPEG-4 AVC / H.264 or MPEG-2. The picture encoding unit 13 stores encoded moving image data including each encoded picture in a storage unit (not shown).

  Next, editing processing when two encoded moving image data are combined will be described. The coupling point identification information processing unit 14 and the data coupling unit 15 are used in this editing process.

  The coupling point identification information processing unit 14 reads, for example, two encoded moving image data selected via a user interface unit (not shown) from a storage unit (not shown). Then, in accordance with an external control signal (not shown), the connection point identification information processing unit 14 combines the second encoded moving image data that is combined later in time among the two encoded moving image data. Specify the point head picture. The control signal from the outside is, for example, the number of encoded pictures from the beginning in the second encoded moving image data, and the connection point identification information processing unit 14 is, for example, the slowest before the number of encoded pictures. The CRA picture is the connection point.

The joint point identification information processing unit 14 is a BLA picture in which a TFD picture may appear from a NalUnitType of “12” in the slice of the CRA picture of the joint point when the reordering number is 1 or more. Rewrite it to '7' to represent this. That is, the value of this NalUnitType is that two encoded moving image data are combined at the connection point, and one or more encoded pictures whose encoding order and decoding order are later than the BLA picture at the connection point. Indicates that it has been removed. In addition, the join point identification information processing unit 14 outputs all the pictures after the CRA picture at the join point in the second encoded moving image data to the data combining unit 15 and also the TFD picture immediately after the join point CRA picture. Give instructions to remove. On the other hand, when the reordering number is 0, for the CRA picture at the connection point, the connection point identification information processing unit 14 sets the NalUnitType of the slice of the picture from “12” to the BLA picture in which the TFD picture and the DLP picture do not appear immediately after Rewrite it to '9' to indicate that.
Next, the joint point identification information processing unit 14 calculates the decoding interval of the TFD picture to be removed, and the value of the NumRemovedTfds field of the non-TFD picture immediately before the TFD picture to be removed follows this non-TFD picture. Add only the decoding interval of the removed TFD picture. When the decoding intervals of each picture are equal, finally, the value of the NumRemovedTfds field of the non-TFD picture becomes the number of deleted pictures in the field unit after the decoding order. Then, the connection point identification information processing unit 14 changes the value of the NumRemovedTfds field of the added PTSEI for a picture earlier in decoding order than the TFD picture to be removed in the second encoded moving image data.

The data combining unit 15 combines the second encoded moving image data output from the combining point identification information processing unit 14 after the first encoded moving image data combined before the combining point. To do. At this time, the data combining unit 15 removes the TFD picture for which normal decoding is not guaranteed immediately after the head picture in the second encoded moving image data. In this case, the data combining unit 15 may also remove the DLP picture by regarding it as a TFD picture.
The data combining unit 15 stores the combined encoded moving image data obtained by combining the first encoded moving image data and the second encoded moving image data in a storage unit (not shown).

FIG. 6 is an operation flowchart of a video encoding process executed by the video encoding apparatus according to the first embodiment. In addition, according to the operation | movement flowchart shown by FIG. 6, the moving image encoder 1 encodes the whole moving image sequence of encoding object.
Prior to starting the encoding process for the entire sequence, for example, the control unit 11 determines a picture prediction structure such as a GOP structure (step S101). Then, the picture prediction structure is notified to the encoding control unit 12.

  The encoding control unit 12 determines an encoding mode to be applied to the encoding target picture based on the notified picture prediction structure, the position of the encoding target picture from the beginning of the moving image data, and the like, and The header information of the encoding target picture is generated (step S102).

  After step S102, the encoding control unit 12 notifies the encoding unit 13 of the data of the current picture to be encoded, the encoding mode of the picture, and header information. The encoding unit 13 encodes the encoding target picture according to the notified encoding mode and header information, and adds header information to the encoded picture data (step S103).

Thereafter, the control unit 11 determines whether or not there is an uncoded picture in the moving image sequence (step S104). When there is an unencoded picture (step S104-Yes), the control unit 11 executes the processing from step S102 onward with the next unencoded picture as an encoding target picture.
On the other hand, when there is no uncoded picture (step S104-No), the control unit 11 ends the coding process.

FIG. 7 is an operation flowchart of a moving image editing process executed by the moving image encoding apparatus according to the first embodiment. In this example, it is assumed that the DLP picture is not deleted and only the TFD picture is deleted.
The joint point identification information processing unit 14 initializes a list L [] of pictures not deleted from the TFD pictures and DLP pictures, and sets a variable m representing a value obtained by adding 2 to the number of pictures not deleted. 2 is initialized (step S201). Note that the variable m may be the number of TFD pictures and DLP pictures that have not been deleted if no TFD picture appears after the last DLP picture in decoding order.

Next, among the first encoded moving image data combined before the connection point, the encoded pictures up to the connection point are sequentially read from the storage unit (not shown) (step S202).
Further, the joining point identification information processing unit 14 sequentially reads out encoded pictures after the joining point from the second encoded moving image data joined after the joining point from a storage unit (not shown) (step S203). ). Next, the connection point identification information processing unit 14 indicates that the NUH NalUnitType value of each slice is a BLA picture for the read first CRA picture in the second encoded moving image data. The value is rewritten (step S204).

  Next, the connection point identification information processing unit 14 determines whether or not the NalUnitType of the next picture is “14” in the decoding order, that is, whether or not it is a TFD picture (step S205). In the case of a TFD picture (step S205—Yes), the joining point identification information processing unit 14 outputs an instruction to delete the TFD picture to the joining unit 15, and the TFD picture is decoded at the decoding interval, that is, in the decoding order. The difference between the CpbRemovalDelay value of PTSEI and the previous picture is added to the 0th to mth entries of the list [], respectively (step S206). Thereafter, the connection point identification information processing unit 14 returns to step S205 and evaluates the NalUnitType of the next picture.

On the other hand, when it is not a TFD picture (step S205—No), the connection point identification information processing unit 14 determines whether or not the NalUnitType of the next picture is “13” in decoding order, that is, whether or not it is a DLP picture (step S207). ). When the next picture is a DLP picture (step S207—Yes), the connection point identification information processing unit 14 increases the variable m by 1 (step S208). Thereafter, the connection point identification information processing unit 14 executes the processing subsequent to step S205 again. On the other hand, if the next picture is not a DLP picture in the decoding order (step S207-No), the next picture is neither a TFD picture nor a DLP picture and is therefore a TP picture. A TFD picture does not exist in a picture whose decoding order is later than a TP picture. Therefore, the connection point identification information processing unit 14 updates the NumRemovedTfds field of the PTSEI of the BLA picture and the DLP picture based on the list L [] (step S209). Specifically, the connection point identification information processing unit 14 sets the value of the NumRemovedTfds field of the PTSEI of the kth picture to L [k] for the m-th non-TFD pictures counted from the BLA picture in decoding order. . Thereafter, the joining point identification information processing unit 14 outputs the BLA picture and all subsequent pictures to the data joining unit 15.
The combining unit 15 combines the pictures after the BLA picture of the second encoded moving image data following the last picture of the first encoded moving image data before the combining point. At that time, the combining unit 15 removes the TFD picture notified to be removed from the combining point identification information processing unit 14.

  Next, a moving picture decoding apparatus that decodes encoded moving picture data encoded or edited by the moving picture encoding apparatus 1 according to the first embodiment will be described.

  FIG. 8 is a schematic configuration diagram of a video decoding device according to the first embodiment. The moving picture decoding apparatus 2 includes a control unit 21, a header information analysis unit 22, a picture decoding / display time determination unit 23, a picture decoding unit 24, and a frame memory 25. Each of these units included in the video decoding device 2 is implemented in the video decoding device 2 as a separate circuit. Alternatively, these units included in the video decoding device 2 may be mounted on the video decoding device 2 as one integrated circuit in which circuits that realize the functions of the units are integrated. Alternatively, these units included in the video decoding device 2 may be functional modules realized by a computer program executed on a processor included in the video decoding device 2.

  The control unit 21 controls processing of each unit of the video decoding device 2 when decoding the encoded video data.

  The header information analysis unit 22 analyzes the header information of the encoded moving image data and calculates parameters necessary for determining the picture decoding / display time, for example, NalUnitType of each picture, and CpbRemovalDelay, DpbOutputDelay, and NumRemovedTfds included in PTSEI. The picture decoding / display time determination unit 23 outputs the result.

  The picture decoding / display time determination unit 23 checks the NUH of the slice of the decoding target picture, which is passed from the header information analysis unit 22. When NalUnitType of NUH is “7”, “8”, or “9”, the picture decoding / display time determination unit 23 determines that the decoding target picture is a BLA picture.

  When the decoding target picture is a BLA picture, the picture decoding / display time determination unit 23 uses, as the decoding delay CpbRemovalDelay of the BLA picture, a value obtained as follows instead of the CpbRemovalDelay of the PTSEI added to the BLA picture. Use.

  The picture decoding / display time determination unit 23 obtains the sum A of the decoding intervals of each picture from the picture next to the picture to which the BPSEI immediately before the BLA picture is added to the BLA picture. Then, the picture decoding / display time determining unit 23 sets the decoding delay CpbRemovalDelay of the BLA picture to A. If the decoding interval of each picture is equal, the picture decoding / display time determination unit 23 calculates the number of pictures in field units from the picture next to the picture to which the BPSEI immediately before the BLA picture is added to the BLA picture. BLA picture decoding delay CpbRemovalDelay may be used.

  The picture decoding / display time determination unit 23 further checks the NumRemovedTfds field of the PTSEI of the BLA picture. When the value of NumRemovedTfds is not zero, the picture decoding / display time determination unit 23 determines that the TFD picture immediately after the BLA picture has been removed, and subtracts NumRemovedTfds from the display delay DpbOutputDelay of the BLA picture, The display delay after correction of the BLA picture is DpbOutputDelay.

The picture decoding / display time determination unit 23 also performs the following processing for the pictures after the BLA picture in the decoding order until the next BPSEI-added picture.
For the decoding delay CpbRemovalDelay of the picture to be processed, the picture decoding / display time determination unit 23 calculates the NumRemovedTfds of PTSEI added to the BLA picture and the NumRemovedTfds of PTSEI added to the processing target picture from the original value of CpbRemovalDelay. The value obtained by subtracting the difference value (corresponding to the sum of the decoding intervals of the removed TFD pictures after the processing target picture) is defined as a corrected decoding delay CpbRemovalDelay. For the display delay DpbOutputDelay of the processing target picture, the picture decoding / display time determining unit 23 subtracts the PTSEI NumRemovedTfds added to the processing target picture from the original value of DpbOutputDelay, and the corrected display delay Set to DpbOutputDelay.
Furthermore, for the TP picture, the picture decoding / display time determination unit 23 subtracts the value obtained by subtracting the PTSEI NumRemovedTfds added to the BLA picture from the original value of the decoding delay CpbRemovalDelay of the picture, and the value of the corrected CpbRemovalDelay To do.

  For pictures other than those described above, the picture decoding / display time determining unit 23 sets the decoding delay CpbRemovalDelay and display delay DpbOutputDelay of the picture as CpbRemovalDelay and DpbOutputDelay of the PTSEI added to the picture, respectively.

  The picture decoding / display time determining unit 23 determines the decoding time of each picture based on the decoding delay CpbRemovalDelay, and issues a decoding instruction to the picture decoding unit 24 at the decoding time. The picture decoding / display time determination unit 23 determines the display time of each picture based on the display delay DpbOutputDelay, and issues a display instruction to the frame memory 25 at the display time.

  When receiving a decoding instruction for the decoding target picture, the picture decoding unit 24 decodes the decoding target picture using the reference picture stored in the frame memory 25. The picture decoding unit 24 stores the decoded picture in the frame memory 25. Note that the picture decoding unit 24 performs a decoding process in accordance with an encoding method similar to the encoding method that the encoding unit of the moving image encoding device 1 conforms to.

  The frame memory 25 stores the decoded picture. In addition, the frame memory 25 outputs the decoded picture to the picture decoding unit 24 as a reference picture of a picture to be decoded after that picture. Further, the frame memory 25 outputs a picture to a display unit (not shown) in accordance with the display instruction received from the picture decoding / display time determination unit 23.

  FIG. 9 is an operation flowchart of a video decoding process executed by the video decoding device according to the first embodiment. Note that the moving image decoding apparatus 2 decodes the entire moving image sequence to be decoded in accordance with the operation flowchart shown in FIG.

  Prior to starting the decoding process for the entire sequence, the control unit 21 initializes a variable flag and sets it to 0 (step S301). The variable flag is a variable indicating whether or not the non-BLA picture needs to be corrected for CpbRemovalDelay and DpbOutputDelay. If the variable flag is '1', CpbRemovalDelay and DpbOutputDelay need to be modified. If the variable flag is '0', CpbRemovalDelay and DpbOutputDelay need not be modified.

Next, the header information analysis unit 22 analyzes the header information of the decoding target picture and outputs parameters necessary for determining the picture decoding / display time to the picture decoding / display time determining unit 23. The header information of the decoding target picture is analyzed (step S302).
The picture decoding / display time determination unit 23 determines whether the variable flag is “1” (step S303).
When the variable flag is “1” (step S303—Yes), the picture decoding / display time determination unit 23 sets the decoding delay CpbRemovalDelay of the decoding target picture that is a non-BLA picture, the NumRemovedTfds of the decoding target picture, and the previous BLA picture. Correction is performed using NumRemovedTfds (step S304). Further, the display delay DpbOutputDelay of the decoding target picture is corrected using NumRemovedTfds of the decoding target picture.

  After step S304, or when the variable flag is '0' in step S303 (step S303-No), the picture decoding / display time determination unit 23 determines whether BPSEI is added to the decoding target picture (step S303: No). Step S305).

  When BPSEI is added to the decoding target picture (step S305—Yes), the picture decoding / display time determination unit 23 determines whether or not the decoding target picture is a BLA picture (step S306). If the decoding target picture is not a BLA picture (step S306-No), the picture decoding / display time determination unit 23 resets the variable flag to 0 (step S307).

  On the other hand, if the decoding target picture is a BLA picture (step S306-Yes), the picture decoding / display time determination unit 23 corrects the decoding delay CpbRemovalDelay and display delay DpbOutputDelay of the picture and sets the variable flag to 1 (Step S308). In this case, as described above, the picture decoding / display time determination unit 23 sets the decoding delay CpbRemovalDelay of the BLA picture to the sum of the decoding intervals of each picture from the picture next to the picture to which the previous BPSEI is added to the BLA picture. And Also, the picture decoding / display time determination unit 23 sets the display delay DpbOutputDelay of the BLA picture to a value obtained by subtracting NumRemovedTfds from the original value.

After step S307 or S308, or when BPSEI is not added to the decoding target picture in step S305 (step S305-No), the control unit 21 includes an undecoded picture in the encoded moving image data. Whether or not (step S309). If there is an undecoded picture (step S309—Yes), the control unit 21 shifts the processing to step S302. Then, the processing from step S302 onward is repeated with the next picture as the decoding target picture in the decoding order.
On the other hand, if there is no undecoded picture (step S309-No), the control unit 21 ends the moving image decoding process.

A specific example of the NumRemovedTfds derivation method, CpbRemovalDelay, and DpbOutputDelay correction method described above will be described with reference to FIG.
Each block shown in the first moving image encoded data 1001 combined before the combining point represents one picture, and the characters in each block are encoded as in FIG. Represents the order of input to the video coding mode and the moving picture coding apparatus.

  In this example, the second moving image encoded data 1002 is combined immediately after the last picture B11 of the first moving image encoded data. Also in the second moving image encoded data 1002, each block represents one picture, and the characters in each block represent the encoding mode and the order of input to the moving image encoding device. The arrows shown on the upper side of the second moving image encoded data 1002 represent reference pictures to which the pictures B4 to B7 to be encoded refer in forward frame prediction, respectively. Also, the arrows shown below the second moving image encoded data 1002 indicate reference pictures to which the pictures B4 to B7 to be encoded refer in backward frame prediction, respectively.

  In the second moving image encoded data 1002, the pictures B4, B2, B1, B3, and B7 are TFD pictures as described below the second moving image encoded data 1002. Pictures B5 and B7 are DLP pictures.

  A block sequence 1003 shown below the second moving image encoded data 1002 is a value of a decoding delay CpbRemovalDelay and a display delay DpbOutputDelay included in PTSEI added to each picture of the second moving image encoded data 1002. Represents. Each block on the upper side of the block sequence 1003 indicates the value of the decoding delay CpbRemovalDelay of the picture of the second moving image encoded data 1002 positioned above the block. On the other hand, each lower block in the block sequence 1003 indicates the value of the display delay DpbOutputDelay of the picture of the second moving image encoded data 1002 positioned above that block.

  Below the block string 1003, combined moving image encoded data 1004 in which the first moving image encoded data 1001 and the second moving image encoded data 1002 are combined is shown. In this example, in the combined moving image encoded data 1004, the TFD pictures B4, B2, B1, B3, and B7 in the second moving image encoded data 1002 are deleted.

  Below the block row 1004, NumRemovedfds 1005 of the combined moving image encoded data 1004 is shown. The NumRemovedfds of I8, which is a BLA picture, is the sum of the decoding intervals of the removed TFD pictures (B4, B2, B1, B3, B5) that become I8 or later in the decoding order, in this example, I8 or later and removed Stores 10 which is the number of pictures in a field unit. The NumRemovedfds of B6, which is a DLP picture, is the sum of the decoding intervals of the removed TFD picture (B5) after decoding B6 in the decoding order. In this example, it is the number of field units of the removed pictures after B6. 2 is stored. After picture B7, there is no removed TFD picture that is later in the decoding order, so NumRemovedfds remains 0.

  A block sequence 1006 below the NumRemovedfds 1005 of the combined moving image encoded data 1004 represents the values of the decoding delay CpbRemovalDelay and the display delay DpbOutputDelay of the combined moving image encoded data 1004 corrected based on NumRemovedfds. Each block above block sequence 1006 represents a modified decoding delay CpbRemovalDelay for the picture shown above, and each block below block sequence 1006 is a modified version for the picture shown above Represents the display delay DpbOutputDelay.

  For the BLA picture I8, 10 obtained by subtracting the value 10 of NumRemovedfds from the original value 20 of the display delay DpbOutputDelay becomes the corrected display delay DpbOutputDelay. As a result, the display delay DpbOutputDelay of the picture I8 after correction is also the decoding time of the picture I8 when the display time of the picture B9 with the largest reordering number after the picture I8 is used as a reference, similarly to the value before correction. To the display time.

  Also, for the DLP picture B6, 2 is obtained by subtracting the difference 8 between the NumRemovedfds (= 10) of the picture I8 and the NumRemovedfds (= 2) of the picture B6 from the original value 10 of the decoding delay CpbRemovalDelay and the corrected decoding delay CpbRemovalDelay and Become. Also, 4 obtained by subtracting NumRemovedfds (2) of picture B6 from the original value 6 of display delay DpbOutputDelay of picture B6 is the corrected display delay DpbOutputDelay. For each picture after picture B7, the value of NumRemovedfds is 0, so the value obtained by subtracting NumRemovedfds of picture I8 from the original value of decoding delay CpbRemovalDelay is the corrected decoding delay CpbRemovalDelay. On the other hand, the display delay DpbOutputDelay for the pictures after the picture B7 does not change.

  As described above, the moving picture coding apparatus according to this embodiment is determined by the number of pictures deleted by combining two or more pieces of encoded moving picture data without decoding. The decoding delay and display delay parameter values determined at the time of encoding need not be corrected by simply recording the decoding delay and display delay correction parameters in the encoded moving image data. And the moving picture decoding apparatus according to this embodiment can correct the decoding delay and the display delay of each picture using the parameters for correcting the decoding delay and the display delay when the encoded moving picture data is combined. Each picture can be decoded and displayed at an appropriate timing.

  Next, a second embodiment will be described. The second embodiment differs from the first embodiment in the structure of encoded moving image data.

  The structure of encoded moving image data in the second embodiment will be described with reference to FIG. Similar to the structure of the coded picture according to the first embodiment shown in FIG. 4, the data structure 1100 of one picture includes six types of NAL units 1110 to 1115. Among these, the BPSEI 1113 and the PTSEI 1114 are different from the BPSEI 413 and the PTSEI 414 shown in FIG. On the other hand, DELIM 1110, SPS 1111, PPS 1112, SLICE 1115, and NUH 1120 are equivalent to DELIM 410, SPS 411, PPS 412, SLICE 415, and NUH 420 shown in FIG. 4, respectively.

The BPSEI 1113 added 1 to the variable m, which is the number obtained by adding 2 to the number of pictures not deleted from the TFD picture and DLP picture located between the BLA picture and the next CRA picture at the time of combination. Contains a NumEntries field that represents a number. Further, the BPSEI 1113 includes NumEntries AltCpbRemovalDelayOffset fields and AltDpbOutputDelayOffset fields. The NumEntries field, the AltCpbRemovalDelayOffset field, and the AltDpbOutputDelayOffset field are another example of correction information for correcting the decoding delay and the display delay.
On the other hand, unlike PTSEI 440, PTSEI 1140 does not include a NumRemovedTfds field.
When the NumEntries field is 0, the video decoding device does not have to change the values of CpbRemovalDelay and DpbOutputDelay in the picture to which BPSEI is added and the subsequent pictures (up to the next BPSEI-added picture). On the other hand, when NumEntries is not zero, the video decoding device calculates the correction value of the decoding delay CpbRemovalDelay of the kth picture in the decoding order from the picture to which BPSEI is added, and AltCpbRemovalDelayOffset [k ] Is subtracted. Similarly, the moving picture decoding apparatus calculates the corrected value of the display delay DpbOutputDelay by subtracting AltDpbOutputDelayOffset [k] from the original value of the display delay DpbOutputDelay.

  As described above, the SEI for storing the correction values of the CpbRemovalDelay field and the DpbOutputDelay field is different from that of the first embodiment. Therefore, the moving picture coding apparatus according to the second embodiment differs from the moving picture coding apparatus according to the first embodiment in the processing of the connection point identification information processing unit 14. Therefore, hereinafter, the process of the connection point identification information processing unit 14 will be described.

  The connection point identification information processing unit 14 stores a value obtained by adding 1 to the variable m calculated based on the operation flowchart of the moving image editing process in FIG. 7 in the NumEntries field. Furthermore, the connection point identification information processing unit 14 puts a value of L [0] −L [k] in the kth (k = [0, m−1]) th AltCpbRemovalDelayOffset field. Also, the value of L [k] is entered in the kth AltDpbOutputDelayOffset field.

Next, the operation of the video decoding device according to the second embodiment will be described. The video decoding device according to the second embodiment also has the same configuration as the video decoding device according to the first embodiment. However, in the moving picture decoding apparatus according to the second embodiment, the processing of the picture decoding / display time determining unit 23 is different from that of the first embodiment. Therefore, hereinafter, the processing of the picture decoding / display time determination unit 23 will be described.
In the BPSEI-added picture immediately before the decoding target picture, only when the NumEntries field of the BPSEI is not zero, the picture decoding / display time determination unit 23 sets the values of the decoding delay CpbRemovalDelay and the display delay DpbOutputDelay of the PTSEI of the decoding target picture as follows: To correct.

  The decoding order is k (k = 0, 1, 2,...) From the immediately preceding BPSEI-added picture (in this case, the BLA picture). When k is equal to or larger than NumEntries, the picture decoding / display time determination unit 23 corrects the value obtained by subtracting the value of AltCpbRemovalDelayOffset [NumEntries-1] from the original value of the decoding delay CpbRemovalDelay of the kth picture. Of CpbRemovalDelay. On the other hand, when k is smaller than NumEntries, the picture decoding / display time determining unit 23 obtains a value obtained by subtracting the value of AltCpbRemovalDelayOffset [k] from the original value of the decoding delay CpbRemovalDelay after correcting the kth picture. The value obtained by subtracting the AltDpbRemovalDelayOffset value from the original value of the display delay DpbRemovalDelay as the CpbRemovalDelay value is set as the corrected DpbOutputDelay value.

  FIG. 12 illustrates the moving picture coding apparatus or the moving picture decoding by the operation of a computer program that implements the functions of the respective units of the moving picture coding apparatus or the moving picture decoding apparatus according to any one of the above-described embodiments or modifications thereof. It is a block diagram of the computer which operate | moves as an apparatus.

  The computer 100 includes a user interface unit 101, a communication interface unit 102, a storage unit 103, a storage medium access device 104, and a processor 105. The processor 105 is connected to the user interface unit 101, the communication interface unit 102, the storage unit 103, and the storage medium access device 104 via, for example, a bus.

  The user interface unit 101 includes, for example, an input device such as a keyboard and a mouse, and a display device such as a liquid crystal display. Alternatively, the user interface unit 101 may include a device such as a touch panel display in which an input device and a display device are integrated. For example, the user interface unit 101 outputs an operation signal for selecting moving image data to be encoded, encoded moving image data to be edited, or encoded moving image data to be decoded to the processor 105 in accordance with a user operation. The user interface unit 101 may display the decoded moving image data received from the processor 105.

  The communication interface unit 102 may include a communication interface for connecting the computer 100 to a device that generates moving image data, for example, a video camera, and a control circuit thereof. Such a communication interface can be, for example, Universal Serial Bus (Universal Serial Bus, USB).

  Furthermore, the communication interface unit 102 may include a communication interface for connecting to a communication network according to a communication standard such as Ethernet (registered trademark) and a control circuit thereof.

  In this case, the communication interface unit 102 acquires the moving image data to be encoded, the encoded moving image data to be edited, or the encoded moving image data to be decoded from another device connected to the communication network. Data is passed to the processor 105. Further, the communication interface unit 102 may output the encoded moving image data, the combined encoded moving image data, or the decoded moving image data received from the processor 105 to another device via the communication network.

  The storage unit 103 includes, for example, a readable / writable semiconductor memory and a read-only semiconductor memory. The storage unit 103 stores a computer program for executing a moving image encoding process or a moving image decoding process executed on the processor 105, and data generated during or as a result of these processes.

  The storage medium access device 104 is a device that accesses a storage medium 106 such as a magnetic disk, a semiconductor memory card, and an optical storage medium. For example, the storage medium access device 104 reads a computer program for moving image encoding processing or moving image decoding processing executed on the processor 105 stored in the storage medium 106 and passes the computer program to the processor 105.

  The processor 105 encodes moving image data by executing a computer program for moving image encoding processing according to any one or each of the above embodiments. Alternatively, the processor 105 generates combined encoded moving image data obtained by combining two encoded moving image data. Then, the processor 105 stores the generated combined encoded moving image data in the storage unit 103 or outputs it to another device via the communication interface unit 102. Furthermore, the processor 105 decodes the encoded moving image data by executing a computer program for moving image decoding processing according to any one of the above-described embodiments or modifications. Then, the processor 105 stores the decoded moving image data in the storage unit 103 and displays it on the user interface unit 101 or outputs it to another device via the communication interface unit 102.

  When executed on a computer, a computer program that realizes the functions of the respective units of the moving picture coding apparatus and the moving picture decoding apparatus according to the above-described embodiment or its modification is recorded on a recording medium such as a semiconductor memory or an optical recording medium. It may be recorded and distributed. However, such a recording medium does not include a carrier wave.

  The moving image encoding device and the moving image decoding device according to the above-described embodiment or its modification are used for various applications. For example, the moving image encoding device and the moving image decoding device are incorporated in a video camera, a video transmission device, a video reception device, a videophone system, a computer, or a mobile phone.

  All examples and specific terms listed herein are intended for instructional purposes to help the reader understand the concepts contributed by the inventor to the present invention and the promotion of the technology. It should be construed that it is not limited to the construction of any example herein, such specific examples and conditions, with respect to showing the superiority and inferiority of the present invention. Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Moving image encoder 11 Control part 12 Encoding control part 13 Picture encoding part 14 Connection point identification information processing part 15 Data connection part 2 Video decoding apparatus 21 Control part 22 Header information analysis part 23 Picture decoding and display time determination Unit 24 picture decoding unit 25 frame memory

Claims (2)

A moving image encoding method for generating combined moving image data encoded by combining first moving image data and second moving image data encoded by an inter-frame prediction method,
In the moving picture decoding apparatus, the first encoded picture combined after the first moving picture data among the pictures included in the second moving picture data is encoded more than the first encoded picture. A decoding delay for continuously decoding and displaying pictures after the first encoded picture in the second moving image data even when one or more pictures in the later order are removed; Obtaining display delay correction information and adding the correction information to the combined moving image data;
Including
The moving picture coding method in which the correction information is calculated based on a decoding interval between each picture to be removed and the picture to be removed and a picture immediately before the decoding order.   The correction information is obtained for the first encoded picture and a picture having a decoding time later than that of the first encoded picture and an earlier display time, and the value of the correction information is calculated in the combined moving image data. For each picture to be removed from the second moving image data after the decoding order of the picture for which the correction information is obtained, decoding of the picture to be removed and the picture immediately before the decoding order The moving image encoding method according to claim 1, wherein the moving image encoding method is a value corresponding to a total of intervals.

Images