JP2010193327A - Image decoding apparatus and packet loss compensation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent a reproduced image from being considerably broken down by an IP packet loss, and to avoid an increase in delay time when performing a refresh operation for complete recovery from influences of the packet loss. <P>SOLUTION: Which macro block portion in an image region is missed by an IP packet loss is acquired by analyzing data before and after the packet loss and a bit stream of the missed portion is generated and inserted according to a system compliant to syntax rules of an encoding system. Furthermore, encoding is performed while reducing a redundant degree in time directions of a decoded image decoded as a compensation image and a decoded IDR picture image. thereby achieving rate smoothing by generating a switching stream as repair switch image data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image decoding apparatus and a packet loss compensation method for highly efficient encoded (compressed) image data transmitted from an IP network.

  When receiving an IP packet of high-efficiency encoded (compressed) image data transmitted from the IP network, and decoding and outputting the image, if the IP packet is lost during transmission, it is recovered There is a need to.

  Various techniques implemented in an image decoder have been conventionally proposed as techniques for interpolating received and reproduced images due to packet loss and the like. For example, in Patent Document 1, an incomplete bit stream due to packet loss or the like is given to an image decoder as a given bit stream, a decoded image is formed in the image decoder, and a defective area is detected and prioritized. The moving image of the image is displayed by taking an image interpolation means for the defective area.

  Further, in Patent Document 2, a missing region due to packet loss and its area are specified, texture analysis is performed on the region around the missing region, and image correction that compensates for the missing region is generated. Furthermore, as a method for taking measures against packet loss as an image signal transmission device, for example, in Patent Document 3, an image region included in a lost packet of a transmitted image is specified based on packet loss information from a reception source. A means for prohibiting the region from being a reference image is taken.

  However, in the above-described conventional technology, loss and restoration of image data due to IP packet loss are mainly due to error shielding in the image decoder, and are instantaneous decoding refresh operations periodically or on demand. There were the following problems.

  The first problem is that when transmitting a highly efficient encoded image signal using an IP network, it becomes non-standard data due to a major failure of the reproduced image due to IP packet loss and lack of data in the packet loss part. This means that there is a problem in robustness such as stopping stable operation due to input to the decoder.

  The reason is that the data omission process is performed based on erroneous determination of misplaced parts and mis-synchronization because error shielding is performed based on limited information in the image decoder. In such a method, since the bit stream data in which the data missing part is simply combined before and after is input to the image decoder, the corresponding part becomes non-standard data, and the application of the specified violation stream causes the image decoder. It will affect the operational stability.

  The second problem is that when performing instantaneous or on-demand instantaneous decoding refresh operation as a means for recovering the mismatch between the reference image on the transmission side and the reception side due to data loss, the amount of generated information increases in bursts, the network bandwidth This will cause an increase in delay and delay.

  The reason is that when the reference image is matched between the transmission and reception, the image used for performing the refresh operation is directly transmitted by performing intra coding that does not use the feature having high correlation between frames. This is because smooth transmission by a rate buffer is required to transmit data transmission of a frame with a large amount of generated information at a fixed bit rate, resulting in an increase in transmission delay time. On the other hand, if transmission is performed for a certain period of time in order to suppress the delay time, the network band during that time increases, causing a problem of tight network bandwidth.

JP 2007-329858 A Japanese Patent Laid-Open No. 2007-281692 JP 2007-288604 A

  The first problem to be solved is that, when an image signal that has been encoded with high efficiency is transmitted using an IP network, there is a significant prevention of failure of a reproduced image due to IP packet loss, and a stable operation of decoding operation is stopped. It is a point to get.

  The second problem to be solved is that when the refresh operation is performed, the amount of generated information is increased in bursts, the network bandwidth is tightened, and the delay time is increased.

  The most important feature of the present invention is that the influence of packet loss on a reproduced image is minimized by generating and inserting a bit stream of all macroblock portions of an image region that is lost due to IP packet loss. To do.

  In addition, when relaying an image used for the refresh operation, encoding is performed while reducing redundancy in the time direction based on the correlation with the reproduced image obtained as described above.

  According to the present invention, when a highly efficient encoded image signal is transmitted using an IP network, the reproduction image is largely prevented from being broken due to an IP packet loss, and the stable operation stop of the decoding operation is avoided. Have advantages.

  The reason is that the macroblock part of the image area is lost due to the IP packet loss by analyzing the data before and after the packet loss, and the bit stream of the missing part conforms to the syntax rules of the encoding method. By generating and inserting using the above-mentioned method, the continuity of the macroblock decoding of the processing in the image data decoding unit is ensured, and the compensation image in which the influence of the packet loss on the reproduced image is minimized is generated. This is to improve the stable operation.

  In addition, there is a second advantage that delay time in the case of restoring the mismatch between the reference image on the transmission side and the reception side due to data loss is reduced, and the burstiness of the network bandwidth is avoided.

  The reason is that the switching image is generated by reducing the redundancy in the time direction between the decoded image decoded as the compensation image and the decoded IDR picture image, and the switch image data for repair is rate smoothed. Is to transmit.

It is a block diagram which shows Example 1 of this invention apparatus. It is a timing chart for demonstrating operation | movement of Example 1 of this invention apparatus. It is a figure for demonstrating operation | movement of the image coding data generation part for loss compensation. It is a block diagram which shows Example 2 of this invention apparatus. It is a block diagram which shows the detail of the stream switch in Example 2. FIG. It is a timing chart for demonstrating operation | movement of Example 2 of this invention apparatus.

  Analyzing the data before and after the packet loss to determine which macroblock part of the image area was lost due to the IP packet loss, with the aim of preventing a significant failure of the playback image due to the IP packet loss and avoiding the stable operation stop of the decoding operation This is realized by generating and inserting the bit stream of the missing part by a method compliant with the syntax rule of the encoding method (Example 1).

  In addition, a decoded image decoded as a compensation image for the purpose of avoiding a burst-like increase in the amount of generated information when performing a refresh operation to completely recover from the effects of packet loss, tight network bandwidth, and an increase in delay time This is realized by rate smoothing by generating a switching stream as restoration switch image data by reducing the redundancy in the time direction with the decoded IDR picture image and encoding (Example 2).

[Description of configuration]
FIG. 1 is a block diagram showing the configuration of Embodiment 1 of the device of the present invention. The image decoding apparatus receives an IP packet of high-efficiency encoded (compressed) image data transmitted from an IP network as input data a, and recovers and decodes the packet loss that the IP packet received during transmission. Output image m is output. To compensate for packet loss, a macroblock which is a processing unit in a packet is used.

  In FIG. 1, an RTP packet receiver 1 receives input data a and outputs RTP data b. The image data extraction unit 2 extracts high-efficiency encoded image data e from the RTP data b supplied from the RTP packet reception unit 1. The post-recovery first MB number detection unit 6 detects the number j of the first macroblock immediately after the recovery included in the high-efficiency encoded image data e. The loss compensation image encoded data generation unit 8 counts the number of macroblocks where packet loss occurs from the number j of the first macroblock immediately after restoration and the number h of the last decoded macroblock, and the macroblock skip information Is generated as loss-compensated image encoded data k. The loss-compensated image encoded data k is generated by a method that complies with the syntax rules of the encoding method for the missing portion bit stream obtained from the leading macroblock number j and the decoded final macroblock number h. .

  The data multiplex insertion unit 9 inserts the loss compensation image encoded data k into the high efficiency encoded image data e. The image data decoding unit 10 performs image decoding on the loss-compensated image encoded data l supplied from the data multiplex insertion unit 9, outputs the output image m to the outside, and outputs the decoded macroblock number i To do.

  The number j of the first macroblock immediately after restoration and the number h of the final decoded macroblock can be obtained as follows. First, the sequence number extraction unit 3 extracts the sequence number c of the RTP packet from the RTP data b, and the packet loss occurrence determination unit 4 determines the presence or absence of the loss of the RTP data b by verifying the continuity. As a result, a packet loss detection notification d is output if there is a loss, and a packet loss non-detection notification f is output if there is no loss.

  When receiving the packet loss detection notification d, the decoded final MB number holding unit 5 holds the number i of the decoded macroblock at that time and sets the number h of the decoded final macroblock as a loss compensation image encoded data generation unit. Output to 8. Further, when the previous packet loss detection unit 7 receives the packet loss non-detection notification f, the previous packet loss detection unit 7 detects the packet loss occurrence state of the previous packet based on the presence or absence of the packet loss detection notification d at that time. As a result, if the packet loss detection notification d is present, the packet loss recovery detection notification g is output to the head MB number detection unit 6 after recovery. The post-restoration leading MB number detection unit 6 outputs the macroblock number included in the high-efficiency coded image data e at that time to the loss compensation image coded data generation unit 8 as the number j of the leading macroblock immediately after restoration. .

[Description of operation]
FIG. 2 is a timing chart for explaining the operation of the image decoding apparatus, and illustrates the relationship among the input data a, the loss compensation image encoded data k, and the output image m. Assume that the input data a sequentially reaches the RTP packet receiving unit 1 as R0, R1, R2 and R3 are unreachable due to packet loss, and then R4 arrives.

  The packet loss occurrence determination unit 4 determines the packet loss of R2 and R3 when R4 arrives. At this time, since the image data decoding unit 10 that decodes the output image m completes the decoding of R1 and notifies the decoded final MB number holding unit 5 of the decoded macroblock number i, the decoded final macroblock number h is supplied to the loss compensation image encoded data generation unit 8. The value of the decoded final macroblock number at this time is MB_end.

  When R4 arrives, the leading macroblock number j included in R4 is supplied to the loss compensation image encoded data generation unit 8. The value of the number of the first macroblock at this time is MB_start. Further, the initial value of the macroblock number of each image frame is set to 0, and the maximum macroblock number is set to MB_num. Here, depending on the magnitude relationship between MB_end and MB_start, the positional relationship included in the image frame of the packet loss is different, and the generation method of the loss compensation image encoded data k is different.

FIG. 3 is a diagram for explaining the operation of the loss-compensated image encoded data generation unit 8 and shows the pattern divided according to the magnitude relationship between MB_end and MB_start.
(1) Pattern 1
In the case of 0 <MB_end <MB_start <MB_num In this case, the range given to the image of packet loss is a pattern limited to one frame. (MB_start−MB_end−1) pieces of macroblock skip information are generated as loss compensation image encoded data k.
(2) Pattern 2
In the case of MB_end = MB_num In this case, the range given to the image of packet loss is a pattern that is considered from the head of the image frame to (MB_start-1). Frame head header information and (MB_start) pieces of macroblock skip information are generated as loss compensation image encoded data k.
(3) Pattern 3
When MB_start = 0 In this case, the range given to the image of packet loss is a pattern that is considered from (MB_end + 1) to the last part. (MB_num−MB_end−1) pieces of macro block skip information are generated as loss compensation image encoded data k.
(4) Pattern 4
In the case of 0 <MB_start = <MB_end <MB_num In this case, the range given to the image of packet loss is considered to extend over 2 frames. The range of packet loss given to an image is from (MB_end + 1) to the last part of the first frame, and from the head of the image frame to (MB_start-1) of the subsequent frame. (MB_num−MB_end-1) pieces of macroblock skip information, frame head header information, and (MB_start) pieces of macroblock skip information are generated as loss compensation image encoded data k.
(5) Pattern 5
In the case of MB_end = MB_num and MB_start = 0 In this case, the range given to the packet loss image is considered to be just one frame from the beginning to the end of the frame. Frame head header information and (MB_num + 1) pieces of macroblock skip information are generated as loss compensation image encoded data k.
(6) Pattern 6
In the case of MB_start = MB_end + 1 In this case, the range given to the image of packet loss is the range of just one frame from (MB_start) to (MB_start-1) of the subsequent frame. This is a special case of pattern 4, and the treatment is based on pattern 4.

  The loss compensation image encoded data k generated as described above is shown as compensated image data C2 in FIG. The compensated image data C2 is inserted between the input data R1 and R4 by the data multiplex inserting unit 9, decoded as the compensated image C23 by the image data decoding unit 10, and output to the outside as a part of the output image m. . As a result, the continuity of the macroblock decoding of the processing in the image data decoding unit 10 is guaranteed, and a compensated image is generated in which the influence of the packet loss on the reproduced image is minimized. This effect is for the compensated image C23 on the subsequent R4 decoded image.

  As described above, in the first embodiment, a compensated image is generated in which the continuity of the macroblock decoding process in the image data decoding unit 10 is guaranteed and the influence of packet loss on the reproduced image is suppressed to a minimum. . However, when multiple image decoding devices are connected in multiple stages and an image signal is transmitted between image decoding devices via a transmission network, the difference from the reference image is often used to reduce the amount of information of the image signal. When an image signal obtained by encoding a signal is to be transmitted, a deviation between a packet loss compensation image and a regular reference image continuously affects the subsequent reproduced image signal.

  In order to avoid this, after the packet loss occurs, a refresh operation for making the contents of the reference frame memory of the encoder and the reference frame memory of the decoder identical is necessary early on. The loss-compensated image encoded data k contributes to an operation in which the influence of image quality deterioration due to packet loss until the refresh operation is performed is minimized, but essentially does not require a refresh operation. It is not a thing. An image used for performing a refresh operation in one image frame is called an instantaneous decoding refresh picture (IDR picture) in the ITU-T recommendation H.264 encoding method in the sense that an instantaneous decoding refresh operation is possible.

  By the way, in an IDR picture, encoding is not performed using inter-frame differences in order to reduce redundancy in the time direction that video signals generally have, so the amount of information generated in the encoding result is based on a method in which inter-frame differences are performed. It has the feature that it is much more than that. Therefore, when an IDR picture is transmitted via a transmission network, it is necessary to increase the transmission path band. As a result, when a constant-rate transmission path is used to send one frame of IDR picture, it is necessary to sacrifice low delay operation such as increased delay time and skipping of subsequent frames and motion reproducibility operation. .

  The second exemplary embodiment is suitable for an image relay type image decoding apparatus that is connected in multiple stages and transmits an image signal via a transmission network. Here, along with the instantaneous decoding refresh operation, the IDR picture is not directly relayed and transmitted, but is encoded by reducing the redundancy in the time direction between the decoded image decoded as the compensation image and the decoded IDR picture image. By adopting a rate smoothing technique that generates a switching stream as restoration switch image data, the above sacrifice is avoided.

[Description of configuration]
FIG. 4 is a block diagram showing the configuration of Embodiment 2 of the device of the present invention. This image decoding apparatus includes an expiration time detection unit 22, a timer 23, a retransmission data detection sampling unit 24, an IDR picture detection sampling unit 25, and an IDR picture decoding in each of the components 1 to 8 and 10 of the first embodiment (FIG. 1). 26, a rate smoothing restoration data generator 27 for recovery, a retransmission request unit 31, and an RTP packet unit 38 are added, and the data multiplex insertion unit 9 is replaced with a stream switch 36. Since the functions of the components having the same names in FIG. 1 and FIG. 2 are substantially the same, their explanation is omitted, and only the difference in connection is mentioned.

  Based on the packet loss detection notification Q supplied from the packet loss occurrence determination unit 30, the retransmission request unit 31 outputs a retransmission request signal R to the transmission source of the input data A in order to make a retransmission request for the lost packet. A retransmission request notification F is output to 23. The retransmission data detection / extraction unit 24 detects and extracts the retransmitted data from the RTP data B supplied from the RTP packet reception unit 21, and outputs it to the IDR picture detection / extraction unit 25 as RTP data information H. Also, a retransmission data detection notification E is output to the timer 23. The timer 23 sets a predetermined expiration time by the retransmission request notification F, starts a timer operation, and stops the timer operation based on the retransmission data detection notification E. The expiration time detection unit 22 detects the expiration time of the timer 23 based on the timer information D supplied from the timer 23 and sends a refresh request C to the transmission source of the input data A.

  The IDR picture detection / sampling unit 25 detects an IDR picture capable of an instantaneous decoding refresh operation from the RTP data information H, outputs IDR picture data J to the IDR picture decoding unit 26, and other high-efficiency encoded image data Is sent to the image data extraction unit 28 and the sequence number extraction unit 29. The image data extraction unit 28 and the sequence number extraction unit 29 use the RTP data signal I as an input signal, like the image data extraction unit 2 and the sequence number extraction unit 3 in FIG. 1 use the RTP data b as an input signal.

  The IDR picture decoder 26 performs a decoding operation on the IDR picture data J, and outputs the IDR picture image K to the restoration data generator 27 for rate smoothing restoration. The IDR picture image K is an image having redundancy in the time direction. The restoration data generation unit 27 for rate smoothing restoration uses the information amount smaller than that of the IDR picture while correlating the IDR picture image K with the decoded image L supplied from the image data decoding unit 37. The restoration switch image data M for restoring the decoded image not affected by the packet loss is generated and output to the stream switch 36.

  The stream switch 36 inserts the retransmission data G into the high-efficiency encoded image data N when the retransmission data G supplied from the retransmission data detection / sampling unit 24 arrives within a predetermined time before the timer expires, When the packet loss occurs when the retransmission data G does not reach within the expiration of the timer, the loss compensation image encoded data W supplied from the loss compensation image encoded data generation unit 35 is inserted. Further, the restoration switch image data M supplied from the rate smoothing restoration data generator 27 is inserted. In normal times when no packet loss occurs, the high-efficiency encoded image data N is passed through and output. The output of the stream switch 36 is supplied to the image data decoding unit 37 and the RTP packet unit 38 as rate smoothed image data X.

  The image data decoding unit 37 performs image decoding on the rate-smoothed image data X, generates the decoded macroblock number T to the decoded final MB number holding unit 32, and generates the decoded image L as restoration data for rate smoothing restoration. Output to the section. Further, similarly to the image data decoding unit 10 in the first embodiment, the decoded image L can be output to the outside and displayed. The RTP packet unit 38 performs RTP packetization on the rate-smoothed image data X and attaches it to the relay so as to output the output data Y to the next-stage image decoding apparatus.

  In this way, by separating the IP layer and the image decoding layer, it can be applied to various uses such as an image-dedicated stream relay device which is a device other than the image decoder. Further, by processing the influence of the IP packet loss in the IP layer, it is possible to construct a countermeasure means separated from the image decoding layer and simplify the circuit / device configuration.

  FIG. 5 shows a specific example of the stream switch 36. The stream switch 36 includes three delay adjustment buffer memories 41 to 43, a selector 44, and a data multiplexer 45. The delay adjustment buffer memory 41 stores the input restoration switch image data M, the delay adjustment buffer memory 42 stores the input high-efficiency encoded image data N, and the delay adjustment buffer memory 43 is input. Loss compensation image encoded data W is stored.

  The selector 44 inputs the retransmission data G and the read loss compensation image encoded data d3 read from the delay adjustment buffer memory 43, and outputs the retransmission data G as selector data d4 when the retransmission data G arrives. When 23 expires (when retransmission data G has not arrived), read loss compensation image encoded data d3 is output as selector data d4. The data multiplexer 45 reads the selector data d4 read as described above, the read restoration switch image data d1 read from the delay adjustment buffer memory 41 at the output timing of the IDR picture data J, and delay adjustment at other times. Read high-efficiency encoded image data d2 read from the buffer memory 42 is input, and one of these is output.

[Description of operation]
FIG. 6 is a timing chart for explaining the operation of the image decoding apparatus, and illustrates the relationship among the input data A, the output data Z, and the decoded image L. The input data A sequentially arrives at the RTP packet receiving unit 21 as R0 and R1 (time t0, t1 in FIG. 6), R2 is not reached due to packet loss, and then R3, R4, and R5 arrive (time t3) To t5), R6 is not reached due to packet loss, and R7, R8, R9, and R10 arrive (time t7 to t10).

  When R3 arrives after arrival at R1 (time t3), the packet loss of R2 is determined by the packet loss occurrence determination unit 30, so the retransmission request unit 31 sends the retransmission request signal R to the data source. . The timer 23 waits for retransmission data to be transmitted from the transmission source for a predetermined time. In the example of FIG. 6, since the retransmission data for R2 has arrived before the timer 23 expires (time t3), it is rearranged in the original order in the output data Z and output.

  In FIG. 6, the retransmission data is shown as the output data Z before the arrival of the retransmission data (time t2), and apparently contradictory, but the input data A, the output data Z, and the decoded data Of course, there is a delay with L. However, until drawing, the input data A, the output data Z, and the decoded data L are shown on the same time axis, and there is no contradiction. For output data Z and decoded data L, see the time axis sliced to the right.

  When R7 arrives (time t7) after arrival at R5 (time t5), the packet loss of R6 is determined by the packet loss occurrence determination unit 30, so that the retransmission request signal R is sent from the retransmission request unit 31 to the data transmission source. Send to The timer 23 waits for retransmission data to be transmitted from the transmission source for a predetermined time.

  In the example of FIG. 6, the retransmitted data for R6 does not reach when the timer 23 expires (time t8), and the loss-compensated image encoded data generation unit 35 uses the method described in detail in the first embodiment. Loss compensation image encoded data W is generated. The loss compensation image encoded data W is inserted into the output data Y at the original R6 transmission timing (time t6) as loss compensation data C6 in the stream switch 36. As a result, the continuity of the macroblock decoding of the processing in the image data decoding unit 10 is guaranteed, and a compensated image is generated in which the influence of the packet loss on the reproduced image is minimized. This effect is for the compensated image C6 on the subsequent R7 decoded image.

  Further, the loss compensation data C6 is decoded by the image data decoding unit 37 and supplied to the rate smoothing restoration data generation unit 27 for recovery as in the case of regular decoded images such as R0 to R5. However, since IDR picture detection / extraction unit 25 has not yet detected an IDR picture, rate smoothing type restoration data generator 27 for recovery does not function.

  When the timer 23 expires (time t8), it is detected that the retransmission data for R6 has not arrived, and at the same time, the refresh request C is sent from the expiration time detector 22 to the data transmission source. . Normal decoding is performed on R7 and R8 reaching time t7 and t8, but loss compensation data C6 affects the R7 decoded image, and the R7 decoded image affects the R8 decoded image.

  In response to the refresh request C, the input data R9 including the IDR picture arrives at time t9, and the IDR picture detection / sampling unit 25 detects the IDR picture and outputs the IDR picture data J. The IDR picture decoder 26 decodes the IDR picture data J and outputs an IDR picture image K.

  The rate-smoothing restoration data generator 27 for restoration is switched by decoding from the decoded image L decoded as the compensation image and the decoded IDR picture image K while reducing the redundancy in the time direction. A stream is generated as restoration switch image data M. The switch image data M for restoration, which is this switching stream, has a smaller amount of information than the original IDR picture encoded information because the redundancy in the time direction is reduced, and is rate smoothed when performing image transmission relay. Can be sent out.

The restoration switch image data M is inserted into the output data Z via the RTP packet unit 38 at the original R9 (IDR) transmission timing (time t10) and output. In FIG. 6, it is shown as SP9. When the restoration switch image data M is decoded by the image data decoding unit 37, the decoded image SP9 becomes equal to the R9 (IDR) decoded image that is not affected by R8. As a result, refresh is performed to make the contents of the reference frame memories of the encoder and decoder the same. Therefore, it is possible for the next R10 to normally decode the difference encoded information from the R9 (IDR) decoded image. The stream generation method for switching from the decoded image L to the reference image from the IDR picture image K as described above is known as, for example, an ITU-T recommendation H.264 SP picture.
[Industrial applicability]
The following examples can be listed as examples of utilization of the present invention.
・ Image transmission service that is performed through a poor network environment that does not guarantee QoS (Quality of Service) where packet loss occurs, and that requires long-term stability and service continuity. Image relay services such as broadcast on the premises for images that have been communicated

DESCRIPTION OF SYMBOLS 1 RTP packet receiving part 2 Image data extraction part 3 Sequence number extraction part 4 Packet loss generation | occurrence | production determination part 5 Decoded final MB number holding part 6 Restoration start MB number detection part 7 Previous packet loss detection part 8 Image coding for loss compensation Data generation part 9 Data multiple insertion part
10 Image data decoder
21 RTP packet receiver
22 Expiration time detector
23 timer
24 Retransmission data detection sampling section
25 IDR picture detection sampling section
26 IDR picture decoder
27 Repair data generator for rate smoothing recovery
28 Image data extraction unit
29 Sequence number extractor
30 Packet loss occurrence determination unit
31 Retransmission request part
32 Decrypted final MB number holding part
33 Previous packet loss detector
34 First MB number detector after recovery
35 Loss compensation image encoded data generator
36 Stream switcher
37 Image data decoder
38 RTP packet part
41 Delay adjustment buffer memory
42 Delay adjustment buffer memory
43 Delay adjustment buffer memory
44 selector
45 Data multiplexer

Claims (12)

  It is obtained by analyzing data before and after the IP packet loss which macroblock part of the image area of the high-efficiency encoded image data is lost due to IP packet loss, and the bit stream of the obtained missing part is encoded. Generated by a method compliant with the syntax rule of the above, inserted into the high-efficiency encoded image data, and guarantees the continuity of macroblock decoding in the processing in the image data decoding unit, and the influence of the packet loss on the reproduced image An image decoding apparatus that generates a compensated image in which is minimized. In an image decoding apparatus that recovers and decodes packet loss of highly efficient encoded image data in RTP data input from an IP network,
An image data extraction unit for extracting the highly efficient encoded image data from the RTP data;
A post-recovery first MB number detection unit that detects the number of the first macroblock immediately after the recovery included in the high-efficiency encoded image data;
A missing portion bit stream obtained from the number of the first macroblock immediately after the restoration and the number of the final decoded macroblock is generated as loss-compensated image encoded data in a method based on the syntax rules of the encoding method. A loss compensation image encoded data generation unit;
A data multiplex insertion unit for inserting the loss-compensated image encoded data into the high-efficiency encoded image data;
An image data decoding unit that performs image decoding on the loss-compensated image encoded data supplied from the data multiplex insertion unit, outputs an output image to the outside, and outputs a decoded macroblock number; An image decoding apparatus characterized by the above. A sequence number extraction unit for extracting a sequence number of the RTP data;
A packet loss occurrence determination unit that determines the presence or absence of the packet loss by verifying the continuity of the sequence number;
If the result of the determination is loss, a decoded final MB number holding unit that outputs the decoded macroblock number at that time as the decoded final macroblock number;
If the result of the determination is lossless and the result of the determination of the previous packet is lossy, it has a previous packet loss detection unit that outputs a packet loss recovery detection notification to the first MB number detection unit after recovery,
The post-restoration leading MB number detection unit sets the macroblock number included in the high-efficiency encoded image data when receiving the packet loss recovery detection notification as the number of the leading macroblock immediately after the restoration. The image decoding apparatus according to claim 2. IDR picture detection that detects IDR picture data capable of instantaneous decoding refresh operation from the RTP data, and sends other RTP data including highly efficient encoded image data to the image data extraction unit and the sequence number extraction unit Sampling section;
An IDR picture decoder that performs a decoding operation on the IDR picture data and outputs an IDR picture image;
With respect to the IDR picture image, a correlation with the decoded image supplied from the image data decoding unit is performed, and the packet loss compensated image is restored to a decoded image that is not affected by the packet loss with a smaller amount of information than the IDR picture. A rate-smoothing restoration data generator for restoration for generating restoration switch image data for
A stream switch that inserts the restoration switch image data into the high-efficiency encoded image data and outputs it as rate-smoothed image data;
Image data that performs image decoding on the rate-smoothed image data, and outputs the decoded macroblock number to the decoded final MB number holding unit and the decoded image to the rate-smoothing restoration data generator for restoration A decryption unit;
4. The image decoding according to claim 3, further comprising an RTP packet unit that relays the rate-smoothed image data as an object and outputs the output data to a subsequent image decoding apparatus. apparatus. Based on the packet loss determination in the packet loss occurrence determination unit, in order to request retransmission of lost packets, a retransmission request unit that outputs a retransmission request signal to the source of the RTP data,
From the RTP data, to detect and take out the retransmitted data, RTP data information to be output to the IDR picture detection sampling unit as RTP data detection sampling unit,
A timer that starts a timer operation by setting a predetermined expiration time by the notification of the retransmission request, and stops the timer operation based on the notification of the retransmission data detection; and
5. The image according to claim 4, further comprising an expiration time detection unit that detects an expiration time of the timer based on timer information supplied from the timer and sends a refresh request to a transmission source of the RTP data. Decoding device.   The stream switch inserts the retransmission data when the retransmission data supplied from the retransmission data detection and extraction unit reaches the high-efficiency encoded image data within a predetermined time before the timer expires, and the timer expires. 6. The image decoding apparatus according to claim 5, wherein, when a packet loss that does not reach retransmission data occurs, the loss compensation image encoded data supplied from the loss compensation image encoded data generation unit is inserted.   It is obtained by analyzing data before and after the IP packet loss which macroblock part of the image area of the high-efficiency encoded image data is lost due to IP packet loss, and the bit stream of the obtained missing part is encoded. Generated by a method compliant with the syntax rule of the above, inserted into the high-efficiency encoded image data, and guarantees the continuity of macroblock decoding in the processing in the image data decoding unit, the influence of the packet loss on the reproduced image A method for compensating for packet loss, comprising generating a compensated image in which is minimized. A packet loss compensation method in an image decoding apparatus that recovers and decodes packet loss of highly efficient encoded image data in RTP data input from an IP network,
A procedure for extracting the highly efficient encoded image data from the RTP data;
A procedure for detecting the number of the first macroblock immediately after restoration included in the high-efficiency encoded image data;
A missing portion bit stream obtained from the number of the first macroblock immediately after the restoration and the number of the final decoded macroblock is generated as loss-compensated image encoded data in a method based on the syntax rules of the encoding method. Procedure and
Inserting the loss compensation image encoded data into the high efficiency encoded image data;
Image decoding is performed on the loss-compensated image encoded data into which the loss-compensated image encoded data is inserted, and an output image is output to the outside and a number of the decoded macroblock is output. A characteristic packet loss compensation method. A procedure for extracting the sequence number of the RTP data;
A procedure for determining the presence or absence of the packet loss by verifying the continuity of the sequence number;
If the result of the determination is lossy, a procedure for outputting the decoded macroblock number at that time as the decoded final macroblock number;
A procedure for outputting a packet loss recovery detection notification if the result of the determination is lossless and the result of the determination of a previous packet is loss;
9. The method according to claim 8, further comprising a step of setting a macroblock number included in the high-efficiency encoded image data at the time of receiving the packet loss recovery detection notification as the number of the first macroblock immediately after the recovery. Packet loss compensation method. A procedure for detecting IDR picture data capable of instantaneous decoding refresh operation from the RTP data and sending RTP data including other high-efficiency encoded image data;
A procedure for performing a decoding operation on the IDR picture data and outputting an IDR picture image;
With respect to the IDR picture image, a restoration switch image for recovering the decoded image from the packet loss compensated image without being affected by the packet loss with a smaller amount of information than the IDR picture while correlating with the decoded image. The steps to generate data,
Inserting the restoration switch image data into the high-efficiency encoded image data and outputting it as rate-smoothed image data;
Performing image decoding on the rate-smoothed image data, and outputting a decoded macroblock number and a decoded image;
10. The packet loss compensation method according to claim 9, further comprising a step of performing relaying so as to perform RTP packetization on the rate-smoothed image data and output the output data to a subsequent image decoding apparatus. . Based on the packet loss determination, a procedure for outputting a retransmission request signal to the transmission source of the RTP data in order to request retransmission of a lost packet;
From the RTP data, to detect and take out the retransmitted data, to output as RTP data information,
A procedure for starting a timer operation by setting a predetermined expiration time by notification of the retransmission request and stopping the timer operation based on the notification of retransmission data detection;
The packet loss compensation method according to claim 10, further comprising a step of detecting an expiration time of the timer based on timer information supplied from the timer and transmitting a refresh request to a transmission source of the RTP data. .   For the high-efficiency encoded image data, the retransmission data is inserted when the retransmission data arrives within a predetermined time before the timer expires, and when the packet loss occurs when the retransmission data does not arrive within the timer expiration, the loss compensation 12. The packet loss compensation method according to claim 11, wherein image encoded data is inserted.

Images