JP2008283253A - Image transmission system, image encoding device, and image decoding device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image transmission system that easily and speedily determines a frame suitable for error concealment and perform error concealment processing even if a transmission error occurs, and also to provide an image encoding device, and an image decoding device. <P>SOLUTION: An error concealment control unit 24 gives a mark indicating that an image frame generated as a result of decoding of an image packet before a non-image packet showing no correlativity between image frames can not be used as an image frame for error concealment, to the image frame (marking processing), based upon control information that a control information decoding unit 23 reports. An error concealment unit 25 determines an image frame having been encoded and to be used for error concealment among image frames to which the mark is not given. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image transmission system, and more particularly to an image transmission system, an image encoding device, and an image decoding device capable of obtaining a reproduced image with little image quality degradation even in an environment where a transmission error occurs.

  In recent years, image data is handled in various digital devices, but in the coding of image (moving image) data used in videophone systems, digital broadcast transmitters, etc., motion compensation prediction, orthogonal transform (discrete transformation) Coding schemes in which a plurality of techniques such as cosine transform and the like and entropy (variable length) coding are combined are used.

  When the image data encoded by the above-described encoding method is transmitted to the image decoding apparatus via the wireless transmission path, the encoded image data can be normally decoded due to the influence of errors mixed in the wireless transmission path. It may disappear.

  In particular, in the case of using the variable-length coding method, once an error is mixed, the synchronization is not recovered until the next synchronization code (for example, a start code indicating the beginning of the frame) is received, and the image is lost or the frame is lost. The image quality of the decoded image is extremely deteriorated due to a decrease in rate or the like.

  In order to prevent such image quality degradation, in general, in an image decoding apparatus, an error block concealment process is performed by copying a block at the same position as the error block in a decoded frame to an error block in which an error has occurred. (Error concealment processing) is performed.

  When executing the error concealment process, a frame having a correlation (correlation degree) with a frame including the error block is used as an error concealment frame. This is because if a frame that is not at all similar to the target frame is used as an error concealment frame, the image quality further deteriorates despite the execution of the error concealment process.

  However, in a mobile terminal decoding device (receiving side) having a low processing capability, when executing a determination process of a frame used as an error concealment frame, in order to save resources of the mobile terminal, such processing is simplified, The amount of processing needs to be reduced.

  Therefore, the image encoding device (transmission side) in the image control system disclosed in Patent Document 1 is encoded including an identification flag for identifying whether the frame is an I picture or a P picture. An encoded stream composed of a frame sequence is transmitted to an image decoding apparatus (reception side), and the image decoding apparatus refers to an identification flag included in the received encoded stream, and uses a frame (picture) used for error concealment Is determined.

  For example, when an error block is included in a P picture, the image decoding apparatus considers that the P picture including the error block and the P picture immediately before the P picture have a correlation, and the error block is Error concealment processing is executed using the P picture immediately before the included P picture as an error concealment frame.

An I picture is a picture that is encoded independently of other frames and used as a reference frame for a P picture.
The P picture is a forward predictive coding screen, and refers to a picture that performs predictive coding with reference to an I picture or P picture located in the past in time.
JP 2002-77922 A

  However, for example, when an I picture that is transmitted / received in a videophone system exists only at the beginning, and thereafter, for example, a scene change occurs in a certain P picture in an encoded stream composed only of P pictures. When the correlation between the P pictures is lost, it cannot be said that there is a correlation between the P picture in which the scene change has occurred and the P picture immediately before the P picture.

  Then, when an error block is included in the P picture itself in which a scene change has occurred, the P picture immediately before the P picture including the error block (which has no correlation with the P picture including the error block) is used for error concealment. When error concealment processing is executed using a frame, as described above, there is a problem that the image quality further deteriorates.

  The present invention has been made in view of such circumstances, and even when a transmission error occurs, an image transmission system capable of easily and quickly determining an optimal frame for error concealment and executing error concealment processing, An object is to provide an image encoding device and an image decoding device.

  The first technical means is an image transmission system comprising: an image encoding device that encodes an image frame sequence to generate an encoded stream; and an image decoding device that receives and decodes the transmitted encoded stream. The image encoding device detects the correlation between the image of the image frame to be encoded and the image of the immediately preceding image frame, and when there is no correlation, at least the control information indicating that there is no correlation is encoded. Inserting an image packet to generate the encoded stream, the image decoding device, the image frame used for error concealment processing of the image frame when an error is detected in the image frame of the received encoded stream, An image transmission system is characterized in that it is determined based on whether or not a non-image packet is detected.

  The second technical means detects the correlation between the image of the image frame to be encoded and the image of the immediately preceding image frame in an image encoding device that encodes an image frame sequence to generate an encoded stream. In the case where there is no image code in the image transmission system of the first technical means, the encoded stream is generated by inserting a non-image packet in which control information indicating that there is no correlation is inserted. Device.

  When a non-image packet is detected in the decoding process for receiving the encoded stream transmitted and generating an image frame, the third technical means is generated as a result of decoding the image packet before the non-image packet. An image decoding apparatus in an image transmission system according to a first technical means, wherein an image frame is not used as an image frame used for error concealment processing of an image frame in which an error is detected.

According to the present invention, in an image decoding device such as a mobile terminal that receives a moving image, an optimal frame for error concealment can be determined easily and quickly, and further, the amount of error concealment processing can be reduced. it can.
Furthermore, since error concealment can be appropriately performed regardless of the picture type, it is possible to prevent image quality deterioration due to transmission errors and the like.

Example 1
FIG. 1 is a diagram illustrating an example of an image frame sequence to be encoded.
A0 to A4, B5 to B9, and A10 to A12 are image frames.
The image frames A0 to A4 have a high correlation between the images between the image frames (hereinafter simply referred to as “image frames”). There is a correlation between them.

  In other words, the image frame A4 and the image frame B5, and the image frame B9 and the image frame A10 have no correlation between the image frames, and the image frame B5 and the image frame A10 as indicated by the arrows CH1 and CH2, For example, assume that a scene change has occurred.

  The image encoding apparatus encodes the image frame sequence shown in FIG. 1 to generate the encoded stream shown in FIG. The encoding method is, for example, H.264. H.264 (MPEG-4) can be applied.

  In FIG. 2, SPS (Sequence Parameter Set) is a packet in which information related to encoding of the entire sequence (image frame sequence) is encoded.

  The PPS (Picture Parameter Set) is a packet in which information indicating the coding mode of the entire image frame (picture) is coded.

  Note that SPS and PPS are non-image packets in which information about image frame encoding is encoded.

SL0 to SL4 are image packets that are encoded data of the image frames A0 to A4 in FIG.
Similarly, SL5 to SL9 and SL10 to SL12 are image packets that are encoded data of the image frames B5 to B9 and the image frames A10 to A12 in FIG.

  X and Y are non-image packets encoded with control information indicating that there is at least no correlation between image frames due to a scene change or the like.

  The image decoding apparatus according to the present invention to be described later refers to the non-image packet X, so that a scene change occurs in the image frame B5 described in FIG. 1, and the image frame is between the image frame B5 and the immediately preceding image frame A4. It can be detected that there is no correlation. Similarly, it can be detected from the non-image packet Y that there is no correlation between the image frames B9 and A10.

  In FIG. 2, one image frame is composed of one image packet, but usually one image frame is composed of a plurality of image packets.

FIG. 3 is a block diagram of the image encoding device 1 that generates an encoded stream including non-image packets in which control information indicating that there is no correlation between image packets and at least image frames is encoded. .
Reference numeral 11 denotes an image input unit. For example, the image input unit 11 takes in the image data from the storage unit 15 in which the image data is stored, and outputs the image data to the image encoding unit 12.

  The image encoding unit 12 compresses and encodes the image data output from the image input unit 11 by a predetermined compression method (for example, H.264, MPEG-4, etc.), and outputs the image data to the multiplexing unit 14. Simultaneously with the compression encoding, the image data (image frame) is output to the frame memory 16 as a reference frame for encoding the next image frame.

  Further, the image encoding unit 12 includes an inter-frame correlation detection unit 12a, and when encoding an image frame, an image frame being encoded and an image frame immediately before the image frame being encoded And when the correlation is lost due to a scene change or the like, for example, this is detected, and the detection result is output to the control information encoding unit 13.

  Note that various conventionally proposed methods can be employed as a method for detecting the correlation between image frames.

  For example, it is assumed that an image frame that includes a predetermined number or more of intra macroblocks that are encoded using only image information in the image frame is an image frame that has no correlation with the immediately preceding image frame of the image frame. For example, it is possible to detect that there is no correlation between image frames by calculating the number of intra macroblocks and comparing it with a predetermined threshold.

  Conversely, an image frame including a predetermined number or less of inter-macro blocks that are encoded using image information between image frames is assumed to have the above-described relationship if there is no correlation with the immediately preceding image frame of the image frame. Similarly, it can be detected that there is no correlation between image frames.

When the detection result indicating that there is no correlation between image frames output from the image frame correlation detection unit 12a is input, the control information encoding unit 13 displays control information indicating that there is no correlation between image frames. An encoded non-image packet is generated. Further, when there is control information necessary for decoding the encoded stream, a non-image packet in which the control information is encoded is generated.
Then, these non-image packets are output to the multiplexing unit 14, and the multiplexing unit 14 multiplexes the image data (image packet) encoded by the image encoding unit 12 and the non-image packet.

  In the example of FIG. 1, for example, when it is detected that the correlation between the frame A4 and the image frame B5 is lost due to a scene change occurring in the image frame B5, there is no correlation between the image frames. The non-image packet in which the control information is encoded is output to the multiplexing unit 14.

  Note that as a non-image packet (control information), H.264 is used. In the case of H.264 (MPEG-4), NAL units such as SPS, PPS, AUD (Access Unit Delimiter), and SEI (Supplemental Enhancement Information) are defined, but control information indicating that there is no correlation between image frames And the encoded information can be included in these non-image packets. Moreover, a predetermined character string (for example, SCN_CHNG) can be used as the control information.

This may be a uniquely defined NAL unit that is distinguishable from a predefined NAL unit.
In the above example, when it is detected that there is no correlation between image frames, one non-image packet is output to the multiplexing unit 14, but a predetermined number of non-image packets may be output continuously.

For example, as illustrated in FIG. 2, the multiplexing unit 14 includes the image packet and the non-image so that the non-image packet is inserted between the image packet SL4 of the image frame A4 and the image packet SL5 of the image frame B5. Multiplex packets.
The multiplexed image packet and non-image packet are transmitted as an encoded stream to an image decoding apparatus to be described later via a wireless network or the like.

The storage unit 15 includes an HDD (Hard Disk Drive) or the like, and stores image data such as still images, moving images, and computer graphics (CG), for example.
In addition to capturing image data from the storage unit 15, digital data in an image format such as RGB or YCbCr output by an imaging unit composed of a CCD camera / CMOS camera or the like may be captured. In this case, the image encoding unit 12 automatically selects a non-image packet when the image type (video stored in the HDD, video captured by a camera, etc.) input from the image input unit 11 is switched. You may make it insert.

Next, the encoded stream creation processing of the image encoding device 1 will be described with reference to the flowchart of FIG.
The image input unit 11 takes in the image data (image frame) from the storage unit 15 and outputs it to the image encoding unit 12 (step S1).

  The image encoding unit 12 generates an image packet obtained by encoding the input image data (step S2), and in step S3, the inter-frame correlation detection unit 12a of the image encoding unit 12 performs encoding. It is determined whether or not there is a correlation between the middle image frame and the image frame immediately before the image frame being encoded.

  When there is a correlation between image frames (step S3 / Yes), the image encoding unit 12 outputs the image packet generated in step S2 as an encoded stream (step S4).

  If there is no correlation (step S3 / No), a non-image packet in which control information indicating that there is no correlation between image frames is encoded is generated (step S5).

  Next, the multiplexing unit 14 multiplexes the image packet generated in step S2 and the non-image packet generated in step S5, and outputs the result as an encoded stream (step S6).

  If there is control information necessary for decoding the image packet generated in step S2, a non-image packet obtained by encoding the control information is multiplexed with the image packet and output as an encoded stream.

Finally, the presence / absence of image data to be encoded is determined (step S7).
When there is image data to be encoded (step S7 / YES), the processing after step S1 is executed.
If there is no image data to be encoded (step S7 / NO), the process ends.

(Example 2)
Next, an error concealment image frame determination process at the time of error block detection, which is executed in an image decoding apparatus described later, will be described.

FIG. 5 is a diagram for explaining an error concealment image frame determination process when an error block is detected.
Reference numeral 31 denotes an image frame developed in a frame memory to be described later when decoding of the image packet SL4 is completed.
When decoding of the image packet SL4 is completed, as shown in the figure, the frame memory stores image frames A2 to A4 as reference images or display-waiting images necessary for inter-frame prediction.

Reference numeral 32 denotes an image frame developed in a frame memory to be described later when decoding of the image packet SL5 is completed.
When the decoding of the image packet SL5 is completed, as shown in the figure, image frames A3, A4, and B5 are stored in the frame memory.

X is a non-image packet in which control information indicating that there is no correlation between the images of the image frame A4 and the image frame B5 is encoded. After the decoding of the image packet SL4 is completed, the control information decoding unit described later Is done.
Then, the image decoding apparatus determines (selects) an image frame to be used as an error concealment image frame from the image frame developed in the frame memory based on the decoded control information.

  In this way, image packets are sequentially developed and decoded in the frame memory.

Next, the determination process will be described in detail.
When the image decoding apparatus receives the encoded stream shown in FIG. 2, the received encoded stream is separated into image packets and non-image packets, and each packet is decoded.

As shown in FIG. 2, after decoding of the image packet SL4, when the non-image packet X encoded with control information indicating that there is no correlation between image frames following the image packet SL4 is decoded, Mark indicating that the image frame cannot be used as an error concealment frame for the image frame A4 generated as a result of decoding the image packet SL4 located before the non-image packet X in the encrypted stream. (M) is added (marking process is performed).
Note that not only the image frame A4 but also the image frame A3 can be marked.

Next, when the decoding process of the image packet SL5 (image frame B5) is completed, the decoding process of the image packet SL6 (image frame B6) is executed.
If an error block as indicated by the symbol ERR is detected during the decoding process of the image packet SL6, it is correlated with an image frame that has not been subjected to the marking process (not marked with the symbol M), that is, the image frame B6. The error concealment process is executed using the image frame B5 having the error concealment as the error concealment image frame.

  When an error block is detected when decoding the image packet SL5 (image frame B5), the image frame A4 is not used as an error concealment image frame because it has no correlation with the image frame B5. The error concealment process is executed with reference to FIG.

  As described above, since it is possible to determine an optimal error concealment image frame without performing complicated processing only by referring to control information indicating that there is no correlation, it is possible to reduce the processing burden related to error concealment. .

FIG. 6 is a block diagram of an image decoding apparatus that can execute the decoding process described above.
The image decoding device 2 receives and decodes the encoded stream transmitted by the image encoding device 1.

  Reference numeral 21 denotes a separation unit. The separation unit 21 separates the input encoded stream into image packets and non-image packets, outputs the separated image packets to the image decoding unit 22, and outputs the non-image packets to the control information decoding unit 23. To do.

  The image decoding unit 22 decodes the input image packet and outputs the decoded image to the frame memory 26. If an error block is detected, the error concealing unit 25 is notified of the error block position information and the like.

  The control information decoding unit 23 encodes control information indicating that there is no correlation between image frames and / or control information used for image packet decoding processing, for example, SPS, PPS, Decode non-image packets such as AUD and SEI.

When the control information indicating that there is no correlation between the image frames is decoded, the control information is notified to the error concealment control unit 24.
Further, when the control information used for the decoding process of the image packet is decoded, the control information is notified to the image decoding unit 22.

The error concealment control unit 24 generates an image frame generated as a result of decoding an image packet before a non-image packet indicating that there is no correlation between image frames based on the control information notified by the control information decoding unit 23. On the other hand, a mark indicating that the image frame cannot be used as an error concealment image frame is attached (marking process).
Specifically, the marking process (symbol M) is performed on the image frame A4 generated as a result of decoding the image packet SL4 before the non-image packet X described in FIG.

  The error concealment unit 25 refers to the position information of the error block notified from the image decoding unit 22 and the mark or the like applied to the image frame by the error concealment control unit 24. The decoded image frame used for error concealment is determined from the inside. Then, for example, by copying a block at the same position as the block in the determined error concealment image frame to the error block, the error concealment process is executed, and the frame subjected to the error concealment process is output to the frame memory 26. .

  In this case, usually, the immediately preceding image frame is used for error concealment as having a correlation, but when the mark (symbol M) is attached as described above, it is an image frame having no correlation, The image frame is not used for error concealment.

Next, the image decoding process of the image decoding apparatus 2 will be described with reference to the flowchart of FIG.
The encoded stream transmitted by the image encoding device 1 is received (step S11), and the separation unit 21 separates the received encoded stream into image packets and non-image packets. In step S12, the separated packets are separated. It is determined whether the packet is an image packet or a non-image packet.

  When the separated packet is an image packet (step S12 / image packet), the packet is output to the image decoding unit 22, and the decoding unit 22 executes a decoding process of the image packet (step S13).

  When the separated packet is a non-image packet (step S12 / non-image packet), the non-image packet is output to the control information decoding unit 23, and the decoding unit 23 executes a decoding process of the non-image packet (step S14). .

  Next, in step S15, the non-image packet is decoded and the presence / absence of control information indicating that there is no correlation between image frames is determined.

  When there is control information indicating that there is no correlation between image frames (step S15 / Yes), the control information decoding unit 23 notifies the error concealment control unit 24 of the information, and the control unit 24 Based on the above, a marking process is performed on a decoded image frame that cannot be used as an error concealment image frame (step S16). Thereafter, in the error concealment process in step S18 described later, the image frame subjected to the marking process is not used as an error concealment image frame.

  When there is no control information indicating that there is no correlation between the image frames (step S15 / No), the decoded control information is control information necessary for decoding the image packet, and thus the control information decoding unit 23 Notifies the image decoding unit 22 of the decoded control information. Thereafter, the control information is used in the decoding process of the image packet in step S13.

  If there is control information indicating that there is no correlation between image frames and control information used for decoding the image packet, the marking process (step S16) and the decoding of the image packet are included. Necessary control information is notified to the image decoding unit 22.

  When the image decoding unit 22 detects an error block during the execution of the image packet decoding process in step S13 (step S17 / YES), the error concealment unit 25 detects the error block position information and the error concealment control unit 24. The error concealment image frame is determined from the image frames not subjected to the mark by referring to the mark etc. given in step S18, and the error concealment process is executed (step S18).

  Next, the image frame subjected to the error concealment process is output to a monitor (not shown) or the like (step S19).

  If the image decoding unit 22 does not detect an error block during the image packet decoding process in step 13 (step S17 / NO), the decoded image frame is output to a monitor (not shown) or the like (step S19).

After the process of step S19 is completed, it is determined whether the reception process of the decoded stream is completed (step S20). If the reception process is not completed (step S20 / NO), the process of step 11 is executed.
When the reception process is finished (step S20 / YES), the decoding process is finished.

  As described above, the image decoding apparatus 2 performs control indicating that there is no correlation between image frames included in the encoded stream even when the correlation between the image frames is lost due to a scene change or the like. By referring to the information, it is possible to appropriately determine an error concealment image frame without performing complicated processing.

Also, regardless of the frame type (picture type) of each image frame, that is, whether it is an intra frame composed only of intra-coded blocks or an inter-frame including inter-coded blocks, an error occurs. Concealment processing can be executed.
For example, the present invention can also be applied to an encoded stream transmission / reception system such as a videophone system that transmits / receives an encoded stream composed of an intra frame only in the first frame and only an inter frame.

H. In the H.264 encoding method, encoding and decoding of SEI (additional information for decoding encoded data such as timing information of each frame and information for random access) is not essential. May not be supported.
However, in the present invention, since a NAL unit such as SPS or PPS can be used as a non-image packet in which the control information is encoded, the present invention can also be applied to an encoded stream transmission / reception system that does not support SEI. .

It is the figure which showed an example of the image frame sequence encoded. It is an example of an encoding stream. 1 is a block diagram of an image encoding device according to the present invention. It is a flowchart for demonstrating an encoding stream production | generation process. It is a figure for demonstrating the determination method of the image frame for error concealment. It is a block diagram of the image decoding apparatus which concerns on this invention. It is a flowchart for demonstrating an image decoding process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Image coding apparatus, 2 ... Image decoding apparatus, 11 ... Image input part, 12 ... Image coding part, 12a ... Inter-frame correlation detection part, 13 ... Control information coding part, 14 ... Multiplexing part, DESCRIPTION OF SYMBOLS 15 ... Accumulation part, 16 ... Frame memory, 21 ... Separation part, 22 ... Image decoding part, 23 ... Control information decoding part, 24 ... Error concealment control part, 25 ... Error concealment part, 26 ... Frame memory

Claims (3)

In an image transmission system comprising an image encoding device that encodes an image frame sequence to generate an encoded stream, and an image decoding device that receives and decodes the transmitted encoded stream,
The image encoding device detects the correlation between the image of the image frame to be encoded and the image of the immediately preceding image frame, and if there is no correlation, encodes control information indicating that there is no correlation Inserting the non-image packet to generate the encoded stream;
The image decoding apparatus determines an image frame to be used for error concealment processing of an image frame when an error is detected in an image frame of a received encoded stream depending on whether or not the non-image packet is detected. Image transmission system. In an image encoding device that generates an encoded stream by encoding an image frame sequence,
The correlation between the image of the image frame to be encoded and the image of the immediately preceding image frame is detected, and when there is no correlation, a non-image packet in which control information indicating that there is no correlation is encoded is inserted. The image encoding apparatus according to claim 1, wherein the encoded stream is generated. In the decoding process of receiving the transmitted encoded stream and generating an image frame,
When the non-image packet is detected, the image frame generated as a result of decoding the image packet before the non-image packet is not used as an image frame used for error concealment processing of the image frame in which an error is detected. The image decoding apparatus in the image transmission system according to claim 1.

Images