WO2019134255A1

WO2019134255A1 - Network transmission characteristic friendly real-time image adaptive recovery system and method

Info

Publication number: WO2019134255A1
Application number: PCT/CN2018/078197
Authority: WO
Inventors: 张联昌; 毕永建; 孔令波
Original assignee: 厦门亿联网络技术股份有限公司
Priority date: 2018-01-03
Filing date: 2018-03-07
Publication date: 2019-07-11
Also published as: CN108235150A

Abstract

Disclosed in the invention are a network transmission characteristic friendly real-time image adaptive recovery system and a method. The system comprises two parts: a coding end and a decoding end. The coding end comprises a coding image index generation module, a coding image macroblock information backup module, a packet loss data propagation area calculation module and a packet loss area coding type control module. The decoding end comprises a decoding image index extraction module, a decoding packet loss information statistics module and a decoding packet loss information feedback module. According to the system and the method, the problem that IDR frames are applied for frequently in a packet loss environment is solved, and the image quality and network transmission effect of a real-time coding/decoding system are improved to a certain degree.

Description

Real-time image adaptive recovery system and method with friendly network transmission characteristics

技术领域Technical field

The present invention relates to the field of real-time encoded image transmission technologies, and in particular, to a real-time image adaptive recovery system and method with network transmission characteristics friendly.

背景技术Background technique

In the application of real-time video codec system (such as video conferencing or video surveillance), network packet loss is a problem that must be faced and solved. Take AVC (H.264) multi-band coding as an example, in AVC (H .264) There is a multi-slice coding mode in the coding, and there is no correlation (or little correlation) between the bands, and in the actual network transmission, it is generally transmitted in the form of one strip and one packet. In the packet loss environment, this method can reduce the spread of abnormal images, but it will inevitably lead to the 'mosaic' phenomenon (because the actual data has been lost, the missing part of the image can only be recovered by some predictive means, with real There is a certain error in the decoded image, and due to the temporal correlation between the encoded frames, the error image will continue to spread, causing a mosaic phenomenon).

Therefore, in order to eliminate image quality loss caused by network packet loss, the usual solution is: after detecting the packet loss, the decoding end sends a message request IDR frame to the encoding end device (immediately refreshing the frame), thereby blocking the abnormality caused by the packet loss. For image delivery, the encoding end needs to apply for an IDR frame to truly block the spread of the abnormal image, and then the application for the IDR frame occurs. The more serious the network packet loss, the more frequently the IDR frame is applied. However, the IDR frame has the following disadvantages: 1) The size of the IDR frame is much larger than the size of the P frame (predictive coded frame) in the case of the same quality, and 2) the image quality of the IDR frame is much worse than that of the P frame under the same size condition. If the IDR frame is frequently applied, the quality of the encoded image will be degraded, and it is not conducive to network transmission. In extreme cases, it will cause network congestion, which will cause more serious packet loss.

Therefore, the inventors have proposed a real-time image adaptive recovery system and method with network transmission characteristics to solve the problem of frequently applying IDR frames in a packet loss environment.

发明内容Summary of the invention

The object of the present invention is to provide a real-time image adaptive recovery system and method with friendly network transmission characteristics, so as to solve the problem of frequently applying IDR frames in a packet loss environment, and have certain image quality and network transmission effects of the real-time codec system. Upgrade.

In order to achieve the above object, the technical solution of the present invention is as follows:

A real-time image adaptive recovery system with friendly network transmission characteristics includes two parts: an encoding end and a decoding end, wherein the encoding end includes:

a coded image index generating module, the module generates a unique image index of the coded sequence, and is used by the encoding end to determine a time position corresponding relationship between the lost image frame and the current frame to be encoded fed back by the encoding end;

a coded image macroblock information backup module, the module backing up macroblock type information of the coded image and motion vector information, for calculating a packet loss data propagation path and a region;

a packet loss data propagation area calculation module, configured to calculate a propagation area of the lost packet data according to the correlation between the inter-frame information, including macroblock type information and motion vector information;

The packet loss type coding type control module is configured to control the coded macroblock type of the coding end. If it is determined that the macroblock to be coded is within the range of the packet loss data propagation area, the intra macroblock coding is performed to block the abnormal propagation. Otherwise, the normal encoding process is performed;

The decoding end includes:

Decoding an image index extraction module, the module extracts an index of the currently decoded frame image, and if a packet loss occurs, the information needs to be transmitted to the encoding end, and the encoding end uses the information to determine the decoded frame between the lost packet and the frame to be encoded. Time correlation

Decoding a packet loss information statistics module, where the module counts the location of the packet loss data and the coverage of the packet loss data, including a stripe level and a macroblock level, which are used by the encoding end to check the type of information;

Decoding the packet loss information feedback module, the module transmits the packet loss packet loss statistics and the packet loss image index information to the encoding end through the network protocol for abnormal image recovery.

A real-time image adaptive recovery method with network transmission characteristics, including:

Preserving a frame image index and encoding macroblock type information and motion vector information in the encoding end;

The encoding end calculates a macroblock correspondence relationship between the decoded frame and the current frame to be encoded according to the packet loss location information fed back by the decoding end; the packet loss location information includes a decoded frame index and a stripe index;

Then, the encoding end encodes the intra macroblock mode in the area where the association exists, and blocks the propagation path of the decoded abnormal image;

Then, the encoding end sends the encoded frame code stream, and can immediately return to the normal image after receiving the encoded frame data at the decoding end.

After adopting the above scheme, the present invention has the following advantages:

(1) The present invention can accurately control an abnormal image propagation path and an area caused by packet loss;

(2) The present invention performs precise control of the encoded macroblock type according to the abnormal image propagation range, and blocks abnormal image propagation;

(3) The present invention can recover the abnormal image caused by the packet loss without applying for the IDR frame, and the image restoration effect is the same as the application IDR mode, and the image abnormality caused by the data packet loss can be completely recovered, and the network bandwidth occupation is more The IDR method is low.

The invention will be further described below in conjunction with the drawings and specific embodiments.

附图说明DRAWINGS

1 is a schematic diagram of a macroblock dependency relationship between an encoding end and a decoding end;

2 is a flow chart of information transmission and processing of the encoding end and the decoding end of the present invention;

3 is a structural block diagram of a real-time image adaptive recovery system with network transmission characteristics friendly according to the present invention;

4 is a flow chart of processing of the encoding end of the present invention;

Figure 5 is a flow chart of the decoding process of the present invention.

具体实施方式Detailed ways

Before describing the detailed solution of the present invention, the codec principle involved in the present invention is explained:

As shown in FIG. 1 , the multi-band coding of AVC (H.264) is taken as an example, but the application range is not limited to AVC, and the macroblock dependency relationship between the encoding end and the decoding end is a simple explanation of related problems, only in the image. For example, the individual macroblocks should be extended to the macroblock dependencies of the entire image.

Each image frame of the encoding end and the decoding end is encoded or decoded in time sequence, and the encoding end and the decoding end are in one-to-one correspondence, taking the frame number as an example, wherein the N-1 frame is first encoded or decoded, and the Nth frame is second, Nth +1 again.

The blocks with the same color between different image frames (the gray scale processing in the figure, the blocks with the same gray scale) represent that there is a dependency relationship between them, and the data of the Nth frame is related to the N-1 frame and N. +1 frame has a relationship, wherein the gray corresponding to yellow, green, blue, orange, and purple in the N frame corresponds to the same gray level of the N-1 frame, and the other gray blocks of the Nth frame and the Nth The +1 frame gray area corresponds.

The area divided by the thick line represents the strip area of the AVC, and the square grid inside represents the macroblock information (Macroblock) in the image codec;

The N-1 frame of the decoding end has a thinner slash coverage area code. The decoded code stream of this part is lost during transmission. Due to the temporal correlation between the decoded pictures, the N area and the corresponding area of the N+1 frame are decoded. The image is abnormal, which in turn causes the abnormal image to spread (the N-frame at the decoding end and the oblique line in the N+1 frame in Fig. 1 represent the position of the diffusion).

Based on the above codec principle, as shown in FIGS. 2 and 3, the present invention discloses a real-time image adaptive recovery system with network transmission characteristics friendly;

The encoding end 100 and the decoding end 200 are two parts, wherein the encoding end 100 includes:

a coded image index generating module 101, the module generates a unique image index of the coded sequence, and is used by the encoding end to determine a time position corresponding relationship between the lost image frame and the current frame to be encoded fed back by the encoding end;

Encoding an image macroblock information backup module 102, the module backing up macroblock type information of the encoded image and motion vector information, for calculating a packet loss data propagation path and a region;

a packet loss data propagation area calculation module 103, configured to calculate a propagation area of the packet loss data according to the correlation between the interframe information, including macroblock type information and motion vector information;

The packet loss area coding type control module 104 is configured to control the coded macroblock type of the coding end 100. If it is determined that the macroblock to be coded is within the range of the 'dropped data propagation area', the intra macroblock coding is performed. Block the abnormal propagation; otherwise, perform the normal encoding process;

The decoding end 200 includes:

The decoded image index extraction module 201 is configured to extract an index of the currently decoded frame image. If a packet loss occurs, the information needs to be transmitted to the encoding end, and the encoding end uses the information to determine the decoded frame between the lost packet and the frame to be encoded. Time correlation;

Decoding a packet loss information statistic module 202, where the module counts the location of the packet loss data and the coverage of the packet loss data, including a stripe level and a macroblock level, which are used by the encoding end to check the information to prevent misjudgment;

The packet loss information feedback module 203 is decoded, and the module transmits the packet loss packet loss statistics and the packet loss image index information to the encoding end through the network protocol for abnormal image recovery.

Based on the above-mentioned network transmission characteristic friendly real-time image adaptive recovery system, the present invention also discloses a network transmission characteristic friendly real-time image adaptive recovery method implemented by the system. Referring to Figures 4 and 5, the method includes :

In the first step, the encoded image index generating module 101 generates a unique image index of the encoded sequence, and the encoded image macroblock information backup module 102 backs up the macroblock type information and the motion vector information of the encoded image, so that the frame image can be saved in advance at the encoding end 100. Index and coded macroblock type information, motion vector information;

In the second step, the decoded image index extraction module 201 extracts an index of the currently decoded frame image. If a packet loss occurs, the decoding packet loss information statistic module 202 counts the location of the lost packet data and the coverage of the lost packet data (including the strip level and the macro). At the block level, the decoding packet loss information feedback module 203 transmits the packet loss packet loss statistics and the packet loss image index information to the encoding end 100 through the network protocol; if no packet loss occurs, the decoding terminal 200 performs a normal decoding process;

In the third step, if the encoding end 100 receives the packet loss location information (including the decoded frame image index extracted by the decoded image index extraction module 201 and the strip index indexed by the decoding packet loss information statistic module 202), the packet loss data propagation area is calculated. The module 103 calculates a macroblock correspondence relationship between the decoded frame and the current frame to be encoded according to the packet loss location information fed back by the decoding end 200;

In the fourth step, the packet loss type coding type control module 104 of the encoding end 100 performs a macroblock type decision to control the coded macroblock type of the encoding end 100, if it is determined that the macroblock to be encoded is in the range of 'dropped data propagation area' Within the frame, the intra macroblock coding is performed to block the abnormal propagation; otherwise, the normal encoding process is performed, and the intra macroblock (intra) mode is used to encode the associated region, and the propagation path of the decoded abnormal image is blocked. ;

In the fifth step, the encoding end 100 sends the encoded frame code stream to the decoding end, and after the decoding end receives the encoded frame data, it can be restored to a normal image by macroblock decoding.

The coded image adaptive recovery system and method for optimizing network transmission characteristics provided by the present invention can accurately estimate the range of diffusion of the 'abnormal image' by using the cooperative cooperation of the encoding end 100 and the decoding end 200 to transmit additional information through the network protocol. The encoding end 100 performs intra macroblock encoding in a specific area according to the calculated 'diffusion range' information, which can actually block the abnormal image diffusion without applying for an IDR frame, and the size of the encoded frame is less than or equal to the IDR frame. Size, the size of the coded frame in the normal network packet loss environment is much smaller than the IDR frame size, thereby avoiding the frequent application of the IDR defect, and has great benefits for the coding quality and the network transmission characteristics. The present invention can be applied to video conferencing systems, video surveillance, and video related APP applications, but is not limited to the above.

The above is only a specific embodiment of the present invention, and is not intended to limit the scope of the present invention. Equivalent changes made in accordance with the design ideas of this case fall within the scope of protection of this case.

Claims

A real-time image adaptive recovery system with network transmission characteristics is characterized in that it comprises two parts: an encoding end and a decoding end, wherein the encoding end comprises:

a coded image index generating module, the module generates a unique image index of the coded sequence, and is used by the encoding end to determine a time position corresponding relationship between the lost image frame and the current frame to be encoded fed back by the encoding end;

a coded image macroblock information backup module, the module backing up macroblock type information of the coded image and motion vector information, for calculating a packet loss data propagation path and a region;

a packet loss data propagation area calculation module, configured to calculate a propagation area of the lost packet data according to the correlation between the inter-frame information, including macroblock type information and motion vector information;

The packet loss type coding type control module is configured to control the coded macroblock type of the coding end. If it is determined that the macroblock to be coded is within the range of the packet loss data propagation area, the intra macroblock coding is performed to block the abnormal propagation. Otherwise, the normal encoding process is performed;

The decoding end includes:

Decoding an image index extraction module, the module extracts an index of the currently decoded frame image, and if a packet loss occurs, the information needs to be transmitted to the encoding end, and the encoding end uses the information to determine the decoded frame between the lost packet and the frame to be encoded. Time correlation

Decoding a packet loss information statistics module, where the module counts the location of the packet loss data and the coverage of the packet loss data, including a stripe level and a macroblock level, which are used by the encoding end to check the type of information;

Decoding the packet loss information feedback module, the module transmits the packet loss packet loss statistics and the packet loss image index information to the encoding end through the network protocol for abnormal image recovery.
A real-time image adaptive recovery method with network transmission characteristics, including:

Preserving a frame image index and encoding macroblock type information and motion vector information in the encoding end;

The encoding end calculates a macroblock correspondence relationship between the decoded frame and the current frame to be encoded according to the packet loss location information fed back by the decoding end; the packet loss location information includes a decoded frame index and a stripe index;

Then, the encoding end encodes the intra macroblock mode in the area where the association exists, and blocks the propagation path of the decoded abnormal image;

Then, the encoding end sends the encoded frame code stream, and can immediately return to the normal image after receiving the encoded frame data at the decoding end.