WO2021056575A1

WO2021056575A1 - Low-delay joint source-channel coding method, and related device

Info

Publication number: WO2021056575A1
Application number: PCT/CN2019/109220
Authority: WO
Inventors: 李卫华; 高林; 郭湛; 张亚凡
Original assignee: 华为技术有限公司
Priority date: 2019-09-29
Filing date: 2019-09-29
Publication date: 2021-04-01
Also published as: CN114424552A

Abstract

Disclosed in embodiments of the present application are a low-delay joint source-channel coding method, and a related device. The low-delay joint source-channel coding method comprises: first, down-sampling a target image to obtain a reference frame image and a non-reference frame image; respectively performing image coding on the reference frame image and the non-reference frame image to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded; and on the basis of a channel environment of a current wireless channel, transmitting the first code stream using a first channel resource and transmitting the second code stream using a second channel resource, wherein the first channel resource is superior to the second channel resource. By implementation of the embodiments of the present application, when the channel capacity is reduced, phenomena such as pixelization, jamming and blurring of video information can be avoided, thereby improving visual experience of a user.

Description

A low-delay source-channel joint coding method and related equipment

Technical field

This application relates to the field of wireless transmission technology, and in particular to a low-delay source-channel joint coding method and related equipment.

Background technique

With the development of wireless transmission technology, it is possible to use wireless networks to carry high-definition video, but at the same time, wireless networks carry high-definition video also put forward new requirements for image coding and decoding technology, because of the short-time and time-varying characteristics of wireless channels during wireless transmission. It will cause rapid changes in channel capacity, so the image codec algorithm needs to be able to quickly track and adapt to this change, so that the delay and image quality at the receiving end can be maintained at an acceptable level.

Such as the Joint Photographic Experts Group (JPEG) standard codec and the Moving Picture Experts Group (MPEG) standard codec, these traditional image codec algorithms focusing on compression multiples and image quality, and Corresponding coding adjustments cannot be made promptly and quickly according to the channel capacity. Therefore, when the channel capacity is reduced to below the source code rate, the video information compressed by the traditional image codec algorithm is likely to lose the relevant code stream information during wireless transmission, resulting in the entire decoded macroblock being unable to be received at the receiving end. Reconstruction will cause different degrees of mosaic, stuttering, blurring and other phenomena in the transmitted video, and the look and feel is very poor.

Therefore, how to reduce information loss during transmission and avoid the inability of the receiving end to reconstruct a complete video image due to information loss is an urgent problem to be solved.

Summary of the invention

The embodiments of the present application provide a low-latency source-channel joint coding method and related equipment. By optimizing image coding or transmission mode, information loss can be reduced during video transmission, so as to avoid the receiving end from being unable to play clear and complete videos. Improve user viewing experience.

In the first aspect, the embodiments of the present application provide a low-delay source-channel joint coding method, which may include: down-sampling the target image to obtain a reference frame image and a non-reference frame image; Images are encoded separately to obtain the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image; based on the channel environment of the current wireless channel, the first channel resource and the second channel resource are determined, The first channel resource is used to send the first code stream and the second channel resource is used to send the second code stream, respectively, wherein the first channel resource is better than the second channel resource.

Video wireless transmission technology mainly includes several main links such as video encoding, wireless transmission, video decoding and display. Through the method described in the first aspect, during video encoding and wireless transmission, the encoding end can down-sample the target image to obtain the reference frame image and the non-reference frame image, and then perform image encoding on the reference frame image to obtain the first bit stream. Perform image encoding on the non-reference frame image to obtain a second code stream, and finally use the first channel resource to send the first code stream, and use the second channel resource to send the second code stream. Among them, the first channel resource and the second channel resource are determined according to the channel environment of the current wireless channel, and the first channel resource is better than the second channel resource, that is, it can be understood that the reference frame image can be encoded during wireless transmission. After the first code stream and the second code stream after encoding the non-reference frame image, the second code stream is sent hierarchically. For example, you can set the priority of the first code stream to be higher than the priority of the second code stream, and then send the first code stream separately With the second code stream, according to the channel environment of the current wireless channel, the service quality of the channel when the first code stream is sent is higher than the service quality of the channel when the second code stream is sent. Or, when the channel environment of the current wireless channel is poor, the encoder can send the more important first code stream first, and then send the second code stream with a lower level; or, when the current wireless channel has a poor channel environment, it can send After receiving the first code stream or after a period of time after sending the first code stream, the end sends the second code stream with a lower level. Based on the channel environment of the current wireless channel, the encoding end sends the first code stream and the second code stream separately, so that when the receiving end cannot receive the complete code stream, the probability that the receiving end receives the first code stream can be improved. The first code stream to get a complete reconstructed image. Secondly, before encoding the target image, the target image is down-sampled, and after obtaining multiple copies of lower resolution images, the image encoding is performed, which can make the code stream obtained after down-sampling smaller than the direct code stream. The size of the code stream obtained when the target image is image-encoded makes it easier for the decoder at the receiving end to receive the complete code stream. Therefore, after the encoding end performs image encoding on the target image through downsampling, the encoded stream information is based on the channel environment of the current wireless channel and sent through the wireless channel respectively, so that the encoded stream information can still adapt to real-time changes. The channel environment further reduces the probability of information loss during wireless transmission, thereby avoiding the inability of the receiving end to reconstruct a complete video image due to information loss, and improving the user’s viewing experience.

In a possible implementation manner, the above-mentioned image encoding is performed on the reference frame image and the non-reference frame image respectively, to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image encoding The latter second code stream includes: performing intra-frame compression on each of the reference frame images included in the reference frame image to obtain the first code stream; performing the frame image with the residual difference between the non-reference frame image and the reference frame image Inner compression obtains the corresponding above-mentioned second code stream. When implementing the embodiments of this application, the encoding end can select a simpler down-sampling to layer the target image and reduce the resolution. At the same time, the intra-frame compression encoding method used when encoding the layered duplicate image, because the frame Inner compression is spatial compression. When compressing a frame of image, only the data of the current frame is considered without considering the redundant information between adjacent frames, while the inter-frame compression needs to refer to other frame data, and the compression rate is large. Therefore, consider In terms of image delay, the embodiment of the present application does not use inter-frame compression when performing image encoding on the down-sampled replica image. Moreover, the use of intra-frame compression can cause the channel capacity to jitter and drop below the initial bit rate, so that the decoder at the receiving end can receive the bit stream information encoded based on the smaller-resolution copy image image, and can be based on The reconstructed image obtained from the code stream information. At the same time, when the channel capacity is reduced, the probability of information loss in the transmission process is reduced, and the phenomenon of mosaic, stuttering, and blurring of the video image information is avoided, and the user's perception and experience are improved.

In a possible implementation manner, the above-mentioned channel environment also includes channel capacity; the above-mentioned image coding is performed on the above-mentioned reference frame image and the above-mentioned non-reference frame image to obtain the first code stream after the coding of the above-mentioned reference frame image, and the above-mentioned non-reference frame image. Before the second code stream after the reference frame image is coded, it further includes: determining the coding parameters corresponding to the reference frame image and the non-reference frame image based on the channel capacity, wherein the coding parameters are used to control the corresponding image in the image processing. During encoding, a corresponding code stream is generated according to the target code rate, and the target code rates respectively corresponding to the first code stream and the second code stream are less than or equal to the channel capacity. When implementing the embodiments of this application, the encoding end can obtain the channel capacity of the current channel through the transmitter when the channel capacity changes, and adjust the encoding parameters of the encoder when encoding the duplicate image according to the feedback channel capacity information. , So that the code rate of the coded code stream can be adapted to the current channel capacity, and the target code stream can be sent smoothly, reducing the loss of the code stream due to the reduced channel capacity. At the same time, it also reduces the inability to transmit the complete code stream, so that the decoder at the receiving end cannot reconstruct the image obtained based on the missing code stream information, which reduces the probability of video image freezing and blurring, and improves the user's viewing experience.

In a possible implementation manner, the foregoing channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate; the foregoing is based on current To determine the first channel resource and the second channel resource, and use the first channel resource to send the first code stream and the second channel resource to send the second code stream, including: based on the current channel environment , Determine the transmission parameters corresponding to the first code stream and/or the second code stream, wherein the transmission parameters are used to transmit the code stream according to the target modulation and coding strategy information and/or the target transmission power, and the first code stream The sending parameter of is better than the sending parameter of the second code stream; the first code stream is sent according to the sending parameter corresponding to the first code stream, and the second code stream is sent according to the sending parameter corresponding to the second code stream. When implementing the embodiments of this application, the encoder can obtain the channel environment of the current wireless channel through the transmitter when the channel changes, such as: channel bandwidth, signal to interference plus noise ratio, signal to noise ratio, received signal strength indicator , Duty cycle, bit rate, etc.), according to the channel environment to adjust the transmission parameters (such as: transmission power, modulation and coding strategy information, etc.) when sending the first code stream and/or when sending the second code stream, it is understandable Yes, after adjusting the sending parameters according to the channel information, the sending parameters of the first code stream are different from the sending parameters of the second code stream, and the sending parameters of the first code stream can make the transmitter more expensive than the sending parameters of the second code stream. The first code stream is sent at a faster transmission speed and higher quality of service (QoS). Therefore, in the process of wireless transmission, the channel resource of the first code stream is better than the channel resource of the second code stream to be sent, that is, the transmission quality when sending the first code stream is higher than when sending the second code stream. Transmission quality, or, preferentially send more important code streams (that is, code streams that have a small number of duplicate images after image encoding), causing jitter in channel capacity, etc., causing the decoder at the receiving end to fail to receive the complete code stream. Because of the characteristics of intra-frame compression, it can avoid the inability of the receiver to reconstruct the complete video image due to the loss of information. The decoder is only based on the reconstructed image obtained by the partial code stream received first, which can reduce the mosaic and stutter of the video information. , Blur and other phenomena to enhance the user’s perception and experience.

In a possible implementation manner, the wireless channel includes a first wireless channel and a second wireless channel; the first channel resource and the second channel resource are determined based on the current channel environment, and the first channel resource is used to transmit the The first code stream and the sending of the second code stream using the second channel resource include: determining the first wireless channel and the second wireless channel based on the current channel environment, and passing the first code stream through the first wireless channel Sending, sending the above-mentioned second code stream through a second wireless channel, wherein the service quality guarantee mechanism of the above-mentioned first wireless channel is higher than that of the above-mentioned second wireless channel. When implementing the embodiments of the present application, the encoding end may classify the video source during the wireless transmission process, and the channel resource of the first code stream is better than the channel resource of the second code stream for transmission. For example, the first code stream is mapped to a video (Video, VI) service, and the second code stream is mapped to a best effort (Best Effort, BE) service. Therefore, when the channel capacity is jittered, the first code stream and the second code stream sent through the wireless channel use different QoS guarantee levels, which increases the probability of receiving the first code stream at the receiving end, so that the decoder at the receiving end can When only receiving the first code stream sent through a channel with a high quality of service guarantee mechanism, the image can still be reconstructed only through the first code stream to avoid the inability of the receiving end to reconstruct the complete video image due to information loss and reduce the video information Mosaic, stuttering, blurring, etc. appear to enhance the user’s perception and experience.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned method further includes: acquiring each of the multi-frame target images included in the above-mentioned target video corresponds to the above-mentioned reference The code stream of the frame image; the audio information in the target video is obtained; each frame image of the multi-frame image included in the target video corresponds to the code stream of the reference frame image and the audio information through the first wireless channel. When implementing the embodiments of this application, the encoding end can obtain all target images in the target video included in the multi-frame target image. The code streams corresponding to the above-mentioned reference frame image and audio information are mapped to the VI service transmission with a high quality of service guarantee mechanism. It is understandable Yes, the above audio information may be audio information that has been encoded by voice. When the channel changes, the decoder at the receiving end can reconstruct the video based only on the code streams and audio information corresponding to all target images received through the channel with a high quality of service guarantee mechanism, so as to increase the probability of receiving the first code stream at the receiving end. Avoid the inability of the receiving end to reconstruct the complete video image due to the loss of information, reduce the phenomenon of no sound and freeze in the video information, and improve the user's visual experience.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned method further includes: acquiring each of the multi-frame target images included in the above-mentioned target video corresponds to the above-mentioned non- The code stream of the reference frame image; the code stream of each frame image in the multi-frame image included in the target video corresponding to the non-reference frame image is sent through the second wireless channel. When implementing the embodiments of the present application, the encoding end may separately map to the BE service transmission with a slightly lower service quality assurance mechanism after acquiring the code streams of all target images corresponding to the aforementioned non-reference frame images in the multi-frame target images included in the target video. Therefore, even if the code streams corresponding to the non-reference frame images of all target images lose part or even all of the code stream information when the channel changes, the decoder can only be based on the results of all target images received through the channel with a high quality of service guarantee mechanism. The code stream and audio information corresponding to the first copy reconstruct the video to prevent the receiving end from being unable to reconstruct the complete video image due to the loss of information, reduce the phenomenon of mosaic, stutter, and blur in the video information, and improve the user's perception and experience.

In the second aspect, the embodiments of the present application provide another low-delay source-channel joint coding method, which may include: down-sampling the target image to obtain reference frame images and non-reference frame images; according to the current wireless channel channel environment, Determine the coding parameters corresponding to the reference frame image and the non-reference frame image; according to the coding parameters, perform image encoding on the reference frame image and the non-reference frame image respectively to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image encoding After the second code stream; send the first code stream and the second code stream separately.

To implement the embodiments of this application, the encoding end may first downsample the target image to obtain the reference frame image and the non-reference frame image; then according to the channel environment of the current wireless channel, determine the encoding parameters corresponding to the reference frame image and the non-reference frame image, Then use the coding parameters to encode the reference frame image and the non-reference frame image, so that the bit rates of the first bit stream and the second bit stream are lower than the channel capacity of the current wireless channel; finally, the first bit stream and the second bit stream are The code streams are sent out through the wireless channel respectively. Among them, according to the channel information of the current wireless channel, adjusting the encoding parameters during image encoding can make the code rate of the generated code stream lower than the channel capacity of the current wireless channel, and the code stream will not be due to the low channel capacity when the code stream is wirelessly transmitted. The complete bit stream cannot be transmitted, or the channel capacity is jittered, or the channel capacity is reduced to the initial code rate (for example: the initial code rate can be the code rate when the target image is compressed according to the traditional encoding method or the preset code rate) Sometimes, part of the stream information is lost. Therefore, the image coding and decoding algorithm of the embodiment of the present application can quickly track and adapt to changes in the channel environment of the wireless channel, so that the delay and image quality of the receiving end can be maintained at an acceptable level. Secondly, before encoding the target image, the encoding end first down-samples the target image, and then performs image encoding after obtaining multiple copies of lower resolution images, which can make the code stream required after the down-sampling is obtained. It is smaller than the size of the code stream obtained when directly encoding the target image, which makes it easier for the decoder at the receiving end to receive the complete code stream. Therefore, the code stream information obtained by encoding the image according to the channel information of the current wireless channel can be adapted to the real-time changing channel environment, thereby reducing the probability of information loss during transmission and preventing the receiving end from being unable to reconstruct complete information due to information loss. Video images to enhance the user’s viewing experience.

In a possible implementation manner, the aforementioned encoding parameters are used to control the corresponding image to generate a corresponding code stream according to the target code rate when the corresponding image is encoded; the aforementioned encoding parameters are used to compare the aforementioned reference frame image and the aforementioned non-reference frame image Image encoding is performed separately to obtain the first code stream after the reference frame image is coded, and the second code stream after the non-reference frame image is coded, including: each of the reference frame images included in the reference frame image according to the target The code rate is compressed within the frame to obtain the first code stream; the residuals of each of the non-reference frame images contained in the non-reference frame image and all the reference frame images included in the reference frame image are in accordance with the target bit rate Perform the above-mentioned intra-frame compression to obtain the corresponding above-mentioned second code stream. When implementing the embodiments of this application, the encoding end can select a simpler down-sampling to layer the target image and reduce the resolution. At the same time, the intra-frame compression encoding method used when encoding the layered duplicate image, because the frame Inner compression is spatial compression. When compressing a frame of image, only the data of the current frame is considered without considering the redundant information between adjacent frames, while the inter-frame compression needs to refer to other frame data, and the compression rate is large. Therefore, consider In terms of image delay, the encoding end uses intra-frame compression on the down-sampled copy image, which can cause the channel capacity to jitter and reduce to below the initial bit rate, so that the decoder at the receiving end can accept the copy based on a smaller resolution. The code stream information of the image image can be reconstructed from the code stream information. At the same time, when the channel capacity is reduced, the probability of information loss in the transmission process is reduced, and the phenomenon of mosaic, stuttering, and blurring of the video image information is avoided, and the user's perception and experience are improved.

In a possible implementation, the above-mentioned channel environment includes one or more of the bandwidth of the channel, the signal to interference plus noise ratio, the signal-to-noise ratio, the received signal strength indicator, the duty cycle, and the bit rate; Before the first code stream and the second code stream, the method further includes: determining the transmission parameters corresponding to the first code stream and/or the second code stream according to the channel environment, wherein the transmission parameters are used to follow the target For modulation and coding strategy information and/or target transmission power to transmit a code stream, the transmission parameters of the first code stream are better than the transmission parameters of the second code stream. When implementing the embodiments of this application, the encoder can obtain the channel environment of the current wireless channel through the transmitter when the channel changes, and adjust the transmission when sending the first code stream and/or the second code stream according to the channel environment. Parameters (such as transmission power, modulation and coding strategy information, etc.). It can be understood that after adjusting the transmission parameters according to the channel information, the transmission parameters of the first code stream are different from the transmission parameters of the second code stream. Compared with the transmission parameters of the second code stream, the transmission parameters enable the transmitter to transmit the first code stream at a faster transmission speed and a higher quality of service (QoS). Therefore, in the process of wireless transmission, the encoding end sends the first code stream using channel resources better than the channel resources of the second code stream, that is, the transmission quality when sending the first code stream is higher than when sending the second code stream. The transmission quality at the time, or the more important code stream is sent first (that is, the code stream after the image is encoded with a small number of duplicate images), so that the decoder at the receiving end can still receive the complete code stream when the channel capacity is jittered. , To avoid information loss, reduce the phenomenon of mosaic, freeze, and blur in the video information, and improve the user's perception and experience.

In a possible implementation manner, the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the above-mentioned sending the above-mentioned first code stream and the above-mentioned second code stream respectively includes: passing the above-mentioned first code stream through a first wireless channel. Channel transmission is to send the above-mentioned second code stream through a second wireless channel, wherein the service quality guarantee mechanism of the above-mentioned first wireless channel is higher than that of the above-mentioned second wireless channel. When implementing the embodiments of the present application, the encoding end may classify the video source during the wireless transmission process, and the channel resource of the first code stream is better than the channel resource of the second code stream for transmission. Therefore, when the channel capacity is jittered, the first code stream and the second code stream sent by the encoding end through the wireless channel use different QoS guarantee levels, which increases the probability that the receiving end receives the first code stream and makes the receiving end decode When only receiving the first bit stream sent through a channel with a high quality of service guarantee mechanism, the receiver can still reconstruct the image only through the first bit stream, avoiding the inability of the receiving end to reconstruct the complete video image due to loss of information, which reduces Mosaic, stuttering, blurring and other phenomena appear in the video information, which improves the user's perception and experience.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned method further includes: acquiring each of the multi-frame target images included in the above-mentioned target video corresponds to the above-mentioned reference The code stream of the frame image; the audio information in the target video is obtained; each frame image of the multi-frame image included in the target video corresponds to the code stream of the reference frame image and the audio information through the first wireless channel. When implementing the embodiments of this application, the encoding end can obtain all target images in the target video included in the multi-frame target image. The code streams corresponding to the above-mentioned reference frame image and audio information are mapped to the VI service transmission with a high quality of service guarantee mechanism. It is understandable Yes, the aforementioned audio information may be audio information encoded by voice. When the channel changes, the decoder at the receiving end can reconstruct the video based only on the code streams and audio information corresponding to all target images received through the channel with a high quality of service guarantee mechanism, so as to increase the probability of receiving the first code stream at the receiving end. Avoid the inability of the receiving end to reconstruct the complete video image due to the loss of information, reduce the phenomenon of no sound and freeze in the video information, and improve the user's visual experience.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned method further includes: acquiring each of the multi-frame target images included in the above-mentioned target video corresponds to the above-mentioned non-target image. The code stream of the reference frame image; the code stream of each frame image in the multi-frame image included in the target video corresponding to the non-reference frame image is sent through the second wireless channel. When implementing the embodiments of the present application, the encoding end may separately map to the BE service transmission with a slightly lower service quality assurance mechanism after acquiring the code streams of all target images corresponding to the aforementioned non-reference frame images in the multi-frame target images included in the target video. Therefore, even if the code streams corresponding to the non-reference frame images of all target images lose part or even all of the code stream information when the channel changes, the decoder can only be based on the results of all target images received through the channel with a high quality of service guarantee mechanism. The code stream and audio information corresponding to the first copy reconstruct the video to prevent the receiving end from being unable to reconstruct the complete video image due to the loss of information, reduce the phenomenon of mosaic, stutter, and blur in the video information, and improve the user's perception and experience.

In a third aspect, embodiments of the present application provide a low-delay source-channel joint decoding method, which may include: receiving a first code stream sent by an encoding end, where the first code stream is a code stream obtained after image encoding of a reference frame image , Wherein the reference frame image includes one or more images obtained after down-sampling the target image; after decoding the first code stream, the reference frame image corresponding to the first code stream is obtained; according to the reference frame image, reconstruct Target image.

In the process of wireless transmission, through the decoding method provided by the third aspect of the present application, the decoding end (ie, the receiving end) can receive the first code stream sent by the encoding end, and then decode the first code stream to obtain the first code stream. A reference frame image corresponding to a code stream is then reconstructed based on the reference frame image. It should be noted that the first code stream is the code stream obtained after image encoding of the reference frame image, where the reference frame image includes one or more images obtained after down-sampling the target image. Therefore, the code received by the receiving end The stream is obtained by down-sampling the target image and then image encoding. Also, because downsampling can sample the target image into multiple copies of lower resolution, part of them can be used as the reference frame image. The code stream size obtained by down-sampling and re-encoding the target image is smaller than the code stream size obtained when the target image is directly encoded. Therefore, the code stream after down-sampling the target image may better adapt to changes in the wireless channel. It also reduces the probability of information loss during the transmission process, and at the same time avoids the inability of the receiving end to reconstruct the complete video image due to the loss of information.

In a possible implementation manner, the target image is any one of multiple target images included in the target video; the reconstruction of the target image according to the reference frame image includes: according to the reference frame image and the target image At least one of the reference frame image and the non-reference frame image corresponding to an adjacent frame of target image is reconstructed by using an interpolation algorithm. To implement the embodiments of this application, the receiving end can use the interpolation algorithm to map the reference frame image corresponding to the first code stream to the target image of the previous frame of the target image corresponding to the first code stream when the receiving end only receives the first code stream. The reference frame image is reconstructed into the target image. Therefore, when the receiving end receives the first bit stream, it can avoid the inability of the receiving end to reconstruct the complete video image due to the loss of information, and reduce the appearance of no sound and freeze in the video information. Phenomenon to enhance the user’s perception and experience.

In a possible implementation, the method further includes: within a preset time period after receiving the first code stream, receiving a second code stream sent by the encoding end, wherein the second code stream is non- The reference frame image is the code stream obtained after the image encoding, and the non-reference frame image includes the remaining images except the reference frame image obtained after down-sampling the target image; if the second code stream is incomplete, the first After the two bit streams are decoded, the corresponding incomplete non-reference frame image is obtained; according to the incomplete non-reference frame image, the peripheral pixels of the incomplete non-reference frame image are determined; and the target image is reconstructed according to the reference frame image, including : According to the surrounding pixels of the reference frame image and the incomplete non-reference frame image, the target image is reconstructed by an interpolation algorithm. To implement the embodiments of this application, the receiving end can determine whether to receive the second code stream within the preset time period of receiving the first code stream, if not, according to the first code stream and the adjacent frame of target image Corresponding to at least one of the reference frame image and the non-reference frame image, reconstruct the target image; if the second code stream is received, determine whether the received second code stream is incomplete, if the received second code stream is incomplete The incomplete second code stream and the complete first code stream can be used to interpolate to obtain the target image, which avoids the inability of the receiving end to reconstruct the complete video image due to partial information loss, and reduces the phenomenon of no sound and freezing in the video information. Improve the user's perception experience.

In a possible implementation, the method further includes: if it is determined that the second code stream is complete, after image decoding the second code stream, the corresponding non-reference frame image is obtained; , Reconstructing the target image includes: splicing the reference frame image and the non-reference frame image to reconstruct the target image. By implementing the embodiments of this application, when the receiving end receives the complete first code stream and the second code stream, the image can be directly decoded and then spliced into an image, which adapts to the change of the wireless channel and preserves the complete information during the transmission process. It prevents the receiving end from being unable to reconstruct the complete video image due to the loss of information, and improves the user's viewing experience.

In a fourth aspect, an embodiment of the present application provides a low-delay source-channel joint coding device, which is characterized by comprising: an encoder and a transmitter, wherein the encoder is used for down-sampling the target image to obtain a reference Frame images and non-reference frame images; image encoding is performed on the reference frame image and the non-reference frame image to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image encoded The second code stream; a transmitter, used to determine a first channel resource and a second channel resource based on the channel environment of the current wireless channel, and respectively use the first channel resource to send the first code stream and use the first channel resource The second code stream is sent with two channel resources, wherein the first channel resource is better than the second channel resource.

In a possible implementation manner, the encoder is configured to perform image encoding on the reference frame image and the non-reference frame image to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image When the second code stream is encoded, the encoder is specifically configured to: perform intra-frame compression on each of the reference frame images included in the reference frame image to obtain the first code stream; combine the non-reference frame image with the reference frame The residual of the image is subjected to the intra-frame compression to obtain the corresponding second code stream.

In a possible implementation manner, the above-mentioned channel environment also includes channel capacity; the above-mentioned transmitter performs image encoding on the above-mentioned reference frame image and the above-mentioned non-reference frame image respectively to obtain the first code after encoding the above-mentioned reference frame image Before the second code stream after the encoding of the non-reference frame image, it is also used to determine the encoding parameters corresponding to the reference frame image and the non-reference frame image based on the channel capacity, wherein the encoding parameters are used to control When the corresponding image is encoded, the corresponding code stream is generated according to the target code rate, and the target code rates corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity.

In a possible implementation manner, the above-mentioned channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate; the above-mentioned transmitter , When used to determine the first channel resource and the second channel resource based on the current channel environment, and use the first channel resource to send the first code stream and the second channel resource to send the second code stream, respectively, It is specifically used for: determining the transmission parameters corresponding to the first code stream and/or the second code stream respectively based on the current channel environment, wherein the transmission parameters are used for target modulation and coding strategy information and/or target transmission Power transmission code stream, the transmission parameters of the first code stream are better than the transmission parameters of the second code stream; the first code stream is transmitted according to the transmission parameters corresponding to the first code stream, and the transmission parameters corresponding to the second code stream are Sending parameters to send the above second code stream.

In a possible implementation manner, the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the above-mentioned transmitter is used in the above-mentioned current channel environment to determine the first channel resource and the second channel resource, and respectively use When the first channel resource is used to send the first code stream and the second channel resource is used to send the second code stream, it is specifically used to: determine the first wireless channel and the second wireless channel based on the current channel environment, and The first code stream is sent through a first wireless channel, and the second code stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned encoder is further used for: acquiring the corresponding target image of each frame of the multi-frame target image included in the above-mentioned target video The code stream of the reference frame image; the audio information in the target video is obtained; the transmitter is also used to: pass each frame image of the multi-frame image included in the target video corresponding to the code stream of the reference frame image and the audio information The above-mentioned first wireless channel transmission.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned encoder is further used for: acquiring the corresponding target image of each frame of the multi-frame target image included in the above-mentioned target video The code stream of the non-reference frame image; the transmitter is further used to: transmit the code stream of each frame image corresponding to the non-reference frame image in the multi-frame image included in the target video through the second wireless channel.

In a fifth aspect, an embodiment of the present application provides another low-delay source-channel joint coding device, which is characterized by comprising: an encoder and a transmitter, wherein the encoder is used to down-sample the target image to obtain Reference frame image and non-reference frame image; according to the channel environment of the current wireless channel, determine the coding parameters corresponding to the reference frame image and non-reference frame image; according to the coding parameters, perform image coding on the reference frame image and the non-reference frame image to obtain The first code stream after the coding of the reference frame image and the second code stream after the coding of the non-reference frame image; the transmitter is used to send the first code stream and the second code stream respectively.

In a possible implementation manner, the aforementioned encoding parameters are used to control the corresponding image to generate a corresponding code stream according to the target code rate during image encoding; the encoder is used to compare the aforementioned reference frame image and the aforementioned reference frame image according to the aforementioned encoding parameters. When the non-reference frame image is encoded separately to obtain the first code stream after the reference frame image is encoded, and the second code stream after the non-reference frame image is encoded, it is specifically used to: Each of the reference frame images is intra-compressed according to the target code rate to obtain the first code stream; the residual difference between the non-reference frame image and the reference frame image is subjected to the intra-frame compression according to the target code rate to obtain the corresponding The above second code stream.

In a possible implementation manner, the foregoing channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate; the foregoing transmitter , Used to separately transmit the first code stream and the second code stream, and also used to: determine the transmission parameters corresponding to the first code stream and/or the second code stream according to the channel environment, wherein the The sending parameter is used to send the code stream according to the target modulation and coding strategy information and/or the target transmission power, and the sending parameter of the first code stream is better than the sending parameter of the second code stream.

In a possible implementation manner, the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the above-mentioned transmitter is used to transmit the above-mentioned first code stream and the above-mentioned second code stream, and is specifically used to: The first code stream is sent through a first wireless channel, and the second code stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a sixth aspect, an embodiment of the present application provides a low-delay source-channel joint coding device, which is characterized by comprising: a first sampling unit for down-sampling a target image to obtain a reference frame image and a non-reference frame image ; The first coding unit is used to perform image coding on the reference frame image and the non-reference frame image respectively, to obtain the first code stream after the reference frame image is coded, and the second code stream after the non-reference frame image is coded; the first transmission The unit is used to determine the first channel resource and the second channel resource based on the channel environment of the current wireless channel, and respectively use the first channel resource to send the first code stream and the second channel resource to send the second code stream, where the first The channel resource is better than the second channel resource.

In a possible implementation manner, the above-mentioned first encoding unit is specifically configured to perform intra-frame compression on each of the above-mentioned reference frame images included in the above-mentioned reference frame image to obtain the above-mentioned first code stream; The residuals of each of the non-reference frame images and all the reference frame images included in the reference frame image are subjected to the intra-frame compression to obtain the corresponding second code stream.

In a possible implementation manner, the foregoing channel environment further includes channel capacity; the device further includes: a first encoding parameter unit, configured to perform image encoding on the foregoing reference frame image and the foregoing non-reference frame image separately to obtain Before the first code stream after the encoding of the reference frame image and the second code stream after the encoding of the non-reference frame image, based on the channel capacity, the encoding parameters corresponding to the reference frame image and the non-reference frame image are determined, wherein, The encoding parameter is used to control the corresponding image to generate a corresponding code stream according to the target code rate when the image is encoded. The target code rate corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity.

In a possible implementation manner, the foregoing channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate; The unit is specifically configured to determine the transmission parameters corresponding to the first code stream and/or the second code stream respectively based on the current channel environment, wherein the transmission parameters are used to modulate and encode strategy information according to the target and/or the target The code stream is transmitted at the transmit power, and the transmission parameters of the first code stream are better than the transmission parameters of the second code stream; the first code stream is transmitted according to the transmission parameters corresponding to the first code stream, and the second code stream corresponds to The sending parameters of sending the above second code stream.

In a possible implementation manner, the wireless channel includes a first wireless channel and a second wireless channel; the first sending unit is specifically configured to: determine the first wireless channel and the second wireless channel based on the current channel environment , Sending the first code stream through a first wireless channel, and sending the second code stream through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned device further includes: a first acquisition unit configured to acquire each of the multi-frame target images included in the above-mentioned target video. One frame of target image corresponds to the code stream of the above-mentioned reference frame image; acquiring the audio information in the above-mentioned target video; the second sending unit is used to correspond each frame of the multi-frame image included in the above-mentioned target video to the code stream of the above-mentioned reference frame image And the above-mentioned audio information is transmitted through the above-mentioned first wireless channel.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned device further includes: a second acquiring unit configured to acquire each of the multi-frame target images included in the above-mentioned target video. One frame of the target image corresponds to the code stream of the non-reference frame image; the third sending unit is used to pass the code stream of each frame image corresponding to the non-reference frame image in the multi-frame image included in the target video through the second wireless channel send.

In a seventh aspect, an embodiment of the present application provides another low-delay source-channel joint coding device, which is characterized in that it includes: a second sampling unit for down-sampling the target image to obtain a reference frame image and a non-reference frame Image; the second coding parameter unit is used to determine the coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel; the second coding unit is used to compare the reference frame image and the non-reference frame image according to the coding parameters Frame images are encoded separately to obtain the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image; the fourth sending unit is used to send the first code stream and the second code stream respectively .

In a possible implementation manner, the foregoing encoding parameters are used to control the corresponding image to generate a corresponding code stream according to the target code rate during image encoding; the foregoing second encoding unit is specifically used to: Each of the aforementioned reference frame images is intra-compressed according to the aforementioned target code rate to obtain the aforementioned first code stream; the residual difference between the aforementioned non-reference frame image and the aforementioned reference frame image is subjected to the aforementioned intra-frame compression according to the aforementioned target code rate to obtain the corresponding The above second code stream.

In a possible implementation manner, the foregoing channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate; the fourth The sending unit is configured to separately send the first code stream and the second code stream, and is also used to: determine the transmission parameters corresponding to the first code stream and/or the second code stream according to the channel environment, wherein The foregoing transmission parameters are used to transmit the code stream according to the target modulation and coding strategy information and/or the target transmission power, and the transmission parameters of the first code stream are better than the transmission parameters of the second code stream.

In a possible implementation manner, the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the above-mentioned fourth sending unit is specifically configured to: send the above-mentioned first code stream through the first wireless channel, and transmit the above-mentioned second code stream through the first wireless channel. The code stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned device further includes: a third acquisition unit configured to acquire each of the multi-frame target images included in the above-mentioned target video. One frame of target image corresponds to the code stream of the aforementioned reference frame image; acquiring the audio information in the aforementioned target video; and the fifth sending unit is used to correspond each frame of the multi-frame image included in the aforementioned target video to the code stream of the aforementioned reference frame image And the above-mentioned audio information is transmitted through the above-mentioned first wireless channel.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned device further includes: a fourth acquisition unit configured to acquire each of the multi-frame target images included in the above-mentioned target video. One frame of target image corresponds to the code stream of the aforementioned non-reference frame image; the sixth sending unit is configured to pass the code stream of each frame image corresponding to the aforementioned non-reference frame image in the multi-frame image included in the aforementioned target video through the second wireless channel send.

In an eighth aspect, an embodiment of the present application provides a low-delay source-channel joint decoding device, which is characterized by comprising: a receiving unit, configured to receive a first code stream sent by an encoding end, and the first code stream is a reference frame image The code stream obtained after image encoding, wherein the reference frame image includes one or more images obtained after down-sampling the target image; the decoding unit is used to decode the first code stream to obtain the first code stream Corresponding to the reference frame image; the image unit is used to reconstruct the target image according to the reference frame image.

In a possible implementation manner, the above-mentioned target image is any one of multiple frames of target images included in the target video; the above-mentioned image unit is specifically used for: according to a frame of target image adjacent to the above-mentioned target image according to the above-mentioned reference frame image Corresponding to at least one of the reference frame image and the non-reference frame image, the above-mentioned target image is reconstructed through an interpolation algorithm.

In a possible implementation manner, the device further includes: a fifth acquiring unit, configured to receive the second code stream sent by the encoding end within a preset time period after receiving the first code stream, where The second code stream is a code stream obtained by encoding a non-reference frame image, and the non-reference frame image includes images obtained after down-sampling the target image except for the reference frame image; if the first The second code stream is incomplete, and the second code stream is decoded to obtain the corresponding incomplete non-reference frame image; the peripheral pixels of the incomplete non-reference frame image are determined according to the incomplete non-reference frame image; the image unit, It is specifically used for: reconstructing the target image through an interpolation algorithm based on the surrounding pixels of the reference frame image and the incomplete non-reference frame image.

In a possible implementation manner, the device further includes: a sixth acquiring unit, configured to, if it is determined that the second code stream is complete, perform image decoding on the second code stream to obtain the corresponding non-reference frame image The image unit is specifically used for: splicing the reference frame image and the non-reference frame image to reconstruct the target image.

In a ninth aspect, an embodiment of the present application provides a service device, the service device includes a processor, and the processor is configured to support the service device to execute the low-latency source-channel joint coding method provided in the first aspect or the foregoing The second aspect provides a corresponding function in another low-delay source-channel joint coding method. The service device may further include a memory, which is used for coupling with the processor, and stores the necessary program instructions and data of the service device. The service device may also include a communication interface for the service device to communicate with other devices or a communication network.

In a tenth aspect, an embodiment of the present application provides a service device, the service device includes a processor, and the processor is configured to support the service device to execute the corresponding low-latency source-channel joint decoding method provided in the third aspect. Function. The service device may further include a memory, which is used for coupling with the processor, and stores the necessary program instructions and data of the service device. The service device may also include a communication interface for the service device to communicate with other devices or a communication network.

In an eleventh aspect, an embodiment of the present application provides a computer program, which includes instructions, when the computer program is executed by a computer, the computer can execute the low-latency source-channel joint coding provided in the fourth aspect. The process performed by the apparatus or another low-delay source-channel joint coding apparatus provided by the above fifth aspect.

In the twelfth aspect, the embodiments of the present application provide a computer program, the computer program includes instructions, when the computer program is executed by a computer, the computer can execute the low-latency source channel joint decoding provided in the sixth aspect. The process performed by the device.

In a thirteenth aspect, an embodiment of the present application provides a computer storage medium for storing the low-latency source-channel joint coding device provided in the fourth aspect, or another low-latency signal provided in the fifth aspect. The source-channel joint coding device, or the computer software instructions used by the low-latency source-channel joint decoding device provided in the sixth aspect described above, includes a program for executing the program designed in the foregoing aspect.

In a fourteenth aspect, an embodiment of the present application provides a chip system, which includes a processor, and is configured to support a service device to implement the functions involved in the first, second, or third aspects described above. In a possible design, the chip system further includes a memory, and the memory is used to store program instructions and data necessary for the data sending device. The chip system can be composed of chips, or include chips and other discrete devices.

In a fifteenth aspect, an embodiment of the present application provides an electronic device, including the processing chip provided by any one of the foregoing first aspect, second aspect, or third aspect, and a discrete device coupled to the chip.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application, the following will describe the drawings that need to be used in the embodiments of the present application or the background art.

Fig. 1 is a schematic structural diagram of a low-delay source-channel joint coding system provided by an embodiment of the present application.

Fig. 2A is a schematic flowchart of a low-delay source-channel joint coding method provided by an embodiment of the present application.

Fig. 2B is a schematic diagram of an application scenario provided by the present application.

FIG. 2C is a schematic diagram of a first code stream and a second code stream respectively sent through the first channel and the second channel provided by an embodiment of the present application.

FIG. 3A is a schematic flowchart of another low-delay source-channel joint coding and decoding method provided by an embodiment of the present application.

FIG. 3B is a schematic diagram of an application for encoding and decoding a target image provided by an embodiment of the present application.

Fig. 3C is a schematic diagram of another application scenario provided by the present application.

Fig. 4A is a schematic structural diagram of a low-delay source-channel joint coding apparatus provided by an embodiment of the present application.

Fig. 4B is a schematic structural diagram of another low-delay source-channel joint coding apparatus provided by an embodiment of the present application.

Fig. 4C is a schematic structural diagram of a low-delay source-channel joint decoding apparatus provided by an embodiment of the present application.

Fig. 5 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.

detailed description

The embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

The terms "first", "second", "third" and "fourth" in the specification and claims of the application and the drawings are used to distinguish different objects, rather than describing a specific order . In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes unlisted steps or units, or optionally also includes Other steps or units inherent in these processes, methods, products or equipment.

Reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.

The terms "component", "module", "system", etc. used in this specification are used to denote computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, the component may be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program, and/or a computer running on a processor. Through the illustration, both the application running on the computing device and the computing device can be components. One or more components may reside in processes and/or threads of execution, and components may be located on one computer and/or distributed among two or more computers. In addition, these components can be executed from various computer readable media having various data structures stored thereon. The component may be based on, for example, data having one or more data packets (for example, data from two components interacting with another component in a local system, a distributed system, and/or a network, for example, the Internet that interacts with other systems through signals) Signals are communicated through local and/or remote processes.

First of all, some terms in this application are explained to facilitate the understanding of those skilled in the art.

(1) The Joint Photographic Experts Group (JPEG) is an International Standardization Organization (ISO)/International Electrotechnical Commission (IEC) expert group. They develop and maintain computer image files A set of standards for compression algorithms. The compression method of JPEG is usually destructive data compression (lossy compression), which means that the quality of the image will suffer visible damage during the compression process.

(2) Moving Picture Experts Group (MPEG). MPEG is an organization established in 1988 by ISO and IEC to formulate international standards for moving images and voice compression. There are mainly the following five MPEG standards, MPEG-1, MPEG-2, MPEG-4, MPEG-7 and MPEG-21. The MPEG standard video compression coding technology mainly uses the inter-frame compression coding technology with motion compensation to reduce the time redundancy, the discrete cosine transform technology to reduce the spatial redundancy of the image, and the entropy coding is used in the aspect of information representation. Reduced statistical redundancy.

(3) RGB image, R stands for red; G stands for green; B stands for blue. In terms of visual effects, they can be synthesized in different proportions: red, green, and blue. Each pixel of the color image is composed of red and green. The different proportions of, blue, such an image is an RGB image.

(4) Wireless Local Area Network (WLAN), which is a very convenient data transmission system. It uses radio frequency (RF) technology and uses electromagnetic waves to replace the old obstructive twisted-pair copper wire (Coaxial). The formed local area network is connected in the air, so that the wireless local area network can use a simple access structure to allow users to use it to achieve the ideal state of "information portable and convenient to travel the world".

(5) Discrete Cosine Transform (DCT) is a transformation related to Fourier Transform, similar to Discrete Fourier Transform, but using only real numbers. The discrete cosine transform is equivalent to a discrete Fourier transform whose length is about twice as long. This discrete Fourier transform is performed on a real even function (because the Fourier transform of a real even function is still a real even function ), in some deformations, it is necessary to move the input or output position by half a unit. There are two related transforms, one is the discrete sine transform, which is equivalent to the discrete Fourier transform of a real odd function whose length is about twice its length; the other is the improved discrete cosine transform, which is equivalent to the overlapped The data undergoes discrete cosine transform.

(6) Run Length Coding (RLC), also known as "Run Length Coding" or "Run Length Coding", is a statistical coding that belongs to lossless compression coding and is an important coding method for raster data compression. It is valid for binary graphs. The basic principle of run-length coding is to replace consecutive symbols with the same value with a symbol value or string length (continuous symbols form a continuous "run". Run-length coding is named after it), so that the length of the symbol is less than that of the original data. length. Only when the code of each row or each column of data changes, the code and the number of repetitions of the same code are recorded at a time, so as to achieve data compression.

(7) Signal to Interference plus Noise Ratio (SINR) refers to: Signal to Interference plus Noise Ratio (SINR) is the strength of the received useful signal and the received interference signal (noise and interference) The ratio of the intensity.

(8) Signal to Noise Ratio (SNR), also known as signal to noise ratio, refers to the ratio of signal to noise in an electronic device or electronic system. The signal here refers to the electronic signal from the outside of the device that needs to be processed by this device, and the noise refers to the irregular extra signal (or information) that does not exist in the original signal generated after passing through the device. The signal does not change with the change of the original signal.

(9) Packet Error Ratio (PER) is an indicator that measures the accuracy of data packets (data packets) transmission within a specified time. The packet error rate is also an important indicator to measure the quality of data services. Packet error rate = number of error packets in transmission/total number of packets transmitted*100%.

(10) Macroblock is a basic concept in video coding technology. In video coding, a coded image is usually divided into several macroblocks, and a macroblock is composed of a luminance pixel block and two additional chrominance pixel blocks. Generally speaking, the luminance block is a 16x16 pixel block, and the size of the two chrominance image pixel blocks depends on the image sampling format. For example, for a YUV420 sampled image, the chrominance block is an 8x8 pixel block. In each image, several macroblocks are arranged in the form of slices, and the video coding algorithm uses macroblocks as the unit to encode macroblocks one by one and organize them into a continuous video stream. .

(11) Data rate refers to the data flow used by a video file in a unit time, also called bit rate, which is the most important part of picture quality control in video encoding.

(12) Entropy coding is a lossless data compression scheme independent of the specific characteristics of the medium. One of the main types of entropy coding is to create and assign a unique prefix code to each input symbol, and then replace each fixed-length input symbol with the corresponding variable-length prefix-free (prefix-free) The output code word is replaced, so as to achieve the purpose of compressing data. The length of each codeword is approximately proportional to the negative logarithm of the probability. Therefore, the shortest code is used for the most common symbols.

(13) Intraframe compression, also known as spatial compression. When compressing a frame of image, only the data of the current frame is considered without considering the redundant information between adjacent frames, which is actually similar to still image compression. Intra-frame compression generally uses a lossy compression algorithm.

(14) Interframe compression is based on the fact that the two consecutive frames before and after many videos or animations have great correlation, or the characteristics of little change in the information of the two frames before and after. That is, continuous video has redundant information between adjacent frames. According to this feature, compressing the redundancy between adjacent frames can further increase the amount of compression and reduce the compression ratio. Inter-frame compression is also called temporal compression (temporal compression).

(15) Modulation and Coding Scheme (MCS), the configuration of the 802.11n radio frequency is realized through the MCS index value. MCS modulation and coding table is a representation form proposed by 802.11n to characterize the communication rate of WLAN. MCS regards the concerned factors affecting the communication speed as the columns of the table, and the MCS index as the rows to form a speed table.

(16) Down-sampling, a sample sequence is sampled once at intervals of several samples, so that the new sequence obtained is the down-sampling of the original sequence. The sampling rate changes are mainly due to different modules of signal processing may have different sampling rate requirements. Compared with the initial continuous-time signal, down-sampling still has to satisfy the sampling theorem, otherwise such down-sampling will cause aliasing of signal components. Down-sampling is decimation, which is one of the basic contents in multi-rate signal processing.

(17) Image coding. Image coding is also called image compression. It refers to a technology that uses a small number of bits to represent an image or the information contained in an image under the condition that a certain quality (signal-to-noise ratio requirement or subjective evaluation score) is met.

(18) Quality of Service (QoS) is a security mechanism of the network, a technology used to solve problems such as network delay and congestion. When the network is overloaded or congested, QoS can ensure that important traffic is not delayed or discarded, while ensuring the efficient operation of the network.

(19) YUV is a color coding method. Often used in various video processing components. When YUV encodes photos or videos, it takes into account human perception and allows the bandwidth of chroma to be reduced. YUV is a type of compiling true-color color space (color space). Proper nouns such as Y'UV, YUV, YCbCr, YPbPr, etc. can all be called YUV, which overlap with each other. "Y" means brightness (Luminance, Luma), "U" and "V" mean chrominance, density (Chrominance, Chroma)

(20) Channel capacity, also known as channel capacity, refers to the minimum upper bound of the achievable rate when information can be reliably transmitted in a channel. The so-called reliable transmission refers to the transmission of information with an arbitrarily small error rate. According to the noisy channel coding theorem, the channel capacity is the limit information rate of a given channel that can be reached with an arbitrarily small error probability. The unit of channel capacity is bit per second, knight per second, etc.

Secondly, analyze and propose the application scenarios of this application and the specific technical problems to be solved.

Application scenario: In the prior art, the development of wireless video transmission solves the problem of data transmission such as audio and video in the case of inconvenient wiring construction. Therefore, wireless projection compresses the screen information of terminals such as mobile phones/tablets, and then sends them to large-screen devices such as TVs wirelessly, shares videos/pictures with friends through wireless networks, and conducts video conferences through wireless networks. One of the typical application scenarios of image coding and decoding through wireless transmission.

A common wireless video transmission system generally consists of two parts: the transmitter and the receiver. In addition, an appropriate gain antenna can be added to increase the transmission distance according to actual needs. Generally, the transmitter and receiver of wireless video transmission mainly include audio and video encoding, wireless transmission, audio and video decoding and display. In the prior art, JPEG standard coding and MPEG standard coding are two common image coding methods. Both of these image coding methods can be converted into YUV space after the original image undergoes color space transformation, and then undergoes segmentation and discrete cosine transformation. After (Discrete Cosine Transform, DCT), quantization, and entropy coding, a compressed image is obtained. These two coding methods have great advantages in terms of compression factor and image quality. However, because the short-time and time-varying characteristics of the wireless channel will cause rapid changes in channel capacity, the audio and video codec algorithm at the transmitting end needs to be able to quickly track and adapt to this change, so that the delay and image quality of the receiving end, that is, the receiving end, can be kept at an acceptable level. However, traditional audio and video coding technologies, such as JPEG standard coding and MPEG standard coding, which only focus on compression ratio and image quality, cannot quickly make corresponding coding adjustments according to channel capacity. Therefore, it is easy to lose part or all of the code stream information during wireless transmission. Moreover, at the receiving end corresponding to the traditional audio and video coding technology, a complete image can be reconstructed only if it has all the code streams corresponding to a certain frame, and a small amount of code stream loss can cause mosaics, freezes and other phenomena, which is very poor. In addition, based on the principle of network layering, wireless transmission generally does not care about the priority of the source part (that is, the code stream information after the audio and video is encoded), and the transmission is performed without difference. Therefore, the encoding end cannot control whether the key information (such as audio information, stream information of important frames) that may be lost in the wireless channel, and cannot quickly adapt to changes in the wireless channel, and has no control over the information source (such as : Stream information or other data information) is sent hierarchically, and the receiving end may not be able to reconstruct a complete video image due to the loss of important information.

In order to facilitate the understanding of the embodiments of the present application, the following describes one of the low-delay source-channel joint coding system architectures based on the embodiments of the present application. Please refer to FIG. 1. FIG. 1 is a schematic structural diagram of a low-delay source-channel joint coding system provided by an embodiment of the present application.

The architecture of the low-delay source-channel joint coding system in this application may include the coding end 10 and the decoding end 11 in FIG. 1. among them. The encoding end 10 of the low-latency source-channel joint encoding system architecture includes a video source 101, an encoder 102, and a transmitter 103. The decoding end 11 of the low-latency source-channel joint coding system architecture includes a decoder 111 and a display device 112. among them,

The video source 101 is an interface, display memory, storage, etc. used to provide target video. The target video can come from various types of interfaces, such as: High Definition Multimedia Interface (HDMI) interface, digital video (Display Port, DP) ) Interface, video graphics array standard (Video Graphics Array, VGA) interface, etc. For example, when the video source 101 is an HDMI interface, the target video may be sent to the encoder 102, so that the encoder 102 encodes the target image of the target video.

The encoder 102 (encoder) is a device that compiles and converts a signal (such as a bit stream) or data into a signal form that can be used for communication, transmission, and storage. The encoder can convert angular displacement or linear displacement into electrical signals. The former is called a code disc and the latter is called a code ruler. The encoder can be related coding software running on a general-purpose central processing unit (CPU) or a dedicated chip On or on an independent encoding chip, or a part of an independent chip, such as a part of a System-on-a-Chip (SoC) chip of an integrated circuit. For example, when the encoder is an independent encoding chip, the target image can be hierarchically encoded, that is, the target image is down-sampled to obtain the reference frame image and the non-reference frame image for image encoding; and then the first image after the reference frame image encoding is obtained. The code stream, and the second code stream after encoding the non-reference frame image; finally, the first code stream and the second code stream are sent to the decoder 111 through the transmitter 103. Wherein, the coding parameters may also receive coding parameters fed back according to the channel information of the current wireless channel sent through the coding parameter feedback control, so as to make the bit rate of the code stream after image coding smaller than the channel capacity of the current wireless channel. For example, when the encoder is an independent encoding chip, the target image can also be down-sampled to obtain the reference frame image and the non-reference frame image; according to the current wireless channel channel environment, determine the corresponding encoding of the reference frame image and the non-reference frame image Parameters; according to the encoding parameters, image encoding is performed on the reference frame image and the non-reference frame image to obtain the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image.

The transmitter 103 is a device that uses radio waves as a data transmission device using a wirelessly connected local area network, and the transmission distance is generally tens of meters. The transmitter 103 may be a WiFi chip, and may include a driver program, firmware, etc. that are matched with the WiFi chip. For example: when the transmitter 103 is a WiFi chip, the encoder 102 can perform image encoding to obtain the first code stream and then the second code stream is sent through the wireless channel respectively. Among them, the channel resources when the first code stream is sent wirelessly are high. Channel resources when sending the second code stream. It should be noted that the transmitter 103 can also obtain the channel environment of the current wireless channel, such as: channel capacity, channel bandwidth, or signal to interference plus noise ratio, etc.; and then send channel information to the encoder 103 through coding parameter feedback control, so that The encoder 103 adjusts the encoding parameters of the first code stream and the second code stream in real time; the transmitter 103 can also adjust the sending parameters of the first code stream and the second code stream in real time. Among them, the encoding parameter feedback control can be used as an algorithm inside or outside the encoder, which may be implemented by software or chip. The encoding parameter feedback control is mainly used to adjust the image of the encoder according to the channel information of the current wireless channel. Encoding parameters during encoding. It is understandable that the encoder 102 and the transmitter 103 may be devices coupled together to receive target video information sent from the video source 101, and send the target video information to the decoder 111 after image encoding.

Optionally, a decoder 111 is a hardware/software device that can decode and restore digital video and audio data streams into analog video and audio signals. The encoder mainly compresses analog video and audio signals into data-encoded files, while the decoder converts the data-encoded files into analog video and audio signals. The decoder can be related decoding software running on a general-purpose CPU or a dedicated chip, or an independent decoding chip, or a part of an independent chip (such as a SoC chip); for example: when the decoder 111 is an independent decoding chip, the decoder 111 can receive and receive The first code stream sent by the transmitter 103 of the encoding end, the first code stream is the code stream obtained after image encoding of the reference frame image, wherein the reference frame image includes the reference frame image obtained after down-sampling the target image; then decode The device 111 decodes the first code stream to obtain a reference frame according to the reverse flow of the process when the encoder 102 encodes the target image of the target video; finally obtains the target image according to the reference frame. For another example, the decoder 111 may also receive the second code stream, and then decode the second code stream according to the reverse process when the encoder 102 encodes the target image of the target video to obtain a complete or incomplete non-reference frame; finally, the decoder uses The reference frame and the complete non-reference frame are spliced into a complete reconstructed frame, that is, the target image; when the non-reference frame is incomplete, adjacent frame images (such as the reference frame image and the non-reference frame image of an adjacent target image) can be used At least one of) or a set of surrounding pixels is interpolated to obtain a reconstructed frame, and then the obtained reconstructed frame is sent to the display device 112 for display and audio playback. It can also be understood that when the decoder 111 determines whether the non-reference frame is incomplete, it can also determine whether the code stream is incomplete when receiving the code stream, and then determine whether the image corresponding to the code stream is incomplete, which is not limited in this application. .

Optionally, the display device 112 is a display tool that displays certain data on a screen through a specific transmission device and then reflects it to the human eye. Generally, it can refer to a display, a projector, a virtual reality head-mounted display device, and a belt. Devices such as smart terminals with display functions. For example, the display device 112 may be: a virtual reality head-mounted display device (such as virtual reality (VR) glasses, VR goggles, VR helmets), smart phones, notebook computers, tablet devices, projectors, cameras, etc., Or display clients, applications, etc. installed or running on computers, tablet devices, smart phones. For example, when the display device is a projector, the projector may project the target video data information sent by the decoder 111 onto the screen through the projection device, and then play the target video. It is understandable that the decoder 111 and the display device 112 may be devices coupled together to receive the code stream information sent from the transmitter 103, decode the code stream information and display the target video corresponding to the code stream information.

It should be noted that the decoding end 11 in the low-delay source-channel joint coding system architecture provided by the embodiment of the present application is equivalent to the receiving end in the present application.

It should also be noted that the low-latency source-channel joint coding system architecture of FIG. 1 is only a partial exemplary implementation in the embodiments of the present application. The low-latency source-channel joint coding system architecture in the embodiments of the present application includes but not only Limited to the above low-latency source-channel joint coding system architecture.

Based on the above-mentioned technical problem analysis and the above-mentioned low-latency source-channel joint coding system architecture, this application proposes a low-latency source-channel joint coding method, which can avoid the requirement for stream integrity in traditional audio and video coding and decoding algorithms, and also It can quickly track changes in channel capacity, keeping the receiving end delay and image quality at an acceptable level. When the channel capacity is reduced, the phenomenon of mosaic, stuttering, and blurring in the video information is reduced, and the user's visual experience is improved.

Please refer to FIG. 2A. FIG. 2A is a schematic flowchart of a low-latency source-channel joint coding method according to an embodiment of the present application, which can be applied to the low-latency source-channel joint coding system architecture described in FIG. 1 Among them, the encoder 102 and transmitter 103 can be used to support and execute steps S201 to S203 of the method flow shown in FIG. 2A.

Step S201: down-sampling the target image to obtain a reference frame image and a non-reference frame image.

Specifically, the encoder 102 down-samples the target image to obtain the reference frame image and the non-reference frame image. The down-sampling is to sample a sequence of samples at intervals of several samples, so that the new sequence obtained is the down-sampling of the original sequence. . Please refer to FIG. 2B, which is a schematic diagram of an application scenario provided by this application. For example, when a 3-second target video needs to be wirelessly transmitted, the encoder 102 can down-sample the input target video with multiple frames of target images, where the down-sampling horizontal sampling rate is 2:1, and the vertical sampling rate is 2:1. The ratio is 2:1, and it is divided into 4 lower-resolution duplicate images. Among them, one duplicate image can be selected as the reference frame image, and the remaining 3 duplicate images are non-reference frame images. It can be understood that the reference frame image may include one or more images, and the non-reference frame image may also include one or more images, and the number of images included in the reference frame image is less than or equal to the number of images included in the non-reference frame image. When down-sampling the target image, you can choose to sample at half the resolution of the target image, or sample at any ratio of the target image's resolution. The resolution of the copy image after downsampling is smaller than the resolution of the target image. Therefore, the bit rate of the code stream obtained after subsequent compression of the copy image is smaller than that of the code stream obtained when the target image is compressed using the same coding technology. The code rate can then be jittered or reduced to the initial code rate (for example: the initial code rate can be the code rate when the target image is compressed according to the traditional encoding method or the code rate of the preset size) , So that the decoder at the receiving end can still reconstruct the image based on the smaller-resolution copy image. Optionally, the target video includes multiple frames of target images, and the target image is any one of the multiple frames of target images.

Optionally, the encoder 102 down-samples the target image to obtain N duplicate images, the resolution of the N duplicate images is lower than the resolution of the target image, where N is a positive integer greater than 1; Divided into reference frame image and non-reference frame image. It should be noted that after the encoder 102 down-samples the target image to obtain multiple copies of lower resolution images, the multiple copies of lower resolution images can be divided into two types of images with different numbers. , The number of images included in the reference frame image is less than or equal to the number of images included in the non-reference frame image. It can also be understood that the intersection of the reference frame image and the non-reference frame image is an empty set. As shown in FIG. 2B, the four duplicate images obtained after down-sampling, among them, any one duplicate image can be selected as the reference frame image, and the remaining 3 duplicate images are non-reference frame images.

Step S202: Perform image encoding on the reference frame image and the non-reference frame image, respectively, to obtain a first code stream after the reference frame image is coded, and a second code stream after the non-reference frame image is coded.

Specifically, the encoder 102 performs image encoding on the reference frame image and the non-reference frame image after the multiple copy images are divided, to obtain the first code stream after the reference frame image is encoded, and the second code after the non-reference frame image is encoded. flow. For example, the encoder 102 may sequentially perform block, DCT, quantization, run-length coding, entropy coding, packing, marking, etc., on the copy image in the reference frame image to generate the first code stream. The encoder 102 can perform blocking, DCT, quantization, and navigation on the residual of each duplicate image in the non-reference frame image relative to the duplicate image in the reference frame image or directly on each duplicate image in the non-reference frame image. Long coding, entropy coding, packing, marking, etc. generate the second code stream.

Optionally, the encoder 102 may perform intra-frame compression on the reference frame image and the non-reference frame image, respectively, to obtain the first code stream after the reference frame image is coded, and the second code stream after the non-reference frame image is coded. It is understandable that while choosing a simpler down-sampling method to layer the target image and reduce the resolution, the intra-frame compression encoding method used when encoding the layered duplicate image, because intra-frame compression is spatial compression. When compressing a frame of image, only the data of the current frame is considered without considering the redundant information between adjacent frames, and the inter-frame compression should refer to other frame data. The compression rate is large. Considering the image delay, this application implements For example, when the down-sampled copy image is coded, the inter-frame compression is not applicable. Therefore, using the intra-frame compression method can cause the decoder to receive the code stream information based on the smaller resolution copy image when the channel capacity is jittered and drops below the initial bit rate, and can be based on the code The reconstructed image obtained from the stream information. At the same time, when the channel capacity is reduced, the phenomenon of mosaic, stuttering, and blurring in the video information is reduced, and the user's perception and experience are improved.

Optionally, the encoder 102 may also use other layered encoding methods to divide the target image into lower-resolution duplicate images, and the resolution, bit rate, and frame rate among the multiple duplicate images may be different. , Can also be the same. For example: The code stream generated by layered video coding contains multiple sub-streams. The sub-streams are divided into a basic layer and an extended layer. Each layer has a different bit rate, frame rate and resolution. The basic layer has the most basic video quality. , Each subsequent extension layer is a supplement to the previous one. The streaming end can be based on the actual network environment (such as: channel bandwidth, signal and interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle Ratio, bit rate, etc.) to select several sub-streams for decoding.

Optionally, the encoder 102 performs image encoding on the reference frame image and the non-reference frame image to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image encoding the first bit stream. Before the second code stream, the coding parameters corresponding to the reference frame image and the non-reference frame image may also be determined based on the channel environment of the current wireless channel, where the channel environment may include channel capacity, and the coding parameters are used In order to control the corresponding image to generate a corresponding code stream according to a target code rate during image encoding, the target code rate corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity. It should be noted that the encoding parameter determined by the encoder may be an encoding parameter sent by the encoding parameter feedback control. Therefore, when the channel capacity changes, the channel capacity of the current channel can be obtained through the transmitter, and the encoding parameters of the encoder when encoding the duplicate image can be adjusted according to the channel capacity information fed back, so that the encoded bit stream The code rate can be adapted to the current channel capacity, and the target code stream can be sent smoothly, reducing the loss of the code stream due to the decrease of the channel capacity. At the same time, it also reduces the inability to transmit the complete code stream, so that the decoder at the receiving end cannot reconstruct the image obtained based on the missing code stream information, which reduces the probability of video image freezing and blurring, and improves the user's viewing experience.

Step S203: Determine the first channel resource and the second channel resource based on the channel environment of the current wireless channel, and use the first channel resource to send the first code stream and the second channel resource to send the second code stream, respectively.

Specifically, the transmitter 103 may use the first code stream and the second code stream received from the encoder 102 after determining the first channel resource and the second channel resource based on the channel environment of the current wireless channel. The channel resource is used to send the first code stream and the second channel resource is used to send the second code stream. As shown in FIG. 2B, the embodiment of the present application may separately send the first code stream and the second code stream based on the channel environment of the current wireless channel that is fed back. For example: you can set the priority of the first code stream to be higher than the priority of the second code stream, and then when the first code stream and the second code stream are sent separately, according to the channel environment of the current wireless channel, the first code stream is sent The quality of service of the channel when streaming is higher than the quality of service of the channel when transmitting the second code stream. Or, when the channel environment of the current wireless channel is poor, you can send the more important first code stream first, and then send the second code stream with a lower level; or, when the channel environment of the current wireless channel is poor, receive it at the receiving end. After the first code stream is reached or after a period of time after the first code stream is sent, the second code stream with a lower level is sent.

Optionally, the transmitter 103 may send link statistical information such as SINR/SNR/RSSI to the encoding end via the air interface, and the encoding end makes statistics on the error sub-frame rate, collision rate, etc., and then based on the link statistical information and the error sub-frame rate , Collision rate, etc. Adjust sending parameters, such as sending MCS, transmitting power, quantization parameters, etc. That is, when the channel changes, the channel information of the current channel (such as: channel bandwidth, signal-to-interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, bit rate, etc.) is obtained through the transmitter, According to the channel information, the transmission parameters (such as transmission power, modulation and coding strategy information, etc.) when transmitting the first code stream and/or the second code stream are adjusted. For example, the transmitter 103 may be based on the current channel environment , Determine the transmission parameters corresponding to the first code stream and/or the second code stream, wherein the transmission parameters are used to transmit the code stream according to the target modulation and coding strategy information and/or the target transmission power, the The transmission parameters of the first code stream are better than the transmission parameters of the second code stream; the first code stream is transmitted according to the transmission parameters corresponding to the first code stream, and the transmission parameters corresponding to the second code stream are transmitted Sending the second code stream. Wherein, the channel environment includes one or more of channel bandwidth, signal-to-interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate. It can be understood that when the transmitter 103 sends the first code stream and the second code stream, it will respond to the sending request of the first code stream more quickly. In the process of wireless transmission, the importance and priority of the video source are classified, that is, the transmission quality when the first code stream is sent is higher than the transmission quality when the second code stream is sent, or the more important code stream is sent first ( That is, the number of duplicate images is small and the code stream after image encoding), so that when the channel capacity is jittered, and the decoder cannot receive the complete code stream, the reconstructed image can be obtained only based on the duplicate image corresponding to the partial code stream. , Can reduce the phenomenon of mosaic, stutter, blur, etc. in the video information, and improve the user's perception and experience.

Optionally, the transmitter 103 may also determine the first wireless channel and the second wireless channel based on the current channel environment, send the first code stream through the first wireless channel, and send the second code stream The stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel. Wherein, the wireless channel includes a first wireless channel and a second wireless channel. Please refer to FIG. 2C. FIG. 2C is a schematic diagram of a first code stream and a second code stream being sent through a first channel and a second channel, respectively, according to an embodiment of the present application. The transmitter 103 transmits the first code stream through the first wireless channel, and transmits the second code stream through the second wireless channel. When the channel capacity is jittered, the first code stream and the second code stream sent through the wireless channel use different QoS guarantee levels, and the decoder can only receive the first code stream sent through the channel with a high quality of service guarantee mechanism. , And then can reconstruct the image only through the first bit stream, reduce the phenomenon of mosaic, freeze, and blur in the video information, and improve the user's perception and experience. It is understandable that after adjusting the transmission parameters according to the channel information, the transmission parameters of the first code stream are different from the transmission parameters of the second code stream, and the transmission parameters of the first code stream can be compared with the transmission parameters of the second code stream. The machine sends the first stream with a faster sending speed and higher QoS. That is, the first code stream is sent with better channel resources and a higher transmission success rate. The success rate may refer to the probability that the code stream information is completely transmitted to the receiving end. Therefore, in the process of wireless transmission, the video source can be classified, and the more important stream can be sent first, so that when the channel capacity is jittered, or the decoder cannot receive the complete stream, it can only be based on partial stream correspondence. The reconstructed image obtained from the copy image of the image, for example: the first code stream and the second code stream are sent over the wireless channel, and the first code stream is mapped to a higher quality of service (QoS) guarantee mechanism, such as: The first code stream is mapped to the VI service, and the second code stream is mapped to the BE service. In the process of wireless transmission, the first code stream and the second code stream can also be sent through the same wireless channel, but only the first code stream can be sent first to ensure that the receiving end can receive at least the first code stream. For another example: when sending the first code stream and the second code stream, the first code stream can be sent first, and only after the first code stream is received at the receiving end or after a preset period of time after the first code stream is sent, The second code stream is sent to ensure that the receiving end can obtain at least one complete, lower-resolution code stream information in order to reconstruct the image. Sending the first code stream and the second code stream separately through the wireless channel can increase the probability of the receiving end receiving the first code stream even when the receiving end cannot receive the complete code stream, so that the receiving end can only receive the first code stream. You can also get a complete reconstructed image when the first code stream is used.

Optionally, obtain the code stream of each target image corresponding to the reference frame image in the multiple frames of target images included in the target video; obtain the audio information in the target video and mark it as audio information; Each of the included multi-frame images corresponds to the code stream of the reference frame image and the audio information is sent through the first wireless channel. It is understandable that in a video, the audio information is as important as the first code stream information of the multi-frame target image. Therefore, in order to avoid the loss of audio information, the first code stream can be sent together with the audio information through a QoS guarantee level comparison. High first channel transmission, reducing the phenomenon of no sound and freezing in the video information, and improving the user's visual experience.

Optionally, acquiring the code stream of the non-reference frame image corresponding to each target image in the multiple frames of the target image included in the target video; and assigning each frame image of the multiple frames included in the target video to the The code stream of the non-reference frame image is sent through the second wireless channel. After acquiring the code streams of all the target images in the multi-frame target images included in the target video corresponding to the aforementioned non-reference frame images, they are individually mapped to the BE service transmission with a slightly lower quality of service guarantee mechanism. Therefore, even if the code streams corresponding to the non-reference frame images of all target images lose part or even all of the code stream information when the channel changes, the decoder can only be based on the results of all target images received through the channel with a high quality of service guarantee mechanism. The code stream and audio information corresponding to the first copy reconstruct the video, reduce the phenomenon of mosaic, stutter, and blur in the video information, and improve the user's perception and experience.

In the embodiment of this application, the target image can be down-sampled to obtain the reference frame image and the non-reference frame image, and then the reference frame image is image-encoded to obtain the first code stream, and the non-reference frame image is image-encoded to obtain the second code stream. Code stream, finally using the first channel resource to send the first code stream, and using the second channel resource to send the second code stream. Among them, the first channel resource and the second channel resource are determined according to the channel environment of the current wireless channel, and the first channel resource is better than the second channel resource, that is, it can be understood that the reference frame image can be encoded during wireless transmission. After the first code stream and the second code stream after encoding the non-reference frame image, the second code stream is sent hierarchically. For example, you can set the priority of the first code stream to be higher than the priority of the second code stream, and then send the first code stream separately With the second code stream, according to the channel environment of the current wireless channel, the service quality of the channel when the first code stream is sent is higher than the service quality of the channel when the second code stream is sent. Or, when the channel environment of the current wireless channel is poor, you can send the more important first code stream first, and then send the second code stream with a lower level; or, when the channel environment of the current wireless channel is poor, receive it at the receiving end. After the first code stream is reached or after a period of time after the first code stream is sent, the second code stream with a lower level is sent. Based on the channel environment of the current wireless channel, the first code stream and the second code stream are sent separately, so that when the receiving end cannot receive the complete code stream, the probability that the receiving end receives the first code stream can be increased, and the received code stream can be used. The first code stream obtains a complete reconstructed image. Secondly, before encoding the target image, the target image is down-sampled, and after obtaining multiple copies of lower resolution images, the image encoding is performed, which can make the code stream obtained after down-sampling smaller than the direct code stream. The size of the code stream obtained when the target image is image-encoded makes it easier for the decoder at the receiving end to receive the complete code stream. Therefore, after image encoding is performed on the target image through downsampling, the encoded stream information is based on the current wireless channel channel environment and sent through the wireless channel respectively, so that the encoded stream information can still adapt to the real-time changing channel environment , Thereby reducing the probability of information loss during wireless transmission, thereby preventing the receiving end from being unable to reconstruct a complete video image due to information loss, and improving the user's viewing experience.

3A and 3B, FIG. 3A is a schematic flowchart of a low-latency source-channel joint coding and decoding method provided by an embodiment of the present application, and FIG. 3B is a schematic diagram of a target image encoding and decoding application provided by an embodiment of the present application . The method shown in FIG. 3A can be applied to the low-delay source-channel joint coding system architecture described in FIG. 1, where the encoder 102 and the transmitter 103 can be used to support and execute the method flow steps shown in FIG. 3A S301-step S304, the decoder 111 can be used to support and execute the method flow step S305-step S307 shown in FIG. 3A. The method may include the following steps S301-S307.

Step S301: down-sampling the target image to obtain a reference frame image and a non-reference frame image.

Specifically, the foregoing description of step S301 may correspond to the related description of step S201 in FIG. 2A, which will not be repeated here.

Step S302: Determine the coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel.

Specifically, the encoder 102 determines the encoding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel. Please refer to FIG. 3C, which is another application scenario provided by this application. The schematic diagram shows that the coding parameters for graphics coding can be determined according to the channel environment of the current wireless channel. For example, when a 3-second target video needs to be wirelessly transmitted, the encoder 102 can down-sample the input target video with multiple frames of target images, where the down-sampling horizontal sampling rate is 2:1, and the vertical sampling rate is 2:1. The ratio is 2:1, and it is divided into 4 lower-resolution duplicate images. Among them, one duplicate image is selected as the reference frame image, and the remaining 3 duplicate images are non-reference frame images, and then based on the channel environment of the current wireless channel, The encoding parameters corresponding to the reference frame image and the non-reference frame image during image encoding are determined, so as to reduce the size of the code stream after encoding.

Optionally, the transmitter 103 may obtain the channel environment of the current wireless channel, where the channel environment includes the channel capacity. When the channel capacity changes, the channel capacity of the current channel is obtained through the transmitter, and the coding parameters of the encoder when encoding the duplicate image can be adjusted according to the channel capacity, so that the code rate of the coded stream is adapted to Under the current channel capacity, the target code stream can be sent smoothly, reducing the loss of the code stream due to the decrease of the channel capacity.

Step S303: Perform image encoding on the reference frame image and the non-reference frame image respectively according to the encoding parameters to obtain a first code stream after encoding the reference frame image and a second code stream after encoding the non-reference frame image.

Specifically, the encoder 102 adjusts the encoding parameters corresponding to the reference frame image and the non-reference frame image during image encoding according to the channel capacity through the encoding parameter feedback control, and then, according to the encoding parameter determined after adjustment, the reference frame image Image coding is performed separately with the non-reference frame image to obtain a first code stream after the reference frame image is coded, and a second code stream after the non-reference frame image is coded. Among them, the encoding parameters are used to control the corresponding copy to generate the corresponding code stream according to the target code rate during image encoding, so that the code rates of the first code stream and the second code stream are less than or equal to the above-mentioned channel capacity to ensure the code after encoding The bit rate of the stream is adapted to the current channel capacity. Therefore, according to the channel information of the current wireless channel, adjusting the coding parameters during image encoding can make the code rate of the generated code stream lower than the channel capacity of the current wireless channel, and the code stream will not be due to the low channel capacity when the code stream is wirelessly transmitted. The complete bit stream cannot be transmitted, or the channel capacity is jittered, or the channel capacity is reduced to the initial code rate (for example: the initial code rate can be the code rate when the target image is compressed according to the traditional encoding method or the preset code rate) Sometimes, part of the stream information is lost. Therefore, the image coding and decoding algorithm of the embodiment of the present application can quickly track and adapt to changes in the channel environment of the wireless channel, so that the delay and image quality of the receiving end can be maintained at an acceptable level.

Optionally, the code rates of the first code stream and the second code stream may be the same or different. When the code rates of the first code stream and the second code stream are the same, the coding parameters of the image coding can be uniformly adjusted. When the code rate of the code stream is different from that of the second code stream, it is first required to ensure that the code rate of the first code stream is less than or equal to the channel capacity, so as to reduce the probability of code stream loss due to reduced channel capacity.

Step S304: Send the first code stream and the second code stream respectively.

Specifically, the foregoing description of step S304 may correspond to the related description of step S203 in FIG. 2A, which will not be repeated here.

Step S305: Receive the first code stream sent by the encoding end.

Specifically, the decoder 111 receives a first code stream sent by the encoding terminal 102, where the first code stream is a code stream obtained after image encoding of a reference frame image. The embodiment of the present application may receive the first code stream sent by the encoding end during the wireless transmission, and then decode the first code stream to obtain the reference frame image corresponding to the first code stream, and then according to the reference frame image, Obtain the target image. It should be noted that the first code stream is a code stream obtained after image encoding of a reference frame image, where the reference frame image includes one or more images obtained after down-sampling the target image. Therefore, the code received by the encoding end The stream is obtained by down-sampling the target image and then image encoding. Downsampling can divide the target image into multiple copies of lower resolution, and select a part of them as the reference frame image. Moreover, the size of the code stream obtained by down-sampling and re-encoding the target image is smaller than the code stream size obtained when the target image is directly coded, and the code stream after down-sampling may be better adapted to changes in the wireless channel, and is transmitted In the process, the probability of information loss is reduced, and the receiving end cannot reconstruct the complete video image due to the loss of information.

Step S306: After image decoding is performed on the first code stream, a reference frame image corresponding to the first code stream is obtained.

Specifically, the decoder 111 obtains the reference frame image corresponding to the first code stream after image decoding of the first code stream. The encoder 102 performs block, DCT, quantization, run-length encoding, entropy encoding, packing, and marking of the reference frame image to obtain the corresponding first code stream; then the decoder 111 can follow the process completely opposite to that of the encoder 102, namely , Unmarking, unpacking, anti-entropy coding, inverse run length coding, inverse quantization, inverse DCT, decode the first bit stream to get YUV, and then convert YUV to RGB to get the reference frame image. As shown in Figure 3B, the encoder at the encoding end converts the target image from RGB to YUV, and then performs block, DCT, quantization, run-length encoding, entropy encoding, packing, and marking in turn; then it is sent to the receiving end, the decoder of the receiving end Decode the first code stream to obtain YUV according to the reverse process of encoding, and then convert it from YUV to RGB to obtain the reference frame.

Step S307: reconstruct the target image according to the reference frame image.

Specifically, the decoder 111 will reconstruct the target image according to the reference frame image. Wherein, the target image is any one of the multiple frames of target images included in the target video; the reconstructing the target image according to the reference frame image includes: according to the reference frame image and the target image At least one of the reference frame image and the non-reference frame image corresponding to an adjacent frame of target image is reconstructed by using an interpolation algorithm. For example: when performing image encoding, a time packet can be marked for the code stream; when the first code stream is received, the second code stream has not been received within the time period specified by the time packet, and it can be regarded as the second code stream. The code stream is lost. At this time, the reconstructed frame, that is, the target image, can be obtained by interpolating the received reference frame code stream and the previous frame. Therefore, when the decoding end receives the first code stream, it can avoid the inability of the receiving end to reconstruct the complete video image due to the loss of information, reduce the phenomenon of no sound and freeze in the video information, and improve the user's visual experience.

Optionally, the decoder 111 receives a second code stream sent by the encoding end within a preset time period after receiving the first code stream, where the second code stream is a non-reference frame image for performing the processing. The code stream obtained after encoding the image, the non-reference frame image includes the remaining images except the reference frame image obtained after down-sampling the target image; if the second code stream is incomplete, the After image decoding is performed on the second bitstream, the corresponding incomplete non-reference frame image is obtained; the peripheral pixels of the incomplete non-reference frame image are determined according to the incomplete non-reference frame image; the reconstruction is based on the reference frame image The target image includes: reconstructing the target image through an interpolation algorithm according to peripheral pixels of the reference frame image and the incomplete non-reference frame image. It is understandable that within the preset time period of receiving the first code stream, it is determined whether the second code stream is received, and if not, it is determined according to the reference frame image corresponding to the first code stream and the target image of the target image. At least one of the reference frame image and the non-reference frame image corresponding to the next frame of the target image is reconstructed; if the second code stream is received, it can be determined whether the received second code stream is incomplete. If the second bit stream is incomplete, the target image can be obtained by interpolating the incomplete second bit stream and the complete first bit stream, which avoids the inability of the receiving end to reconstruct the complete video image due to partial information loss, and reduces the appearance of the video information. Sounds, freezes, and other phenomena enhance the user’s perception and experience. For example: when performing image encoding, a time packet can be marked for the code stream; when the first code stream is received, the second code stream is received within the time period specified by the time packet, and the second code stream needs to be determined at this time Whether the stream is incomplete, if it is determined that the second code stream is incomplete, the target image can be reconstructed according to the information of the incomplete second code stream by using an interpolation algorithm and the complete first code stream information; or it can be based on the information of the incomplete second code stream At least one of a reference frame image and a non-reference frame image corresponding to the complete first code stream information and an adjacent frame of the target image of the target image is used to reconstruct the target image using an interpolation algorithm.

Optionally, the decoder 111 receives the second code stream sent by the encoding end within a preset time period after receiving the first code stream; if the second code stream is complete, the second code stream is After image decoding of the stream, the corresponding non-reference frame image is obtained; the reconstruction of the target image according to the reference frame image includes: stitching the reference frame image and the non-reference frame image to Reconstruct the target image. For example: when performing image encoding, a time packet can be marked on the code stream; when the first code stream is received, the second code stream is received within the time period specified by the time packet, if the second code stream is determined If the stream is complete, the reference frame image decoded by the first code stream and the non-reference frame image decoded by the second code stream can be directly spliced into the target image.

When implementing the embodiments of the present application, the image coding and decoding algorithms in the embodiments of the present application can quickly track and adapt to changes in the channel environment of the wireless channel, so that the delay and image quality of the receiving end can be maintained at an acceptable level. First of all, the encoding end adjusts the encoding parameters during image encoding according to the channel information of the current wireless channel, so that the code rate of the generated code stream is lower than the channel capacity of the current wireless channel, and the code stream will not be affected by the channel during wireless transmission. The capacity is too low to transmit the complete code stream, or the channel capacity is jittered, or the channel capacity is reduced to the initial code rate (for example: the initial code rate can be the code rate when the target image is compressed according to the traditional encoding method or the preset code rate ) Below, and part of the code stream information is lost. Secondly, the code stream received by the encoding end is obtained by down-sampling the target image and then image encoding. Also, because downsampling can divide the target image into multiple copies of lower resolution, the size of the code stream obtained by down-sampling and re-encoding the target image is smaller than the code stream size obtained when the target image is directly encoded. Furthermore, the down-sampled code stream may better adapt to changes in the wireless channel, and reduce the probability of information loss during transmission, and at the same time avoid the inability of the receiving end to reconstruct a complete video image due to information loss. At the same time, the adjustment of transmission parameters and the allocation of channel resources also reduce the probability of information loss during transmission, avoid the inability of the receiving end to reconstruct the complete video image due to information loss, and reduce the occurrence of mosaic, stuttering, and blurring of video information. Phenomenon to enhance the user’s viewing experience.

The foregoing describes the method of the embodiment of the present application in detail, and the relevant apparatus of the embodiment of the present application is provided below.

Please refer to FIG. 4A. FIG. 4A is a schematic structural diagram of a low-delay source-channel joint coding apparatus provided by an embodiment of the present application. The low-delay source-channel joint coding apparatus 10 may include a first sampling unit 401 and a first coding unit. The unit 402 and the first sending unit 403 may also include: a first encoding parameter unit 404, a first acquiring unit 405, a second sending unit 406, a second acquiring unit 407, and a third sending unit 408. Detailed description of each unit as follows.

The first sampling unit 401 is used for down-sampling the target image to obtain a reference frame image and a non-reference frame image.

The first encoding unit 402 is configured to perform image encoding on the reference frame image and the non-reference frame image to obtain a first code stream after encoding the reference frame image and a second code stream after encoding the non-reference frame image.

The first sending unit 403 is configured to determine the first channel resource and the second channel resource based on the channel environment of the current wireless channel, and respectively use the first channel resource to send the first code stream and the second channel resource to send the second code stream, Among them, the first channel resource is better than the second channel resource.

In a possible implementation manner, the above-mentioned first encoding unit 402 is specifically configured to perform intra-frame compression on each of the above-mentioned reference frame images included in the above-mentioned reference frame image to obtain the above-mentioned first code stream; The residual of the reference frame image is subjected to the intra-frame compression to obtain the corresponding second code stream.

In a possible implementation manner, the foregoing channel environment further includes channel capacity; the device further includes: a first encoding parameter unit 404, configured to perform image encoding on the foregoing reference frame image and the foregoing non-reference frame image separately, Before obtaining the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image, based on the channel capacity, the encoding parameters corresponding to the reference frame image and the non-reference frame image are determined, wherein The above coding parameters are used to control the corresponding image to generate a corresponding code stream according to the target code rate during image encoding. The target code rates corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity .

In a possible implementation manner, the above-mentioned channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate; The unit 403 is specifically configured to determine the transmission parameters corresponding to the first code stream and/or the second code stream respectively based on the current channel environment, where the transmission parameters are used for target modulation and coding strategy information and/or Send the code stream at the target transmission power, the transmission parameters of the first code stream are better than the transmission parameters of the second code stream; send the first code stream according to the transmission parameters corresponding to the first code stream, and according to the second code stream The corresponding sending parameter sends the above-mentioned second code stream.

In a possible implementation manner, the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the first sending unit 403 is specifically configured to: determine the above-mentioned first wireless channel and the above-mentioned second wireless channel based on the current channel environment. Channel, the first code stream is sent through a first wireless channel, and the second code stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned apparatus further includes: a first obtaining unit 405, configured to obtain the multi-frame target images included in the above-mentioned target video Each frame of target image corresponds to the code stream of the aforementioned reference frame image; the audio information in the aforementioned target video is acquired; the second sending unit 406 is configured to correspond each frame of the aforementioned target video to the aforementioned reference frame image. The code stream and the audio information are sent through the first wireless channel.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned apparatus further includes: a second acquisition unit 407, configured to acquire the multi-frame target images included in the above-mentioned target video Each frame of the target image corresponds to the code stream of the aforementioned non-reference frame image; the third sending unit 408 is configured to pass the code stream of each frame image corresponding to the aforementioned non-reference frame image among the multiple frames of images included in the aforementioned target video through the aforementioned second Wireless channel transmission.

It should be noted that the functions of each functional unit in the low-delay source-channel joint coding device 20 described in the embodiment of the present application can be referred to the relevant description in the method embodiment described in FIG. 2A-2C. Here, No longer.

Please refer to FIG. 4B. FIG. 4B is a schematic structural diagram of another low-delay source-channel joint coding apparatus provided by an embodiment of the present application. The low-delay source-channel joint coding apparatus 20 may include a second sampling unit 411 and a second sampling unit 411. The encoding parameter unit 412, the second encoding unit 413, and the fourth sending unit 414 may also include: a third acquiring unit 415, a fifth sending unit 416, a fourth acquiring unit 417, and a sixth sending unit 418. The details of each unit Described as follows.

The second sampling unit 411 is used for down-sampling the target image to obtain a reference frame image and a non-reference frame image.

The second encoding parameter unit 412 is configured to determine the encoding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel.

The second encoding unit 413 is configured to perform image encoding on the reference frame image and the non-reference frame image according to the encoding parameters, to obtain the first code stream after the reference frame image is coded, and the second code stream after the non-reference frame image is coded .

The fourth sending unit 414 is configured to send the first code stream and the second code stream respectively.

In a possible implementation manner, the foregoing encoding parameters are used to control the corresponding image to generate a corresponding code stream according to the target bit rate during image encoding; the foregoing second encoding unit 413 is specifically configured to: include the foregoing reference frame image Each of the aforementioned reference frame images is intra-compressed according to the aforementioned target code rate to obtain the aforementioned first code stream; the residual difference between the aforementioned non-reference frame image and the aforementioned reference frame image is subjected to the aforementioned intra-frame compression according to the aforementioned target code rate to obtain Corresponding to the above second code stream.

In a possible implementation manner, the foregoing channel environment includes one or more of channel bandwidth, signal to interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate; the fourth The sending unit 414 is configured to separately send the first code stream and the second code stream, and is further configured to: determine the transmission parameters corresponding to the first code stream and/or the second code stream according to the channel environment, Wherein, the transmission parameters are used to transmit the code stream according to the target modulation and coding strategy information and/or the target transmission power, and the transmission parameters of the first code stream are better than the transmission parameters of the second code stream.

In a possible implementation manner, the above-mentioned wireless channel includes a first wireless channel and a second wireless channel; the above-mentioned fourth sending unit 414 is specifically configured to: send the above-mentioned first code stream through the first wireless channel, and transmit the above-mentioned first code stream to the The two code streams are sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned apparatus further includes: a third acquisition unit 415, configured to acquire the multi-frame target images included in the above-mentioned target video Each frame of target image corresponds to the code stream of the aforementioned reference frame image; the audio information in the aforementioned target video is obtained; the fifth sending unit 416 is configured to correspond each frame of the aforementioned target video to the aforementioned reference frame image. The code stream and the audio information are sent through the first wireless channel.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned device further includes: a fourth obtaining unit 417, configured to obtain the multi-frame target images included in the above-mentioned target video Each frame of the target image corresponds to the code stream of the aforementioned non-reference frame image; the sixth sending unit 418 is configured to pass the code stream of each frame image corresponding to the aforementioned non-reference frame image in the multi-frame image included in the aforementioned target video through the aforementioned second Wireless channel transmission.

It should be noted that the function of each functional unit in the low-delay source-channel joint coding apparatus 20 described in the embodiment of the present application can be referred to the related description in the method embodiment described in FIG. 3A-FIG. 3C, here No longer.

Please refer to FIG. 4C. FIG. 4C is a schematic structural diagram of a low-delay source-channel joint decoding apparatus provided by an embodiment of the present application. The low-delay source-channel joint decoding apparatus 30 may include a receiving unit 421, a decoding unit 422, and an image. The unit 423 may further include: a fifth acquiring unit 424 and a sixth acquiring unit 425, wherein the detailed description of each unit is as follows.

The receiving unit 421 is configured to receive a first code stream sent by an encoding end, the first code stream is a code stream obtained after image encoding of a reference frame image, wherein the reference frame image includes a code stream obtained by down-sampling the target image Reference frame image.

The decoding unit 422 is configured to decode the image of the first code stream to obtain the reference frame image corresponding to the first code stream.

The image unit 423 is configured to reconstruct the target image according to the reference frame image.

In a possible implementation manner, the above-mentioned target image is any one of the multi-frame target images included in the target video; the above-mentioned image unit 423 is specifically configured to: according to the adjacent one of the reference frame image and the target image At least one of the reference frame image and the non-reference frame image corresponding to the frame target image is reconstructed by using an interpolation algorithm.

In a possible implementation manner, the device further includes: a fifth acquiring unit 424, configured to receive the second code stream sent by the encoding end within a preset time period after receiving the first code stream, where The second code stream is a code stream obtained by encoding a non-reference frame image, and the non-reference frame image includes images obtained after down-sampling the target image except for the reference frame image; if the first The second code stream is incomplete. After the second code stream is image-decoded, the corresponding incomplete non-reference frame image is obtained; the image unit 423 is specifically configured to: according to the peripheral pixels of the above-mentioned reference frame image and the above-mentioned incomplete non-reference frame image, The above-mentioned target image is reconstructed by interpolation algorithm.

In a possible implementation, the device further includes: a sixth obtaining unit 425, configured to, if the second code stream is complete, perform image decoding on the second code stream to obtain the corresponding non-reference frame image The image unit 423 is specifically configured to: stitch the reference frame image and the non-reference frame image to reconstruct the target image.

As shown in Figure 5, Figure 5 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. The terminal device 40 is an intelligent terminal capable of wireless image transmission, such as a projector, a video recorder, a mobile phone, a tablet computer, and a vehicle-mounted terminal. And so on, the device includes at least one processor 501, at least one memory 502, and at least one communication interface 503. In addition, the device may also include general components such as antennas, which will not be described in detail here.

The processor 501 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gata Array, FPGA) circuit, or one or more An integrated circuit used to control the execution of the above program program.

The communication interface 503 is used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), core network, wireless local area networks (WLAN), etc.

The memory 502 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types that can store information and instructions The dynamic storage device can also be electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), CD-ROM (Compact Disc Read-Only Memory, CD-ROM) or other optical disc storage, optical disc storage (Including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program codes in the form of instructions or data structures and can be used by a computer Any other media accessed, but not limited to this. The memory can exist independently and is connected to the processor through a bus. The memory can also be integrated with the processor.

Wherein, the memory 502 is used to store application program codes for executing the above solutions, and the processor 501 controls the execution. The processor 501 is configured to execute application program codes stored in the memory 502.

The code stored in the memory 502 can execute the low-delay source-channel joint coding method provided in Figure 2A, such as: down-sampling the target image to obtain a reference frame image and a non-reference frame image; separate the reference frame image and the non-reference frame image Perform image encoding to obtain the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image; based on the channel environment of the current wireless channel, determine the first channel resource and the second channel resource, and use them respectively The first channel resource is used to send the first code stream and the second channel resource is used to send the second code stream, wherein the first channel resource is better than the second channel resource.

The code stored in the memory 502 can also execute the low-latency source-channel joint coding method provided in FIG. 3A, such as: down-sampling the target image to obtain the reference frame image and the non-reference frame image; according to the current wireless channel channel environment, determine Encoding parameters corresponding to the reference frame image and the non-reference frame image; according to the encoding parameters, image encoding is performed on the reference frame image and the non-reference frame image to obtain the encoded reference frame image The first code stream and the second code stream after encoding the non-reference frame image; the first code stream and the second code stream are sent separately.

It should be noted that the functions of the functional units in the low-delay source-channel joint coding apparatus 30 described in the embodiments of the present application can be referred to the relevant descriptions in the method embodiments described in FIGS. 2A-3C. Here, No longer.

In the above-mentioned embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited by the described sequence of actions. Because according to this application, some steps may be performed in other order or at the same time. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or integrated. To another system, or some features can be ignored, or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.

The units described above as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.

If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , Including several instructions to enable a computer device (which may be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to execute all or part of the steps of the above methods of the various embodiments of the present application. Among them, the aforementioned storage media may include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (Read-Only Memory, abbreviation: ROM) or Random Access Memory (Random Access Memory, abbreviation: RAM), etc. A medium that can store program codes.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions recorded in the embodiments are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

A low-delay source-channel joint coding method, which is characterized in that it comprises:

Down-sampling the target image to obtain the reference frame image and the non-reference frame image;

Performing image encoding on the reference frame image and the non-reference frame image, respectively, to obtain a first code stream after encoding the reference frame image and a second code stream after encoding the non-reference frame image;

Based on the channel environment of the current wireless channel, determine the first channel resource and the second channel resource, and use the first channel resource to send the first code stream and the second channel resource to send the second code stream, respectively, Wherein, the first channel resource is better than the second channel resource.
The method according to claim 1, wherein the image encoding is performed on the reference frame image and the non-reference frame image to obtain the first code stream after the reference frame image is encoded, and the non-reference frame image The second code stream after the coding of the reference frame image includes:

Performing intra-frame compression on each of the reference frame images included in the reference frame image to obtain the first code stream;

The intra-frame compression is performed on the residuals of the non-reference frame image and the reference frame image to obtain the corresponding second code stream.
The method according to claim 1 or 2, wherein the channel environment further includes channel capacity; the image coding is performed on the reference frame image and the non-reference frame image to obtain the reference frame image coding After the first code stream, and before the second code stream after encoding the non-reference frame image, it further includes:

Based on the channel capacity, the coding parameters corresponding to the reference frame image and the non-reference frame image are determined, where the coding parameters are used to control the corresponding image to generate a corresponding code stream according to the target bit rate during image coding. The target code rates corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity.
The method according to claim 3, wherein the channel environment includes one or more of channel bandwidth, signal-to-interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, and bit rate ；

Said determining a first channel resource and a second channel resource based on the current channel environment, and respectively using the first channel resource to send the first code stream and the second channel resource to send the second code stream, include:

Based on the current channel environment, determine the respective transmission parameters corresponding to the first code stream and/or the second code stream, where the transmission parameters are used for target modulation and coding strategy information and/or target transmission Power transmission code stream, the transmission parameter of the first code stream is better than the transmission parameter of the second code stream;

The first code stream is sent according to the sending parameter corresponding to the first code stream, and the second code stream is sent according to the sending parameter corresponding to the second code stream.
The method according to any one of claims 1-4, wherein the wireless channel includes a first wireless channel and a second wireless channel; the first channel resource and the second channel resource are determined based on the current channel environment Respectively using the first channel resource to send the first code stream and using the second channel resource to send the second code stream, including:

Based on the current channel environment, determine the first wireless channel and the second wireless channel, send the first code stream through the first wireless channel, and send the second code stream through the second wireless channel, where , The quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
The method according to claim 5, wherein the target image is any one of multiple target images included in the target video; the method further comprises:

Acquiring a code stream corresponding to the reference frame image of each frame of target image in the multiple frames of target images included in the target video;

Acquiring audio information in the target video;

The code stream and the audio information of each frame image corresponding to the reference frame image in the multi-frame images included in the target video are sent through the first wireless channel.
The method according to claim 5, wherein the target image is any one of multiple target images included in the target video; the method further comprises:

Acquiring a code stream of the non-reference frame image corresponding to each frame of the target image in the multiple frames of the target image included in the target video;

The code stream of each frame image corresponding to the non-reference frame image in the multi-frame images included in the target video is sent through the second wireless channel.
A low-delay source-channel joint coding method, which is characterized in that it comprises:

Down-sampling the target image to obtain the reference frame image and the non-reference frame image;

Determine the encoding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel;

According to the encoding parameters, image encoding is performed on the reference frame image and the non-reference frame image to obtain the first code stream after the reference frame image is encoded, and the second code stream after the non-reference frame image is encoded. Code stream

Sending the first code stream and the second code stream respectively.
8. The method according to claim 8, wherein the encoding parameter is used to control the corresponding image to generate the corresponding code stream according to the target bit rate when the image is encoded;

According to the encoding parameters, image encoding is performed on the reference frame image and the non-reference frame image to obtain a first code stream after the reference frame image is encoded, and the non-reference frame image is encoded The second stream includes:

Performing intra-frame compression on each of the reference frame images included in the reference frame image according to the target bit rate to obtain the first bit stream;

The residual difference between the non-reference frame image and the reference frame image is intra-compressed according to the target bit rate to obtain the corresponding second code stream.
The method according to claim 8 or 9, wherein the channel environment includes one of channel bandwidth, signal-to-interference plus noise ratio, signal-to-noise ratio, received signal strength indicator, duty cycle, bit rate, or Multiple

Before the sending the first code stream and the second code stream separately, the method further includes:

According to the channel environment, determine the respective transmission parameters corresponding to the first code stream and/or the second code stream, wherein the transmission parameters are used to transmit according to target modulation and coding strategy information and/or target transmission power Code stream, the sending parameter of the first code stream is better than the sending parameter of the second code stream.
The method according to any one of claims 8-10, wherein the wireless channel includes a first wireless channel and a second wireless channel; the sending the first code stream and the second code stream respectively, include:

The first code stream is sent through a first wireless channel, and the second code stream is sent through a second wireless channel, wherein the quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
The method according to claim 11, wherein the target image is any one of multiple target images included in the target video; the method further comprises:

Acquiring a code stream corresponding to the reference frame image of each frame of target image in the multiple frames of target images included in the target video;

Acquiring audio information in the target video;

The code stream and the audio information of each frame image corresponding to the reference frame image in the multi-frame images included in the target video are sent through the first wireless channel.
The method according to claim 11, wherein the target image is any one of multiple target images included in the target video; the method further comprises:

Acquiring a code stream of the non-reference frame image corresponding to each frame of the target image in the multiple frames of the target image included in the target video;

The code stream of each frame image corresponding to the non-reference frame image in the multi-frame images included in the target video is sent through the second wireless channel.
A low-delay source-channel joint decoding method, which is characterized in that it comprises:

Receive a first code stream sent by an encoding end, where the first code stream is a code stream obtained after image encoding of a reference frame image, wherein the reference frame image includes one or more images obtained after down-sampling the target image image;

Performing image decoding on the first code stream to obtain the reference frame image corresponding to the first code stream;

According to the reference frame image, the target image is reconstructed.
The method according to claim 14, wherein the target image is any one of multiple frames of target images included in the target video;

The reconstructing the target image according to the reference frame image includes:

According to at least one of a reference frame image and a non-reference frame image corresponding to a frame of target image adjacent to the reference frame image and the target image, the target image is reconstructed by an interpolation algorithm.
The method according to claim 14, wherein the method further comprises:

Within a preset period of time after receiving the first code stream, receive a second code stream sent by the encoding end, where the second code stream is a code obtained after the image encoding is performed on a non-reference frame image Stream, the non-reference frame image includes the remaining images except for the reference frame image obtained after down-sampling the target image;

If the second code stream is incomplete, after performing image decoding on the second code stream, a corresponding incomplete non-reference frame image is obtained;

Determine the peripheral pixels of the incomplete non-reference frame image according to the incomplete non-reference frame image;

The reconstructing the target image according to the reference frame image includes:

According to the reference frame image and the surrounding pixels, the target image is reconstructed through an interpolation algorithm.
The method according to claim 16, wherein the method further comprises:

If the second code stream is complete, after image decoding is performed on the second code stream, the corresponding non-reference frame image is obtained;

The reconstructing the target image according to the reference frame image includes:

Stitching the reference frame image and the non-reference frame image to reconstruct the target image.
A low-delay source-channel joint coding device, which is characterized by comprising: an encoder and a transmitter, wherein:

The encoder is used for:

Down-sampling the target image to obtain the reference frame image and the non-reference frame image;

Performing image encoding on the reference frame image and the non-reference frame image, respectively, to obtain a first code stream after encoding the reference frame image and a second code stream after encoding the non-reference frame image;

The transmitter is used for:

Based on the channel environment of the current wireless channel, determine the first channel resource and the second channel resource, and use the first channel resource to send the first code stream and the second channel resource to send the second code stream, respectively, Wherein, the first channel resource is better than the second channel resource.
A low-delay source-channel joint coding device, which is characterized by comprising: an encoder and a transmitter, wherein:

The encoder is used for:

Down-sampling the target image to obtain the reference frame image and the non-reference frame image;

Determine the encoding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel;

According to the encoding parameters, image encoding is performed on the reference frame image and the non-reference frame image to obtain the first code stream after the reference frame image is encoded, and the second code stream after the non-reference frame image is encoded. Code stream

The transmitter is configured to send the first code stream and the second code stream respectively.
A computer storage medium, characterized in that the computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1-7 or 8-13 is realized.
A computer program, characterized in that the computer program includes instructions, which when the computer program is executed by a computer, cause the computer to execute the method according to any one of claims 1-7 or 8-13.
A low-latency source-channel joint coding system, which is characterized by comprising an encoding end and a decoding end, wherein the encoding end is used to execute the method according to claim 1-7 or 8-13, and the decoding end For performing the method according to claims 14-17.