CN114424552A

CN114424552A - Low-delay source-channel joint coding method and related equipment

Info

Publication number: CN114424552A
Application number: CN201980100614.8A
Authority: CN
Inventors: 李卫华; 高林; 郭湛; 张亚凡
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-09-29
Filing date: 2019-09-29
Publication date: 2022-04-29
Also published as: WO2021056575A1

Abstract

The embodiment of the application discloses a low-delay source-channel joint coding method and related equipment, wherein the low-delay source-channel joint coding method comprises the following steps: firstly, down-sampling a target image to obtain a reference frame image and a non-reference frame image; respectively carrying out image coding on the reference frame image and the non-reference frame image to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded; and based on the channel environment of the current wireless channel, transmitting the first code stream by using the first channel resource and transmitting the second code stream by using the second channel resource, wherein the first channel resource is superior to the second channel resource. By implementing the embodiment of the application, the phenomena of mosaic, blockage, blurring and the like of video information can be reduced when the channel capacity is reduced, and the impression experience of a user is improved.

Description

Low-delay source-channel joint coding method and related equipment

Technical Field

The present application relates to the field of wireless transmission technologies, and in particular, to a low-latency source-channel joint coding method and related devices.

Background

With the development of wireless transmission technology, it is possible to use a wireless network to carry high-definition video, but at the same time, the wireless network also puts new requirements on the image coding and decoding technology, because the short-time changing characteristic of a wireless channel can cause the channel capacity to change rapidly during wireless transmission, so the image coding and decoding algorithm needs to be able to track and adapt to the change rapidly, and the delay and the image quality of a receiving end are kept at an acceptable level.

However, for example, Joint Photographic Experts Group (JPEG) standard codec and Moving Picture Experts Group (MPEG) standard codec, these traditional image coding and decoding algorithms focusing on compression factor and image quality cannot make corresponding coding adjustment according to channel capacity in time and quickly. Therefore, when the channel capacity is reduced to be lower than the source code rate, the video information compressed by the traditional image coding and decoding algorithm is easy to lose related code stream information in the wireless transmission process, so that the decoded whole macro block cannot be reconstructed at the receiving end, and the phenomena of mosaic, blockage, blurring and the like of videos transmitted in the past can be caused, and the appearance is poor.

Therefore, how to reduce information loss in the transmission process and avoid the problem that the receiving end cannot reconstruct a complete video image due to the information loss is an urgent problem to be solved.

Disclosure of Invention

The embodiment of the application provides a low-delay information source and channel joint coding method and related equipment, and by optimizing the coding or transmission mode of an image, information loss can be reduced in the video transmission process, so that a receiving end cannot play a clear and complete video, and the watching experience of a user is improved.

In a first aspect, an embodiment of the present application provides a low-delay source-channel joint coding method, which may include: down-sampling the target image to obtain a reference frame image and a non-reference frame image; respectively carrying out image coding on the reference frame image and the non-reference frame image to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded; determining a first channel resource and a second channel resource based on the channel environment of the current wireless channel, and respectively transmitting a first code stream by using the first channel resource and transmitting a second code stream by using the second channel resource, wherein the first channel resource is superior to the second channel resource.

The video wireless transmission technology mainly comprises video coding, wireless transmission, video decoding, display and other main links. By the method of the first aspect, during video encoding and wireless transmission, the encoding end may perform downsampling on a target image to obtain a reference frame image and a non-reference frame image, then perform image encoding on the reference frame image to obtain a first code stream, perform image encoding on the non-reference frame image to obtain a second code stream, and finally transmit the first code stream by using the first channel resource and transmit the second code stream by using the second channel resource. The first channel resource and the second channel resource are determined according to the channel environment of the current wireless channel, and the first channel resource is better than the second channel resource, that is, in the process of wireless transmission, the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image may be sent in a hierarchical manner, for example: the priority of the first code stream can be set to be higher than that of the second code stream, and then when the first code stream and the second code stream are respectively sent, the service quality of a channel when the first code stream is sent is higher than that of a channel when the second code stream is sent according to the channel environment of the current wireless channel. Or when the channel environment of the current wireless channel is poor, the coding end can preferentially send a first code stream which is more important, and then send a second code stream which is lower in level; or when the channel environment of the current wireless channel is poor, after the receiving end receives the first code stream or after a period of time after the first code stream is sent, the second code stream with a lower level is sent. Based on the channel environment of the current wireless channel, the coding end respectively sends the first code stream and the second code stream, so that when the receiving end cannot receive the complete code stream, the probability that the receiving end receives the first code stream is improved, and the received first code stream is utilized to obtain a complete reconstructed image. Secondly, before the target image is subjected to image coding, downsampling is carried out on the target image, and after a plurality of duplicate images with lower resolution are obtained, image coding is carried out, so that the code stream obtained after downsampling is smaller than the code stream obtained when the target image is directly subjected to image coding, and a decoder at a receiving end can receive the complete code stream more easily. Therefore, after the target image is subjected to image coding by the coding end through down sampling, the coded code stream information is sent through the wireless channels respectively based on the channel environment of the current wireless channel, so that the coded code stream information can still adapt to the channel environment which changes in real time, the probability of information loss in the wireless transmission process is further reduced, the situation that a receiving end cannot reconstruct a complete video image due to information loss is further avoided, and the watching experience of a user is improved.

In a possible implementation manner, the performing image coding on the reference frame image and the non-reference frame image respectively to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded includes: performing intra-frame compression on each reference frame image included in the reference frame images to obtain the first code stream; and performing intra-frame compression on the residual error between the non-reference frame image and the reference frame image to obtain the corresponding second code stream. When the method is implemented, an encoding end can select simpler down-sampling to layer a target image and reduce resolution, and meanwhile, an intra-frame compression encoding method used for image encoding of the layered duplicate image is adopted. Moreover, by using the intra-frame compression method, when the channel capacity is jittered and is reduced to be lower than the initial code rate, a decoder at a receiving end can receive the code stream information coded based on the copy image with the smaller resolution, and a reconstructed image can be obtained according to the code stream information. Meanwhile, when the channel capacity is reduced, the probability of information loss in the transmission process is reduced, the phenomena of mosaic, blocking, blurring and the like of video image information are avoided, and the impression experience of a user is improved.

In one possible implementation, the channel environment further includes a channel capacity; before the above-mentioned performing image coding on the reference frame image and the non-reference frame image respectively to obtain the first code stream after the reference frame image is coded and the second code stream after the non-reference frame image is coded, the method further includes: and determining coding parameters corresponding to the reference frame image and the non-reference frame image based on the channel capacity, wherein the coding parameters are used for controlling the corresponding images to generate corresponding code streams according to target code rates when image coding is performed, and the target code rates respectively corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity. When the method and the device are implemented, the coding end can obtain the channel capacity of the current channel through the transmitter when the channel capacity changes, and adjust the coding parameters of the coder during image coding of the duplicate image according to the fed back channel capacity information, so that the code rate of the coded code stream can be smoothly sent under the condition of adapting to the current channel capacity, and the condition that the code stream is lost due to the reduction of the channel capacity is reduced. Meanwhile, the method also reduces the problem that a decoder at the receiving end cannot obtain a reconstructed image according to the missing code stream information because the complete code stream cannot be transmitted, reduces the jam and blur probability of the video image, and improves the watching experience of a user.

In a possible implementation manner, the channel environment includes one or more of a bandwidth, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty ratio, and a bit rate of a channel; the determining a first channel resource and a second channel resource based on the current channel environment, and respectively transmitting the first code stream using the first channel resource and the second code stream using the second channel resource, includes: determining sending parameters respectively corresponding to the first code stream and/or the second code stream based on the current channel environment, wherein the sending parameters are used for sending the code streams according to target modulation and coding strategy information and/or target transmitting power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream; and sending the first code stream according to the sending parameters corresponding to the first code stream, and sending the second code stream according to the sending parameters corresponding to the second code stream. When the embodiment of the application is implemented, the encoding end may obtain, through the transmitter, a channel environment (for example, a bandwidth of a channel, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty ratio, a bit rate, and the like) of a current wireless channel when the channel changes, and adjust, according to the channel environment, transmission parameters (for example, transmission power, modulation and coding policy information, and the like) when the first code stream and/or the second code stream are transmitted, it can be understood that, after the transmission parameters are adjusted according to the channel information, the transmission parameters of the first code stream are different from the transmission parameters of the second code stream, and the transmission parameters of the first code stream can enable the transmitter to transmit the first code stream at a faster transmission speed and with a higher Quality of Service (QoS) compared with the transmission parameters of the second code stream. Therefore, in the process of wireless transmission, the first code stream is sent by using the channel resource which is superior to the channel resource of the second code stream, that is, the transmission quality when the first code stream is sent is higher than the transmission quality when the second code stream is sent, or a more important code stream (that is, a code stream which is subjected to image coding and has a small quantity of duplicate images) is sent preferentially, so that when the decoder at a receiving end cannot receive a complete code stream due to jitter of channel capacity and the like, due to the characteristic of intra-frame compression, the situation that the receiving end cannot reconstruct a complete video image due to information loss can be avoided, the decoder obtains a reconstructed image only based on a part of the code stream which is received preferentially, the phenomena of mosaic, blockage, blurring and the like of video information can be reduced, and the viewing experience of a user can be improved.

In one possible implementation, the wireless channel includes a first wireless channel and a second wireless channel; the determining a first channel resource and a second channel resource based on the current channel environment, and respectively transmitting the first code stream using the first channel resource and the second code stream using the second channel resource, includes: and determining the first wireless channel and the second wireless channel based on the current channel environment, transmitting the first code stream through the first wireless channel, and transmitting the second code stream through the second wireless channel, wherein the service quality guarantee mechanism of the first wireless channel is higher than that of the second wireless channel. When the embodiment of the application is implemented, the encoding end can grade the video source in the wireless transmission process, and the first code stream is sent by utilizing the channel resource which is superior to the channel resource of the second code stream. For example: the first bitstream is mapped to a Video (VI) service, and the second bitstream is mapped to a Best Effort (BE) service. Therefore, when jitter occurs in channel capacity and the like, different QoS guarantee levels are respectively used for the first code stream and the second code stream sent through the wireless channel, the probability of receiving the first code stream by the receiving end is improved, so that the decoder of the receiving end can still reconstruct an image only through the first code stream when only receiving the first code stream sent through a channel with a high service quality guarantee mechanism, the phenomenon that the receiving end cannot reconstruct a complete video image due to information loss is avoided, the phenomena of mosaic, blockage, blurring and the like of video information are reduced, and the viewing experience of a user is improved.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the method further comprises the following steps: acquiring a code stream of each frame of target image corresponding to the reference frame image in a plurality of frames of target images included in the target video; acquiring audio information in the target video; and transmitting the code stream and the audio information of each frame image in the multi-frame images included in the target video corresponding to the reference frame image through the first wireless channel. When the embodiment of the application is implemented, the encoding end may obtain that code streams of all target images in a plurality of target images included in a target video, which correspond to the reference frame image, are mapped to a VI service with a high service quality guarantee mechanism together with audio information to be transmitted, and it can be understood that the audio information may be audio information after being encoded by voice. When the channel changes, the decoder of the receiving end can reconstruct the video only based on the code stream and the audio information corresponding to all the target images received by the channel with high service quality guarantee mechanism, so that the probability of receiving the first code stream by the receiving end is improved, the phenomenon that the receiving end cannot reconstruct the complete video image due to information loss is avoided, the phenomena of no sound, blocking and the like of the video information are reduced, and the impression experience of a user is improved.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the method further comprises the following steps: acquiring a code stream of each frame of target image corresponding to the non-reference frame image in a plurality of frames of target images included in the target video; and transmitting a code stream of each frame image in the multi-frame images included in the target video corresponding to the non-reference frame image through the second wireless channel. When the embodiment of the application is implemented, the encoding end can map the code stream of the non-reference frame image corresponding to all the target images in the multi-frame target images included in the target video to the BE service with a slightly lower service quality guarantee mechanism to transmit after acquiring the code stream. Therefore, even if the code stream corresponding to the non-reference frame image of all the target images loses part of or even all code stream information when the channel changes, the decoding end can reconstruct the video only based on the code stream corresponding to the first copy of all the target images and the audio information received through the channel with high service quality guarantee mechanism, thereby avoiding that the receiving end can not reconstruct the complete video image due to information loss, reducing the phenomena of mosaic, blockage, blur and the like of the video information, and improving the viewing experience of users.

In a second aspect, an embodiment of the present application provides another low-delay source-channel joint coding method, which may include: down-sampling the target image to obtain a reference frame image and a non-reference frame image; determining coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel; respectively carrying out image coding on the reference frame image and the non-reference frame image according to the coding parameters to obtain a first code stream obtained after coding the reference frame image and a second code stream obtained after coding the non-reference frame image; and respectively transmitting the first code stream and the second code stream.

By implementing the embodiment of the application, the encoding end can firstly carry out down-sampling on the target image to obtain a reference frame image and a non-reference frame image; then according to the channel environment of the current wireless channel, determining coding parameters corresponding to the reference frame image and the non-reference frame image, and then performing image coding on the reference frame image and the non-reference frame image by using the coding parameters, so that the code rates of the first code stream and the second code stream are lower than the channel capacity of the current wireless channel; and finally, respectively sending the first code stream and the second code stream out through wireless channels. The code parameters during image coding are adjusted according to the channel information of the current wireless channel, so that the code rate of the generated code stream can be lower than the channel capacity of the current wireless channel, and further, when the code stream is transmitted wirelessly, the code stream cannot lose part of the code stream information when the channel capacity is too low to transmit a complete code stream or the channel capacity is jittered or the channel capacity is reduced to be lower than an initial code rate (for example, the initial code rate can be the code rate or a preset code rate when a target image is compressed according to a traditional coding method). Therefore, the image coding and decoding algorithm of the embodiment of the application can quickly track and adapt to the change of the channel environment of the wireless channel, and the delay and the image quality of the receiving end are kept at an acceptable level. Secondly, before the encoding end performs image encoding on the target image, down-sampling is performed on the target image, and after a plurality of duplicate images with lower resolution are obtained, image encoding is performed, so that the code stream obtained after down-sampling is smaller than the code stream obtained when the target image is directly subjected to image encoding, and a decoder at a receiving end can receive the complete code stream more easily. Therefore, the code stream information obtained by the encoding end by image encoding according to the channel information of the current wireless channel can adapt to the channel environment changing in real time, the information loss probability in the transmission process is further reduced, the situation that the receiving end cannot reconstruct the complete video image due to the information loss is avoided, and the watching experience of a user is improved.

In a possible implementation manner, the coding parameters are used for controlling the corresponding image to generate a corresponding code stream according to a target code rate when the image coding is performed; the above-mentioned image coding, according to the above-mentioned coding parameter, the above-mentioned reference frame image and the above-mentioned non-reference frame image, obtain the first code stream after the above-mentioned reference frame image coding, and the second code stream after the above-mentioned non-reference frame image coding, including: performing intraframe compression on each reference frame image included in the reference frame images according to the target code rate to obtain the first code stream; and performing intra-frame compression on the residual errors of all the reference frame images included in the non-reference frame images and each non-reference frame image included in the reference frame images according to the target code rate to obtain the corresponding second code stream. When the method is implemented, an encoding end can select simpler and more convenient down-sampling to layer a target image and reduce the resolution, and meanwhile, an intra-frame compression encoding method used for image encoding of the layered duplicate image is adopted. Meanwhile, when the channel capacity is reduced, the probability of information loss in the transmission process is reduced, the phenomena of mosaic, blocking, blurring and the like of video image information are avoided, and the impression experience of a user is improved.

In a possible implementation, the channel environment includes one or more of a bandwidth of a channel, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty cycle, and a bit rate; before the sending the first code stream and the second code stream respectively, the method further includes: and determining sending parameters respectively corresponding to the first code stream and/or the second code stream according to the channel environment, wherein the sending parameters are used for sending the code streams according to target modulation and coding strategy information and/or target transmitting power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream. When the embodiment of the application is implemented, the encoding end can obtain the channel environment of the current wireless channel through the transmitter when the channel changes, and adjust the sending parameters (such as the sending power, the modulation and coding strategy information and the like) when the first code stream and/or the second code stream are sent according to the channel environment, it can be understood that after the sending parameters are adjusted according to the channel environment, the sending parameters of the first code stream are different from the sending parameters of the second code stream, and the sending parameters of the first code stream can enable the transmitter to send the first code stream at a faster sending speed and a higher Quality of Service (QoS) compared with the sending parameters of the second code stream. Therefore, in the process of wireless transmission, the encoding end transmits the first code stream by using the channel resource which is superior to the channel resource of the second code stream, that is, the transmission quality when the first code stream is transmitted is higher than that when the second code stream is transmitted, or preferentially transmits the more important code stream (that is, the code stream after image encoding with less copy images), so that when the channel capacity and the like are jittered, a decoder at the receiving end can still receive the complete code stream, the information loss is avoided, the phenomena of mosaic, blockage, blurring and the like of video information are reduced, and the viewing experience of a user is improved.

In one possible implementation, the wireless channel includes a first wireless channel and a second wireless channel; the transmitting the first code stream and the second code stream respectively includes: and sending the first code stream through a first wireless channel, and sending the second code stream through a second wireless channel, wherein the service quality guarantee mechanism of the first wireless channel is higher than that of the second wireless channel. When the embodiment of the application is implemented, the encoding end can grade the video source in the wireless transmission process, and the first code stream is sent by utilizing the channel resource which is superior to the channel resource of the second code stream. Therefore, when jitter occurs in channel capacity and the like, the first code stream and the second code stream sent by the coding end through the wireless channel respectively use different QoS guarantee levels, the probability of receiving the first code stream by the receiving end is improved, a decoder of the receiving end can still reconstruct an image only through the first code stream when only receiving the first code stream sent by the channel with high service quality guarantee mechanism, the phenomenon that the receiving end cannot reconstruct a complete video image due to information loss is avoided, mosaic, blocking, blurring and the like of video information are reduced, and the viewing experience of a user is improved.

In a third aspect, an embodiment of the present application provides a low-delay source-channel joint decoding method, which may include: receiving a first code stream sent by an encoding end, wherein the first code stream is obtained after image encoding is carried out on a reference frame image, and the reference frame image comprises one or more images obtained after down-sampling is carried out on a target image; after the first code stream is subjected to image decoding, a reference frame image corresponding to the first code stream is obtained; and reconstructing the target image according to the reference frame image.

In the process of wireless transmission, by the decoding method provided by the third aspect of the present application, a decoding end (i.e., a receiving end) can receive a first code stream sent by an encoding end, further perform image decoding on the first code stream, obtain a reference frame image corresponding to the first code stream, and then reconstruct a target image according to the reference frame image. It should be noted that the first code stream is a code stream obtained by performing image coding on a reference frame image, where the reference frame image includes one or more images obtained by down-sampling a target image, and therefore, the code stream received by the receiving end is obtained by down-sampling the target image and then performing image coding. The down-sampling can sample the target image into a plurality of duplicate images having a lower resolution, and some of the duplicate images can be used as reference frame images. The size of the code stream obtained by down-sampling and re-encoding the target image is smaller than that obtained when the target image is directly subjected to image encoding, so that the code stream obtained by down-sampling the target image can better adapt to the change of a wireless channel, the probability of information loss is reduced in the transmission process, and the situation that a receiving end cannot reconstruct a complete video image due to information loss is avoided.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the reconstructing the target image from the reference frame image includes: and reconstructing the target image by an interpolation algorithm based on at least one of a reference frame image and a non-reference frame image corresponding to one frame of target image in which the reference frame image is adjacent to the target image. By implementing the embodiment of the application, the receiving end can reconstruct the reference frame image corresponding to the first code stream and the reference frame image corresponding to the previous frame target image of the target image corresponding to the first code stream into the target image by utilizing an interpolation algorithm when the receiving end only receives the first code stream, so that when the receiving end receives the first code stream, the receiving end can avoid the situation that the receiving end cannot reconstruct a complete video image due to information loss, the phenomena of no sound, blockage and the like of video information are reduced, and the impression experience of a user is improved.

In one possible implementation, the method further includes: receiving a second code stream sent by the encoding end in a preset time period after receiving the first code stream, wherein the second code stream is a code stream obtained after the image encoding is performed on a non-reference frame image, and the non-reference frame image comprises a residual image obtained after down-sampling the target image except the reference frame image; if the second code stream is incomplete, performing image decoding on the second code stream to obtain a corresponding incomplete non-reference frame image; determining peripheral pixels of the incomplete non-reference frame image according to the incomplete non-reference frame image; the reconstructing the target image from the reference frame image includes: and reconstructing the target image through an interpolation algorithm according to the peripheral pixels of the reference frame image and the incomplete non-reference frame image. By implementing the embodiment of the application, a receiving end can determine whether a second code stream is received or not within a preset time period when the first code stream is received, and if not, the target image is reconstructed according to the first code stream and at least one of the reference frame image and the non-reference frame image corresponding to the adjacent frame of the target image; if the second code stream is received, whether the received second code stream is incomplete or not is determined, if the received second code stream is incomplete, the incomplete second code stream and the complete first code stream can be used for interpolation to obtain a target image, the situation that a receiving end cannot reconstruct a complete video image due to partial information loss is avoided, the phenomena of no sound, blocking and the like of video information are reduced, and the viewing experience of a user is improved.

In one possible implementation, the method further includes: if the second code stream is determined to be complete, carrying out image decoding on the second code stream to obtain the corresponding non-reference frame image; the reconstructing the target image from the reference frame image includes: and splicing the reference frame image and the non-reference frame image to reconstruct the target image. By implementing the embodiment of the application, the receiving end can directly decode the image and then splice the image into the image when receiving the complete first code stream and the complete second code stream, so that the change of a wireless channel is adapted, the information is completely stored in the transmission process, the situation that the receiving end cannot reconstruct the complete video image due to information loss is avoided, and the watching experience of a user is improved.

In a fourth aspect, an embodiment of the present application provides a low-delay source-channel joint coding apparatus, including: an encoder and a transmitter, wherein the encoder is configured to: down-sampling the target image to obtain a reference frame image and a non-reference frame image; respectively carrying out image coding on the reference frame image and the non-reference frame image to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded; a transmitter to: determining a first channel resource and a second channel resource based on a channel environment of a current wireless channel, and respectively transmitting the first code stream by using the first channel resource and transmitting the second code stream by using the second channel resource, wherein the first channel resource is superior to the second channel resource.

In a possible implementation manner, when the encoder is configured to perform image encoding on the reference frame image and the non-reference frame image respectively to obtain a first code stream after the reference frame image is encoded and a second code stream after the non-reference frame image is encoded, the encoder is specifically configured to: performing intra-frame compression on each reference frame image included in the reference frame images to obtain the first code stream; and performing intra-frame compression on the residual error between the non-reference frame image and the reference frame image to obtain the corresponding second code stream.

In one possible implementation, the channel environment further includes a channel capacity; the transmitter, before the performing image coding on the reference frame image and the non-reference frame image respectively to obtain the first code stream after the reference frame image is coded and the second code stream after the non-reference frame image is coded, is further configured to: and determining coding parameters corresponding to the reference frame image and the non-reference frame image based on the channel capacity, wherein the coding parameters are used for controlling the corresponding images to generate corresponding code streams according to target code rates when image coding is performed, and the target code rates respectively corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity.

In one possible implementation, the channel environment includes one or more of a bandwidth of a channel, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty cycle, and a bit rate; the transmitter, when configured to determine a first channel resource and a second channel resource based on the current channel environment, and respectively transmit the first code stream using the first channel resource and the second code stream using the second channel resource, is specifically configured to: determining sending parameters respectively corresponding to the first code stream and/or the second code stream based on the current channel environment, wherein the sending parameters are used for sending the code streams according to target modulation and coding strategy information and/or target transmitting power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream; and sending the first code stream according to the sending parameters corresponding to the first code stream, and sending the second code stream according to the sending parameters corresponding to the second code stream.

In one possible implementation, the wireless channel includes a first wireless channel and a second wireless channel; the transmitter, when configured to determine a first channel resource and a second channel resource based on the current channel environment, and respectively transmit the first code stream using the first channel resource and the second code stream using the second channel resource, is specifically configured to: and determining the first wireless channel and the second wireless channel based on the current channel environment, transmitting the first code stream through the first wireless channel, and transmitting the second code stream through the second wireless channel, wherein the service quality guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the encoder is further configured to: acquiring a code stream of each frame of target image corresponding to the reference frame image in a plurality of frames of target images included in the target video; acquiring audio information in the target video; the transmitter is further configured to: and transmitting the code stream and the audio information of each frame image in the multi-frame images included in the target video corresponding to the reference frame image through the first wireless channel.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the encoder is further configured to: acquiring a code stream of each frame of target image corresponding to the non-reference frame image in a plurality of frames of target images included in the target video; the transmitter is further configured to: and transmitting a code stream of each frame image in the multi-frame images included in the target video corresponding to the non-reference frame image through the second wireless channel.

In a fifth aspect, an embodiment of the present application provides another low-delay source-channel joint coding apparatus, including: an encoder and a transmitter, wherein the encoder is configured to: down-sampling the target image to obtain a reference frame image and a non-reference frame image; determining coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel; respectively carrying out image coding on the reference frame image and the non-reference frame image according to the coding parameters to obtain a first code stream obtained after coding the reference frame image and a second code stream obtained after coding the non-reference frame image; a transmitter to: and respectively transmitting the first code stream and the second code stream.

In a possible implementation manner, the coding parameters are used for controlling the corresponding image to generate a corresponding code stream according to a target code rate when the image coding is performed; an encoder, configured to perform image encoding on the reference frame image and the non-reference frame image according to the encoding parameter, to obtain a first code stream after the reference frame image is encoded and a second code stream after the non-reference frame image is encoded, and specifically configured to: performing intraframe compression on each reference frame image included in the reference frame images according to the target code rate to obtain the first code stream; and performing intra-frame compression on the residual error between the non-reference frame image and the reference frame image according to the target code rate to obtain the corresponding second code stream.

In a possible implementation, the channel environment includes one or more of a bandwidth of a channel, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty cycle, and a bit rate; the transmitter is configured to, before transmitting the first code stream and the second code stream, further: and determining sending parameters respectively corresponding to the first code stream and/or the second code stream according to the channel environment, wherein the sending parameters are used for sending the code streams according to target modulation and coding strategy information and/or target transmitting power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream.

In one possible implementation, the wireless channel includes a first wireless channel and a second wireless channel; the transmitter is configured to, when transmitting the first code stream and the second code stream respectively, specifically: and sending the first code stream through a first wireless channel, and sending the second code stream through a second wireless channel, wherein the service quality guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a sixth aspect, an embodiment of the present application provides a low-delay source-channel joint coding apparatus, including: the first sampling unit is used for carrying out downsampling on the target image to obtain a reference frame image and a non-reference frame image; the first coding unit is used for respectively carrying out image coding on the reference frame image and the non-reference frame image to obtain a first code stream obtained after the reference frame image is coded and a second code stream obtained after the non-reference frame image is coded; the first transmitting unit is configured to determine a first channel resource and a second channel resource based on a channel environment of a current wireless channel, and respectively transmit a first code stream using the first channel resource and a second code stream using the second channel resource, where the first channel resource is better than the second channel resource.

In a possible implementation manner, the first encoding unit is specifically configured to perform intra-frame compression on each reference frame image included in the reference frame image to obtain the first code stream; and performing intra-frame compression on the residual error between each non-reference frame image included in the non-reference frame images and all the reference frame images included in the reference frame images to obtain the corresponding second code stream.

In one possible implementation, the channel environment further includes a channel capacity; the device further comprises: and a first encoding parameter unit, configured to determine, based on the channel capacity, encoding parameters corresponding to the reference frame image and the non-reference frame image before the reference frame image and the non-reference frame image are respectively subjected to image encoding to obtain a first code stream after the reference frame image is encoded and a second code stream after the non-reference frame image is encoded, where the encoding parameters are used to control a corresponding image to generate a corresponding code stream according to a target code rate during image encoding, and the target code rates corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity.

In one possible implementation, the channel environment includes one or more of a bandwidth of a channel, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty cycle, and a bit rate; a first sending unit, configured to determine, based on the current channel environment, sending parameters corresponding to the first code stream and/or the second code stream, where the sending parameters are used to send the code stream according to target modulation and coding policy information and/or target transmission power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream; and sending the first code stream according to the sending parameters corresponding to the first code stream, and sending the second code stream according to the sending parameters corresponding to the second code stream.

In one possible implementation, the wireless channel includes a first wireless channel and a second wireless channel; the first sending unit is specifically configured to: and determining the first wireless channel and the second wireless channel based on the current channel environment, transmitting the first code stream through the first wireless channel, and transmitting the second code stream through the second wireless channel, wherein the service quality guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the above-mentioned device still includes: the first acquisition unit is used for acquiring a code stream of each frame of target image corresponding to the reference frame image in a plurality of frames of target images included in the target video; acquiring audio information in the target video; and a second sending unit, configured to send, through the first wireless channel, a code stream and the audio information, where each frame image in the multiple frame images included in the target video corresponds to the reference frame image.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the above-mentioned device still includes: a second obtaining unit, configured to obtain a code stream of each target image of multiple target images included in the target video, where the target image corresponds to the non-reference frame image; and a third sending unit, configured to send, through the second wireless channel, a code stream in which each frame image in the multiple frame images included in the target video corresponds to the non-reference frame image.

In a seventh aspect, an embodiment of the present application provides another low-delay source-channel joint coding apparatus, including: the second sampling unit is used for carrying out down-sampling on the target image to obtain a reference frame image and a non-reference frame image; the second coding parameter unit is used for determining coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel; the second coding unit is used for respectively carrying out image coding on the reference frame image and the non-reference frame image according to the coding parameters to obtain a first code stream obtained after the reference frame image is coded and a second code stream obtained after the non-reference frame image is coded; and the fourth sending unit is used for respectively sending the first code stream and the second code stream.

In a possible implementation manner, the coding parameters are used for controlling the corresponding image to generate a corresponding code stream according to a target code rate when the image coding is performed; the second encoding unit is specifically configured to: performing intraframe compression on each reference frame image included in the reference frame images according to the target code rate to obtain the first code stream; and performing intra-frame compression on the residual error between the non-reference frame image and the reference frame image according to the target code rate to obtain the corresponding second code stream.

In a possible implementation, the channel environment includes one or more of a bandwidth of a channel, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty cycle, and a bit rate; the fourth transmitting unit, before being configured to transmit the first code stream and the second code stream, is further configured to: and determining sending parameters respectively corresponding to the first code stream and/or the second code stream according to the channel environment, wherein the sending parameters are used for sending the code streams according to target modulation and coding strategy information and/or target transmitting power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream.

In one possible implementation, the wireless channel includes a first wireless channel and a second wireless channel; the fourth sending unit is specifically configured to: and sending the first code stream through a first wireless channel, and sending the second code stream through a second wireless channel, wherein the service quality guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the above-mentioned device still includes: a third obtaining unit, configured to obtain a code stream of each frame of target image in multiple frames of target images included in the target video, where the target image corresponds to the reference frame image; acquiring audio information in the target video; and a fifth transmitting unit, configured to transmit, through the first wireless channel, a code stream and the audio information, where each frame image in a plurality of frame images included in the target video corresponds to the reference frame image.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the above-mentioned device still includes: a fourth obtaining unit, configured to obtain a code stream of each target image of multiple target images included in the target video, where the target image corresponds to the non-reference frame image; and a sixth transmitting unit, configured to transmit, through the second wireless channel, a code stream in which each frame image in the multiple frame images included in the target video corresponds to the non-reference frame image.

In an eighth aspect, an embodiment of the present application provides a low-latency source-channel joint decoding apparatus, including: the device comprises a receiving unit, a decoding unit and a processing unit, wherein the receiving unit is used for receiving a first code stream sent by an encoding end, the first code stream is obtained after image encoding is carried out on a reference frame image, and the reference frame image comprises one or more images obtained after down-sampling is carried out on a target image; the decoding unit is used for obtaining the reference frame image corresponding to the first code stream after the first code stream is subjected to image decoding; and an image unit for reconstructing the target image according to the reference frame image.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the image unit is specifically configured to: and reconstructing the target image through an interpolation algorithm according to at least one of a reference frame image and a non-reference frame image corresponding to one frame of target image, adjacent to the target image, of the reference frame image.

In one possible implementation, the apparatus further includes: a fifth obtaining unit, configured to receive a second code stream sent by the encoding end in a preset time period after the first code stream is received, where the second code stream is a code stream obtained after a non-reference frame image is subjected to the image encoding, and the non-reference frame image includes a residual image obtained after the target image is downsampled, except for the reference frame image; if the second code stream is incomplete, performing image decoding on the second code stream to obtain a corresponding incomplete non-reference frame image; determining peripheral pixels of the incomplete non-reference frame image according to the incomplete non-reference frame image; the image unit is specifically configured to: and reconstructing the target image through an interpolation algorithm according to the peripheral pixels of the reference frame image and the incomplete non-reference frame image.

In one possible implementation, the apparatus further includes: a sixth obtaining unit, configured to, if it is determined that the second code stream is complete, perform image decoding on the second code stream, and obtain the corresponding non-reference frame image; the image unit is specifically configured to: and splicing the reference frame image and the non-reference frame image to reconstruct the target image.

In a ninth aspect, an embodiment of the present application provides a service device, where the service device includes a processor, and the processor is configured to support the service device to perform corresponding functions in the low-latency source-channel joint coding method provided in the first aspect or another low-latency source-channel joint coding method provided in the second aspect. The service device may also include a memory, coupled to the processor, that stores program instructions and data necessary for the service device. The service device may also include a communication interface for the service device to communicate with other devices or a communication network.

In a tenth aspect, an embodiment of the present application provides a service device, where the service device includes a processor, and the processor is configured to support the service device to perform corresponding functions in the low-latency source-channel joint decoding method provided in the third aspect. The service device may also include a memory, coupled to the processor, that stores program instructions and data necessary for the service device. The service device may also include a communication interface for the service device to communicate with other devices or a communication network.

In an eleventh aspect, the present application provides a computer program, where the computer program includes instructions, when the computer program is executed by a computer, the computer can execute the procedures executed by the low-latency source-channel joint coding apparatus provided in the fourth aspect or the low-latency source-channel joint coding apparatus provided in the fifth aspect.

In a twelfth aspect, the present application provides a computer program, where the computer program includes instructions, when the computer program is executed by a computer, the computer can execute the procedures executed by the low-latency source-channel joint decoding apparatus provided in the above sixth aspect.

In a thirteenth aspect, an embodiment of the present application provides a computer storage medium for storing computer software instructions for a low-latency source-channel joint encoding apparatus provided in the fourth aspect, or another low-latency source-channel joint encoding apparatus provided in the fifth aspect, or a low-latency source-channel joint decoding apparatus provided in the sixth aspect, which contains programs for executing the methods designed in the above aspects.

In a fourteenth aspect, an embodiment of the present application provides a chip system, where the chip system includes a processor, configured to support a service device to implement the functions recited in the first aspect, the second aspect, or the third aspect. In one possible design, the system-on-chip further includes a memory for storing program instructions and data necessary for the data transmission device. The chip system may be constituted by a chip, or may include a chip and other discrete devices.

In a fifteenth aspect, an embodiment of the present application provides an electronic device, including the processing chip provided in any implementation manner of the first aspect, the second aspect, or the third aspect, and a discrete device coupled to the chip.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments or the background of the present application will be described below.

Fig. 1 is a schematic diagram of a low-delay source-channel joint coding system architecture provided by an embodiment of the present application.

Fig. 2A is a schematic flowchart of a low-delay source-channel joint coding method according to an embodiment of the present application.

Fig. 2B is a schematic diagram of an application scenario provided in the present application.

Fig. 2C is a schematic diagram that a first code stream and a second code stream provided in the embodiment of the present application are respectively sent through a first channel and a second channel.

Fig. 3A is a schematic flowchart of another low-latency source channel joint coding and decoding method according to an embodiment of the present application.

Fig. 3B is a schematic diagram of an application of encoding and decoding a target image according to an embodiment of the present application.

Fig. 3C is a schematic diagram of another application scenario provided in the present application.

Fig. 4A is a schematic structural diagram of a low-delay source-channel joint encoding apparatus according to an embodiment of the present application.

Fig. 4B is a schematic structural diagram of another low-delay source-channel joint encoding apparatus according to an embodiment of the present application.

Fig. 4C is a schematic structural diagram of a low-latency source-channel joint decoding apparatus according to an embodiment of the present application.

Fig. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present application.

Detailed Description

The embodiments of the present application will be described below with reference to the drawings.

The terms "first," "second," "third," and "fourth," etc. in the description and claims of this application and in the accompanying drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

As used in this specification, the terms "component," "module," "system," and the like are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and/or thread of execution and a component can be localized on one computer and/or distributed between 2 or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The components may communicate by way of local and/or remote processes such as in accordance with a signal having one or more data packets (e.g., data from two components interacting with another component in a local system, distributed system, and/or across a network such as the internet with other systems by way of the signal).

First, some terms in the present application are explained so as to be easily understood by those skilled in the art.

(1) Joint Photographic Experts Group (JPEG), an International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) expert Group, develops and maintains a set of compression algorithm standards for computer image files. The compression of JPEG is usually destructive data compression (lossy compression), meaning that the quality of the image suffers visible damage during the compression process.

(2) The Moving Picture Experts Group (MPEG), an organization established by ISO and IEC in 1988 and specifically developing international standards for Moving Picture and voice compression. The MPEG standards include five major standards, MPEG-1, MPEG-2, MPEG-4, MPEG-7, MPEG-21, and the like. The video compression coding technology of the MPEG standard mainly utilizes an inter-frame compression coding technology with motion compensation to reduce temporal redundancy, utilizes a discrete cosine transform technology to reduce spatial redundancy of images, and utilizes entropy coding to reduce statistical redundancy in terms of information representation.

(3) RGB image, R stands for red; g represents green; b represents blue, and in visual effect, the following components can be used in different proportions: red, green and blue, each pixel of the color image is represented by a different ratio of red, green and blue, and the image is an RGB image.

(4) A Wireless Local Area Network (WLAN), which is a very convenient data transmission system, and utilizes Radio Frequency (RF) technology to replace the old lan formed by twisted copper (coax) wires, which obstruct the hands and feet, by using electromagnetic waves, and performs communication connection in the air, so that the WLAN can make users pass through it by using a simple access structure, thereby achieving an ideal situation of 'information portability and convenience for walking down the sky'.

(5) Discrete Cosine Transform (DCT), which is a Transform related to fourier Transform, is similar to Discrete fourier Transform, but uses only real numbers. The discrete cosine transform corresponds to a discrete fourier transform of approximately twice its length, which is performed on a real even function (since the fourier transform of a real even function is still a real even function), and within some variants it is necessary to shift the position of the input or output by half a unit. There are two related transforms, one is the discrete sine transform, which corresponds to a discrete fourier transform of a real odd function approximately twice as long; another is a modified discrete cosine transform, which is equivalent to a discrete cosine transform of overlapping data.

(6) Run Length Coding (RLC), also known as Run Length Coding or Run Length Coding, is a statistical Coding, which belongs to lossless compression Coding and is an important Coding method for grid data compression. Valid for binary maps. The basic principle of run-length coding is as follows: successive symbols having the same value are replaced by a symbol value or string length (successive symbols constitute a continuous "run". the run-length code is thus named), making the symbol length smaller than the length of the original data. And only when the codes of the data in each row or each column are changed, the codes and the repeated number of the same codes are recorded once, so that the data compression is realized.

(7) The Signal to Interference plus Noise Ratio (SINR) is the Ratio of the received strength of the desired Signal to the received strength of the interfering Signal (Noise and Interference).

(8) Signal to Noise Ratio (SNR), also called Signal to Noise Ratio, refers to the Ratio of Signal to Noise in an electronic device or system. The signal refers to an electronic signal from the outside of the device to be processed by the device, the noise refers to an irregular extra signal (or information) which does not exist in the original signal generated after passing through the device, and the signal does not change along with the change of the original signal.

(9) Packet Error Rate (PER), an index for measuring the transmission accuracy of a data Packet (data Packet) in a specified time, is also an important index for measuring the service quality of a data service. The packet error rate is 100% of the number of packets in error per total number of packets transmitted.

(10) Macroblock (Macroblock), is a basic concept in video coding technology. In video coding, a coded picture is usually divided into macroblocks, a macroblock consisting of a block of luminance pixels and additionally two blocks of chrominance pixels. In general, a luminance block is a 16 × 16 pixel block, and the size of two chrominance image pixel blocks depends on the sampling format of its image, such as: for a YUV420 sample image, the chroma block is a block of pixels 8x8 in size. In each image, a plurality of macro blocks are arranged in a form of a slice, and a video coding algorithm codes the macro blocks one by taking the macro blocks as units to organize a continuous video code stream. .

(11) The code stream (Data Rate) is a Data flow Rate used by a video file in a unit time, also called a bit Rate, and is the most important part in picture quality control in video coding.

(12) Entropy coding is a scheme for lossless data compression that is independent of the specific characteristics of the medium. One major type of entropy coding is to create and assign a unique prefix code to each input symbol, and then achieve the goal of compressing the data by replacing each fixed length input symbol with a corresponding variable length prefix-independent (prefix-free) output codeword. The length of each codeword is approximately proportional to the negative logarithm of the probability. Therefore, the most common symbols use the shortest code.

(13) Intraframe (Intraframe) compression, also known as Spatial compression (Spatial compression). When compressing an image of one frame, only the data of the present frame is considered without considering redundant information between adjacent frames, which is actually similar to still image compression. Intra-frame compression typically employs lossy compression algorithms.

(14) Interframe (Interframe) compression is based on the characteristic that two continuous frames before and after a plurality of videos or animations have great correlation, or the information of the two frames before and after is changed little. That is, the adjacent frames of the continuous video have redundant information, and according to the characteristic, the compression amount can be further improved by compressing the redundant amount between the adjacent frames, and the compression ratio is reduced. Interframe compression is also referred to as Temporal compression (Temporal compression).

(15) Modulation and Coding Scheme (MCS), the configuration of 802.11n radio frequency rate is realized by MCS index value. The MCS modulation coding table is a representation of 802.11n proposed for characterizing the communication rate of a WLAN. The MCS forms a rate table with the MCS index as a row and the columns of the table as the factors that affect the communication rate concerned.

(16) Down-sampling, sampling once every several samples for a sample sequence, so that the obtained new sequence is the down-sampling of the original sequence. The sampling rate variation is mainly due to the fact that different modules of the signal processing may have different sampling rate requirements. Down-sampling is performed with respect to the original continuous-time signal, but still satisfies the sampling theorem, otherwise such down-sampling causes aliasing of the signal components. Downsampling is decimation, and is one of the basic contents in multi-rate signal processing.

(17) Image coding, also called image compression, refers to a technique of representing an image or information included in an image with a small number of bits under a condition that a certain quality (a requirement of a signal-to-noise ratio or a subjective evaluation score) is satisfied.

(18) Quality of Service (QoS) is a security mechanism for a network, and is a technique for solving the problems of network delay and congestion. When the network is overloaded or congested, QoS can ensure that important traffic is not delayed or dropped while ensuring efficient operation of the network.

(19) YUV, a color coding method. Are often used in various video processing components. YUV allows for reduced bandwidth of chrominance in view of human perception when encoding photos or videos. YUV is a kind of compiled true-color space (color space), and the proper terms such as Y' UV, YUV, YCbCr, YPbPr, etc. may be called YUV, overlapping with each other. "Y" represents brightness (Luminince, Luma) and "U" and "V" represent Chroma and concentration (Chroma)

(20) Channel capacity (Channel capacity), also known as Channel capacity, refers to the minimum upper bound on the achievable rate at which information can be reliably transmitted in a Channel. By reliable transmission is meant that information can be communicated at an arbitrarily small error rate. According to the noisy channel coding theorem, the channel capacity is the limit information rate of a given channel that can be reached with arbitrarily small error probability. The unit of channel capacity is bits per second, nits per second, etc

Secondly, the application scenario and the specific technical problem to be solved of the application are analyzed and provided.

Application scenarios: in the prior art, the development of wireless video transmission solves the problem of transmitting data such as audio and video under the condition of inconvenient wiring construction. Therefore, the wireless screen projection is one of typical application scenarios in which image encoding and decoding are wirelessly transmitted, such as compressing screen information of terminals such as a mobile phone and a tablet computer, then wirelessly projecting the screen information to large-screen equipment such as a television, sharing videos/pictures to friends through a wireless network, and carrying out a video conference through the wireless network.

A conventional wireless video transmission system generally comprises two parts, namely a transmitting end and a receiving end. In addition, an appropriate gain antenna can be added to increase the transmission distance according to the requirements of the actual situation. The transmitting end and the receiving end of the general wireless video transmission mainly comprise several main links such as audio and video coding, wireless transmission, audio and video decoding and displaying and the like. In the prior art, JPEG standard encoding and MPEG standard encoding are two common image encoding methods, both of which can convert an original image into a YUV space after color space conversion, and then obtain a compressed image after segmentation, Discrete Cosine Transform (DCT), quantization and entropy encoding. However, since the short-term time-varying characteristic of the wireless channel can cause the channel capacity to change rapidly, the audio/video codec algorithm at the transmitting end needs to be able to track and adapt to the change rapidly, so that the delay and the image quality at the receiving end, i.e. the receiving end, are kept at acceptable levels, whereas the conventional audio/video coding techniques such as JPEG standard coding and MPEG standard coding, which only focus on the compression factor and the image quality, cannot make corresponding coding adjustment rapidly according to the channel capacity. Therefore, part or even all of the code stream information is easily lost in the wireless transmission process. Moreover, a receiving end corresponding to the traditional audio and video coding technology can reconstruct a complete image only if all code streams corresponding to a certain frame are available, and mosaic, katon and other phenomena can be caused by loss of a small amount of code streams, so that the appearance is poor. In addition, based on the network layering principle, the priority of a source part (namely, code stream information after audio and video coding) is generally not concerned during wireless transmission, and non-differential transmission is executed. Therefore, the encoding end cannot control whether critical information (such as audio information and code stream information of important frames) is lost in the wireless channel, and cannot adapt to the change of the wireless channel quickly, and does not transmit information sources (such as code stream information or other data information) in a grading manner, so that the receiving end may not reconstruct a complete video image due to the loss of the important information.

In order to facilitate understanding of the embodiments of the present application, a description will be given below of one of low-delay source-channel joint coding system architectures on which the embodiments of the present application are based. Referring to fig. 1, fig. 1 is a schematic diagram of a low-delay source-channel joint coding system architecture according to an embodiment of the present application.

The low-delay source-channel joint coding system architecture in the present application may include an encoding end 10 and a decoding end 11 in fig. 1. Wherein. The encoding end 10 of the low-delay source-channel joint coding system architecture comprises a video source 101, an encoder 102 and a transmitter 103. The decoding end 11 of the low-delay source-channel joint coding system architecture comprises a decoder 111 and a display device 112. Wherein the content of the first and second substances,

the video source 101 is an interface, video memory, storage, etc. for providing target video, wherein the target video may come from various types of interfaces, such as: an HDMI (High Definition Multimedia Interface) Interface, a digital Video Port (DP) Interface, a Video Graphics Array (VGA) Interface, and the like. For example: when the video source 101 is an HDMI interface, the target video may be transmitted to the encoder 102 so that the encoder 102 encodes the target image of the target video.

An encoder 102(encoder) is a device that compiles, converts, and formats signals (e.g., bitstreams) or data into a form of signals that can be communicated, transmitted, and stored. The encoder can convert angular displacement or linear displacement into an electrical signal, the former is called code disc, the latter is called code ruler, the encoder can be related encoding software running on a general Central Processing Unit (CPU) or a special chip, or a separate encoding chip, or a part of a separate chip, such as: a portion of a System-on-a-Chip (SoC) Chip of an integrated circuit. For example, when the encoder is an independent encoding chip, the target image may be hierarchically encoded, that is, the target image is downsampled to obtain a reference frame image and a non-reference frame image, and then the image encoding is performed; then, a first code stream after the reference frame image coding and a second code stream after the non-reference frame image coding are obtained; finally, the first code stream and the second code stream are sent to the decoder 111 through the transmitter 103. The coding parameters can also receive coding parameters which are sent by the coding parameter feedback control and fed back according to the channel information of the current wireless channel, and the coding parameters are used for enabling the code rate of the code stream after image coding to be smaller than the channel capacity of the current wireless channel. For example, when the encoder is an independent encoding chip, the target image may be downsampled to obtain a reference frame image and a non-reference frame image; determining coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel; and respectively carrying out image coding on the reference frame image and the non-reference frame image according to the coding parameters to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded.

The transmitter 103 is a device that uses a wireless-connected local area network to transmit data using radio waves, and the transmission distance is typically several tens of meters. The transmitter 103 may be a WiFi chip, and may include drivers, firmware, etc. associated with the WiFi chip. For example: when the transmitter 103 is a WiFi chip, the encoder 102 may perform image encoding to obtain a first code stream, and then the second code stream is respectively transmitted through a wireless channel, where a channel resource when the first code stream is wirelessly transmitted is higher than a channel resource when the second code stream is wirelessly transmitted. It should be noted that the transmitter 103 may also obtain the channel environment of the current wireless channel, such as: channel capacity, bandwidth of the channel or signal to interference plus noise ratio, etc.; then, channel information is sent to the encoder 103 through encoding parameter feedback control, so that the encoder 103 adjusts encoding parameters of the first code stream and the second code stream in real time; the transmitter 103 may also adjust the transmission parameters of the first code stream and the second code stream in real time. The coding parameter feedback control is mainly used for adjusting the coding parameters of the encoder during image coding according to the channel information of the current wireless channel. It is understood that the encoder 102 and the transmitter 103 may be coupled devices for receiving the target video information from the video source 101 and encoding the target video information to the decoder 111.

Optionally, the decoder (decoder)111 is a hardware/software device capable of decoding the digital video/audio data stream to an analog video/audio signal. The coder mainly compresses the analog video and audio signal into a data coding file, and the decoder converts the data coding file into the analog video and audio signal. The decoder can be a part of a general-purpose CPU or a special-purpose chip, or a separate decoding chip or a separate chip (such as an SoC chip) with relevant decoding software running thereon; for example: when the decoder 111 is an independent decoding chip, the decoder 111 may receive a first code stream sent by the transmitter 103 at the receiving and encoding end, where the first code stream is a code stream obtained by performing image encoding on a reference frame image, where the reference frame image includes a reference frame image obtained by performing down-sampling on a target image; then, the decoder 111 decodes the first code stream according to a flow opposite to that when the encoder 102 encodes the target image of the target video to obtain a reference frame; and finally, obtaining a target image according to the reference frame. Another example is: the decoder 111 may also receive the second code stream, and decode the second code stream according to a process opposite to that of the encoder 102 when encoding the target image of the target video to obtain a complete or incomplete non-reference frame; finally, the decoder uses the reference frame and the complete non-reference frame to splice a complete reconstruction frame, namely a target image; when the non-reference frame is incomplete, a reconstructed frame can be obtained by interpolation of adjacent frame images (for example, at least one of a reference frame image and a non-reference frame image of an adjacent frame target image) or a peripheral pixel set, and then the obtained reconstructed frame is sent to the display device 112 to be displayed and played. It can also be understood that, when determining whether the non-reference frame is incomplete, the decoder 111 may determine whether the code stream is incomplete when receiving the code stream, and further determine whether an image corresponding to the code stream is incomplete, which is not limited in this application.

Optionally, the display device 112 is a display tool that displays certain data on a screen through a specific transmission device and reflects the data to human eyes, and may generally refer to a display, a projector, a virtual reality head-mounted display device, an intelligent terminal with a display function, and the like. For example, the display device 112 may be: virtual Reality head-mounted display devices (e.g., Virtual Reality (VR) glasses, VR eyepieces, VR helmets), smart phones, laptops, tablets, projectors, cameras, etc., or display clients, applications, etc. installed or run on computers, tablets, smart phones. For example: when the display device is a projector, the projector may project the data information of the target video sent by the decoder 111 onto a screen through the projection device, and then play the target video. It is understood that the decoder 111 and the display device 112 may be coupled together to receive the bitstream information transmitted from the transmitter 103, and decode the bitstream information to display the target video corresponding to the bitstream information.

It should be noted that the decoding end 11 in the low-delay source-channel joint coding system architecture provided in the embodiment of the present application is equivalent to the receiving end in the present application.

It should be further noted that the low-latency source-channel joint coding system architecture of fig. 1 is only a partial exemplary implementation manner in the embodiment of the present application, and the low-latency source-channel joint coding system architecture in the embodiment of the present application includes, but is not limited to, the above low-latency source-channel joint coding system architecture.

Based on the technical problem analysis and the low-delay source-channel joint coding system architecture, the application provides a low-delay source-channel joint coding method which can avoid the requirement on the integrity of code streams in the traditional audio-video coding and decoding algorithm and can also quickly track the change of channel capacity, so that the delay of a receiving end and the image quality are kept at an acceptable level. When the channel capacity is reduced, the phenomena of mosaic, blockage, blurring and the like of video information are reduced, and the viewing experience of a user is improved.

Referring to fig. 2A, fig. 2A is a schematic flow chart of a low-latency source-channel joint coding method according to an embodiment of the present application, which can be applied to the architecture of the low-latency source-channel joint coding system described in fig. 1, where the encoder 102 and the transmitter 103 can be used to support and execute the method steps S201 to S203 shown in fig. 2A.

Step S201: and performing downsampling on the target image to obtain a reference frame image and a non-reference frame image.

Specifically, the encoder 102 down-samples the target image to obtain a reference frame image and a non-reference frame image, and the down-sampling is performed every several samples of a sample sequence, so that the obtained new sequence is the down-sampling of the original sequence. Referring to fig. 2B, fig. 2B is a schematic view of an application scenario provided in the present application. For example: when a target video of a segment of 3 seconds needs to be wirelessly transmitted, the encoder 102 may perform downsampling on a plurality of target images of the input target video, where a horizontal sampling rate of the downsampling is 2:1 and a vertical sampling rate is 2:1, the downsampling is divided into 4 duplicate images with lower resolutions, one of the duplicate images may be selected to be classified as a reference frame image, and the remaining 3 duplicate images are non-reference frame images. It is to be understood that the reference frame image may include one or more images, and the non-reference frame image may also include one or more images, and the reference frame image includes a number of images that is less than or equal to the number of images included in the non-reference frame image. When the target image is downsampled, the target image may be sampled at half the resolution, or may be sampled at an arbitrary ratio of the resolution of the target image. The resolution of the down-sampled replica image is smaller than that of the target image, so that the code rate of the code stream obtained after the compression of the replica image is smaller than that of the code stream obtained when the target image is compressed by using the same coding technology, and the decoder at the receiving end can still obtain a reconstructed image based on the replica image with a smaller resolution when the channel capacity is jittered or the code rate is reduced to be lower than the initial code rate (for example, the initial code rate can be the code rate or the preset code rate when the target image is compressed according to the traditional coding method). Optionally, the target video includes multiple frames of target images, and the target image is any one of the multiple frames of target images.

Optionally, the encoder 102 performs downsampling on the target image to obtain N duplicate images, where the resolution of the N duplicate images is lower than that of the target image, where N is a positive integer greater than 1; and dividing the N duplicate images into a reference frame image and a non-reference frame image. After the encoder 102 down-samples the target image to obtain a plurality of low-resolution duplicate images, the plurality of low-resolution duplicate images may be divided into two different numbers of images, wherein the number of images included in the reference frame image is less than or equal to the number of images included in the non-reference frame image. It will also be appreciated that the intersection of the reference frame image and the non-reference frame image is an empty set. As shown in fig. 2B, 4 duplicate images obtained after the down-sampling, wherein any 1 duplicate image may be selected as the reference frame image, and the remaining 3 duplicate images are non-reference frame images.

Step S202: and respectively carrying out image coding on the reference frame image and the non-reference frame image to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded.

Specifically, the encoder 102 performs image encoding on the reference frame image and the non-reference frame image into which the multiple duplicate images are divided, and obtains a first code stream after encoding the reference frame image and a second code stream after encoding the non-reference frame image. For example: the encoder 102 may sequentially perform blocking, DCT, quantization, run length encoding, entropy encoding, packing, marking, and the like on the duplicate image in the reference frame image to generate a first code stream. The encoder 102 may perform blocking, DCT, quantization, run length encoding, entropy encoding, packing, labeling, etc. on a residual of each duplicate image in the non-reference frame image with respect to a duplicate image in the reference frame image or directly on each duplicate image in the non-reference frame image to generate a second code stream.

Optionally, the encoder 102 may perform intra-frame compression on the reference frame image and the non-reference frame image, respectively, to obtain a first code stream after encoding the reference frame image and a second code stream after encoding the non-reference frame image. It can be understood that, when a simpler down-sampling is selected to layer a target image and reduce resolution, and an intra-frame compression encoding method is used for image encoding of a layered duplicate image, since intra-frame compression is spatial domain compression, when a frame image is compressed, only data of the frame is considered and redundant information between adjacent frames is not considered, and inter-frame compression refers to other frame data, the compression ratio is large, and in view of image delay, the embodiment of the present application is not suitable for inter-frame compression when image encoding of a down-sampled duplicate image is performed. Therefore, by using the intra-frame compression method, when the channel capacity is jittered and is reduced to be lower than the initial code rate, the decoder can receive the code stream information encoded based on the copy image with the smaller resolution, and the reconstructed image can be obtained according to the code stream information. Meanwhile, when the channel capacity is reduced, the phenomena of mosaic, blocking, blurring and the like of video information are reduced, and the viewing experience of a user is improved.

Optionally, the encoder 102 may also use other layered coding methods to divide the target image into lower resolution duplicate images, and the resolution, the bit rate, and the frame rate of the multiple duplicate images may be different or the same. For example: the code stream generated by the layered video coding comprises a plurality of sub-code streams, each sub-code stream is divided into a basic layer and an expansion layer, each layer has different code rates, frame rates and resolutions, the basic layer has the most basic video quality, each expansion layer is a supplement to the previous layer, and the stream pulling end can select and decode several sub-code streams according to the actual network environment (such as the bandwidth of a channel, the signal-to-interference-plus-noise ratio, the signal-to-noise ratio, the received signal strength indication, the duty ratio, the bit rate and the like).

Optionally, before the encoder 102 performs image coding on the reference frame image and the non-reference frame image respectively to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded, it may further determine, based on a channel environment of a current wireless channel, coding parameters corresponding to the reference frame image and the non-reference frame image, where the channel environment may include a channel capacity, the coding parameters are used to control a corresponding image to generate a corresponding code stream according to a target code rate when performing image coding, and the target code rates corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity. It should be noted that the encoding parameter determined by the encoder may be an encoding parameter sent by the encoding parameter feedback control. Therefore, when the channel capacity changes, the transmitter can acquire the channel capacity of the current channel, and adjust the coding parameters of the coder during image coding of the duplicate image according to the fed back channel capacity information, so that the code rate of the coded code stream can smoothly send the target code stream under the condition of adapting to the current channel capacity, and the condition that the code stream is lost due to the reduction of the channel capacity is reduced. Meanwhile, the method also reduces the problem that a decoder at the receiving end cannot obtain a reconstructed image according to the missing code stream information because the complete code stream cannot be transmitted, reduces the jam and blur probability of the video image, and improves the watching experience of a user.

Step S203: determining a first channel resource and a second channel resource based on the channel environment of the current wireless channel, and respectively transmitting a first code stream by using the first channel resource and transmitting a second code stream by using the second channel resource.

Specifically, the transmitter 103 may respectively transmit the first code stream and the second code stream by using the first channel resource after determining the first channel resource and the second channel resource based on the channel environment of the current wireless channel for the first code stream and the second code stream received from the encoder 102. As shown in fig. 2B, in the embodiment of the present application, the first code stream and the second code stream may be respectively transmitted based on the fed back channel environment of the current wireless channel. For example: the priority of the first code stream can be set to be higher than that of the second code stream, and then when the first code stream and the second code stream are respectively sent, the service quality of a channel when the first code stream is sent is higher than that of a channel when the second code stream is sent according to the channel environment of the current wireless channel. Or when the channel environment of the current wireless channel is poor, the more important first code stream can be preferentially sent, and then the second code stream with the lower level is sent; or when the channel environment of the current wireless channel is poor, after the receiving end receives the first code stream or after a period of time after the first code stream is sent, the second code stream with a lower level is sent.

Optionally, the transmitter 103 may send link statistical information such as SINR/SNR/RSSI to the encoding end through the air interface, where the encoding end counts the frame error rate, the collision rate, and then adjusts the sending parameters according to the link statistical information, the frame error rate, the collision rate, and the like, where: MCS, transmit power, quantization parameters, etc. are sent. That is, when the channel changes, the transmitter acquires the channel information of the current channel (e.g. the bandwidth of the channel, the signal-to-interference-plus-noise ratio, the signal-to-noise ratio, the received signal strength indication, the duty ratio, the bit rate, etc.), and adjusts the transmission parameters (e.g. the transmission power, the modulation and coding strategy information, etc.) when the first code stream is transmitted and/or the second code stream is transmitted according to the channel information, for example: the transmitter 103 may determine, based on the current channel environment, sending parameters corresponding to the first code stream and/or the second code stream, where the sending parameters are used to send the code streams according to target modulation and coding policy information and/or target sending power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream; and sending the first code stream according to the sending parameters corresponding to the first code stream, and sending the second code stream according to the sending parameters corresponding to the second code stream. Wherein the channel environment comprises one or more of a bandwidth, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty cycle, a bit rate of a channel. It can be understood that the transmitter 103 responds to the transmission request of the first code stream more quickly when transmitting the first code stream and the second code stream. In the process of wireless transmission, the importance and the priority of video sources are classified, that is, the transmission quality when a first code stream is sent is higher than that when a second code stream is sent, or more important code streams (that is, code streams after image coding is carried out on the copy images with less quantity) are sent preferentially, so that when the channel capacity and the like are jittered and a decoder cannot receive complete code streams, reconstructed images can be obtained only on the basis of the copy images corresponding to partial code streams, the phenomena of mosaic, blockage, blurring and the like of video information can be reduced, and the viewing experience of users is improved.

Optionally, the transmitter 103 may further determine the first wireless channel and the second wireless channel based on a current channel environment, send the first code stream through the first wireless channel, and send the second code stream through the second wireless channel, where a quality of service guarantee mechanism of the first wireless channel is higher than that of the second wireless channel. Wherein the wireless channels include a first wireless channel and a second wireless channel. Referring to fig. 2C, fig. 2C is a schematic diagram illustrating that a first code stream and a second code stream are respectively transmitted through a first channel and a second channel according to an embodiment of the present application. The transmitter 103 transmits the first code stream through a first wireless channel and transmits the second code stream through a second wireless channel. When the channel capacity and the like are jittered, the first code stream and the second code stream sent through the wireless channel respectively use different QoS guarantee levels, the decoder can only receive the first code stream sent through the channel with a high service quality guarantee mechanism, and then can only reconstruct an image through the first code stream, so that the phenomena of mosaic, blockage, blurring and the like of video information are reduced, and the viewing experience of a user is improved. It can be understood that, after the transmission parameters are adjusted according to the channel information, the transmission parameters of the first code stream are different from the transmission parameters of the second code stream, and the transmission parameters of the first code stream can enable the transmitter to transmit the first code stream at a faster transmission speed and a higher QoS than the transmission parameters of the second code stream. That is, the first code stream is transmitted with better channel resources and higher transmission success rate, which may refer to the probability that the code stream information is completely transmitted to the receiving end. Therefore, in the process of wireless transmission, video sources can be classified, and more important code streams are preferentially sent, so that when channel capacity and the like are jittered or a decoder cannot receive complete code streams, reconstructed images obtained only based on duplicate images corresponding to partial code streams can be obtained, for example: the first code stream and the second code stream are sent through a wireless channel, and the first code stream is mapped to a higher Quality of Service (QoS) guarantee mechanism, such as: and mapping the first code stream to the VI service, and mapping the second code stream to the BE service. In the process of wireless transmission, the first code stream and the second code stream can also be sent through the same wireless channel, but only the first code stream can be sent preferentially, so that a receiving end can be ensured to at least receive the first code stream. Another example is: when the first code stream and the second code stream are sent, the first code stream can be sent first, and the second code stream is sent only after the receiving end receives the first code stream or after a preset time period after the first code stream is sent, so that the receiving end can be ensured to obtain at least one complete code stream information with lower resolution, and an image can be reconstructed conveniently. The first code stream and the second code stream are respectively sent through the wireless channel, so that the probability that the receiving end receives the first code stream is improved even if the receiving end cannot receive the complete code stream, and the receiving end can obtain a complete reconstructed image only when receiving the first code stream.

Optionally, a code stream of each frame of target image corresponding to the reference frame image in a plurality of frames of target images included in the target video is obtained; acquiring audio information in the target video and marking the audio information as audio information; and transmitting the code stream and the audio information of each frame image in the multi-frame images included in the target video corresponding to the reference frame image through the first wireless channel. It can be understood that, in a video, the audio information is as important as the first code stream information of the multi-frame target image, so that in order to avoid the loss of the audio information, the audio information and the first code stream can be sent together through the first channel with higher QoS guarantee level, the phenomena of no sound, blockage and the like of the video information are reduced, and the impression experience of a user is improved.

Optionally, a code stream of each frame of target image corresponding to the non-reference frame image in multiple frames of target images included in the target video is obtained; and transmitting a code stream of each frame image in the multi-frame images included in the target video corresponding to the non-reference frame image through the second wireless channel. And after code streams of all target images in multi-frame target images included in the target video, which correspond to the non-reference frame images, are obtained, the code streams are mapped to BE services with a slightly lower service quality guarantee mechanism to BE sent. Therefore, even if the code streams corresponding to the non-reference frame images of all the target images lose part or even all of the code stream information when the channel changes, the decoder can reconstruct the video only on the basis of the code streams corresponding to the first copies of all the target images and the audio information received through the channel with a high service quality guarantee mechanism, reduce the phenomena of mosaic, blockage, blurring and the like of the video information, and improve the experience of the user.

In the embodiment of the application, the target image may be downsampled to obtain a reference frame image and a non-reference frame image, then the reference frame image is subjected to image coding to obtain a first code stream, the non-reference frame image is subjected to image coding to obtain a second code stream, finally the first code stream is sent by using the first channel resource, and the second code stream is sent by using the second channel resource. The first channel resource and the second channel resource are determined according to the channel environment of the current wireless channel, and the first channel resource is better than the second channel resource, that is, in the process of wireless transmission, the first code stream after encoding the reference frame image and the second code stream after encoding the non-reference frame image may be sent in a hierarchical manner, for example: the priority of the first code stream can be set to be higher than that of the second code stream, and then when the first code stream and the second code stream are respectively sent, the service quality of a channel when the first code stream is sent is higher than that of a channel when the second code stream is sent according to the channel environment of the current wireless channel. Or when the channel environment of the current wireless channel is poor, the more important first code stream can be preferentially sent, and then the second code stream with the lower level is sent; or when the channel environment of the current wireless channel is poor, after the receiving end receives the first code stream or after a period of time after the first code stream is sent, the second code stream with a lower level is sent. The first code stream and the second code stream are respectively sent based on the channel environment of the current wireless channel, so that the probability that the receiving end receives the first code stream is improved when the receiving end cannot receive the complete code stream, and a complete reconstructed image is obtained by utilizing the received first code stream. Secondly, before the target image is subjected to image coding, downsampling is carried out on the target image, and after a plurality of duplicate images with lower resolution are obtained, image coding is carried out, so that the code stream obtained after downsampling is smaller than the code stream obtained when the target image is directly subjected to image coding, and a decoder at a receiving end can receive the complete code stream more easily. Therefore, after the target image is subjected to image coding through downsampling, the coded code stream information is sent through the wireless channels respectively based on the channel environment of the current wireless channel, so that the coded code stream information can still adapt to the channel environment which changes in real time, the probability of information loss in the wireless transmission process is further reduced, the situation that a receiving end cannot reconstruct a complete video image due to information loss is further avoided, and the watching experience of a user is improved.

Referring to fig. 3A and fig. 3B, fig. 3A is a schematic flowchart of a low-delay source-channel joint coding and decoding method provided in an embodiment of the present application, and fig. 3B is a schematic diagram of an application of coding and decoding a target image provided in an embodiment of the present application. The method shown in fig. 3A can be applied to the architecture of the low-latency source-channel joint coding system described in fig. 1, where the encoder 102 and the transmitter 103 can be used to support and execute the method flow steps S301-S304 shown in fig. 3A, and the decoder 111 can be used to support and execute the method flow steps S305-S307 shown in fig. 3A. The method may comprise the following steps S301-S307.

Step S301: and performing downsampling on the target image to obtain a reference frame image and a non-reference frame image.

Specifically, the above description of step S301 may correspond to the related description of step S201 in fig. 2A, and is not repeated herein.

Step S302: and determining coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel.

Specifically, the encoder 102 determines the encoding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel, please refer to fig. 3C, where fig. 3C is another application scenario diagram provided in the present application, that is, the encoding parameters during the image encoding can be determined according to the channel environment of the current wireless channel. For example: when a target video of a segment of 3 seconds needs to be wirelessly transmitted, the encoder 102 may perform downsampling on a plurality of frame target images of the input target video, where a horizontal sampling rate of the downsampling is 2:1 and a vertical sampling rate is 2:1, the downsampling is divided into 4 duplicate images with lower resolutions, one of the duplicate images is selected to be classified as a reference frame image, and the remaining 3 duplicate images are non-reference frame images, and then, based on a channel environment of a current wireless channel, encoding parameters corresponding to the reference frame image and the non-reference frame images during image encoding are determined to reduce a size of an encoded code stream.

Alternatively, the transmitter 103 may acquire a channel environment of the current wireless channel, wherein the channel environment includes a channel capacity. When the channel capacity changes, the transmitter acquires the channel capacity of the current channel, and can adjust the coding parameters of the coder during image coding of the duplicate image according to the channel capacity, so that the code rate of the coded code stream can smoothly send the target code stream under the condition of adapting to the current channel capacity, and the condition of code stream loss caused by the reduction of the channel capacity is reduced.

Step S303: and respectively carrying out image coding on the reference frame image and the non-reference frame image according to the coding parameters to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded.

Specifically, the encoder 102 adjusts, through encoding parameter feedback control, encoding parameters corresponding to a reference frame image and a non-reference frame image during image encoding according to channel capacity, and then performs image encoding on the reference frame image and the non-reference frame image according to the determined encoding parameters after adjustment, to obtain a first code stream after encoding the reference frame image and a second code stream after encoding the non-reference frame image. The coding parameters are used for controlling the corresponding copies to generate corresponding code streams according to target code rates when image coding is carried out, so that the code rates of the first code stream and the second code stream are both smaller than or equal to the channel capacity, and the code rates of the coded code streams are ensured to be suitable for the current channel capacity. Therefore, the coding parameters during image coding are adjusted according to the channel information of the current wireless channel, so that the code rate of the generated code stream can be lower than the channel capacity of the current wireless channel, and further, when the code stream is transmitted wirelessly, the code stream cannot lose part of the code stream information when the channel capacity is too low to transmit a complete code stream or the channel capacity is jittered or the channel capacity is reduced to be lower than the initial code rate (for example, the initial code rate can be the code rate or the preset code rate when a target image is compressed according to the traditional coding method). Therefore, the image coding and decoding algorithm of the embodiment of the application can quickly track and adapt to the change of the channel environment of the wireless channel, and the delay and the image quality of the receiving end are kept at an acceptable level.

Optionally, the code rates of the first code stream and the second code stream may be the same or different, when the code rates of the first code stream and the second code stream are the same, the coding parameters during image coding may be uniformly regulated, and when the code rates of the first code stream and the second code stream are different, it is first required to ensure that the code rate of the first code stream is less than or equal to the channel capacity, so as to reduce the probability of code stream loss caused by the reduction of the channel capacity.

Step S304: and respectively transmitting the first code stream and the second code stream.

Specifically, the above description of step S304 may correspond to the related description referring to step S203 of fig. 2A, and is not repeated herein.

Step S305: and receiving a first code stream sent by an encoding end.

Specifically, the decoder 111 receives a first code stream sent by the encoding end 102, where the first code stream is a code stream obtained after image encoding is performed on a reference frame image. According to the embodiment of the application, the first code stream sent by the encoding end can be received in the wireless transmission process, the first code stream is subjected to image decoding, a reference frame image corresponding to the first code stream is obtained, and then the target image is obtained according to the reference frame image. It should be noted that the first code stream is a code stream obtained by performing image coding on a reference frame image, where the reference frame image includes one or more images obtained by down-sampling a target image, and therefore, the code stream received by the coding end is obtained by down-sampling the target image and then performing image coding. The down-sampling may divide the target image into a plurality of lower resolution duplicate images, and select a portion of them as the reference frame image. The size of a code stream obtained by down-sampling and re-encoding the target image is smaller than that obtained by directly encoding the target image, so that the down-sampled code stream can better adapt to the change of a wireless channel, the probability of information loss is reduced in the transmission process, and the problem that a receiving end cannot reconstruct a complete video image due to information loss is avoided

Step S306: and after the first code stream is subjected to image decoding, a reference frame image corresponding to the first code stream is obtained.

Specifically, the decoder 111 performs image decoding on the first code stream to obtain the reference frame image corresponding to the first code stream. The encoder 102 performs blocking, DCT, quantization, run-length encoding, entropy encoding, packaging and marking on the reference frame image to obtain a corresponding first code stream; the decoder 111 may decode the first code stream to obtain YUV according to the procedure completely opposite to that of the encoder 102, i.e., label decoding, unpacking, inverse entropy coding, inverse run length coding, inverse quantization, inverse DCT, and then convert YUV into RGB to obtain the reference frame image. The encoder at the encoding end shown in fig. 3B converts the target image from RGB to YUV, and then sequentially performs blocking, DCT, quantization, run length encoding, entropy encoding, packing, and labeling; and then, the code stream is sent to a receiving end, a decoder of the receiving end decodes the first code stream according to a flow opposite to the coding to obtain YUV, and the YUV is converted into RGB to obtain a reference frame.

Step S307: and reconstructing the target image according to the reference frame image.

Specifically, the decoder 111 will reconstruct the target image from the reference frame image. The target image is any one of a plurality of frames of target images included in the target video; reconstructing the target image according to the reference frame image includes: and reconstructing the target image through an interpolation algorithm according to at least one of a reference frame image and a non-reference frame image corresponding to one frame of target image adjacent to the target image in the reference frame image. For example: a time packet may be marked to the code stream when the image coding is performed; when the first code stream is received, the second code stream is not received in the time period specified by the time packet, that is, the second code stream is considered to be lost, and at this time, a reconstructed frame, that is, a target image, can be obtained according to the received reference frame code stream and the previous frame interpolation. Therefore, when the decoding end receives the first code stream, the situation that the receiving end cannot reconstruct a complete video image due to information loss can be avoided, the phenomena of no sound, blocking and the like of video information are reduced, and the impression experience of a user is improved.

Optionally, the decoder 111 receives a second code stream sent by the encoding end within a preset time period after receiving the first code stream, where the second code stream is a code stream obtained after performing image encoding on a non-reference frame image, and the non-reference frame image includes a residual image obtained after performing down-sampling on the target image, except for the reference frame image; if the second code stream is incomplete, performing image decoding on the second code stream to obtain a corresponding incomplete non-reference frame image; determining peripheral pixels of the incomplete non-reference frame image according to the incomplete non-reference frame image; reconstructing the target image according to the reference frame image includes: and reconstructing the target image through an interpolation algorithm according to the reference frame image and the peripheral pixels of the incomplete non-reference frame image. It can be understood that, in a preset time period when the first code stream is received, whether a second code stream is received is determined, and if not, the target image is reconstructed according to at least one of a reference frame image corresponding to the first code stream and a non-reference frame image corresponding to a frame of target image adjacent to the target image of the target image; if the second code stream is received, whether the received second code stream is incomplete or not can be determined, if the received second code stream is incomplete, the incomplete second code stream and the complete first code stream can be used for interpolation to obtain a target image, the situation that a receiving end cannot reconstruct a complete video image due to partial information loss is avoided, the phenomena of no sound, blocking and the like of video information are reduced, and the viewing experience of a user is improved. Such as: a time packet may be marked to the code stream when the image coding is performed; after receiving the first code stream, receiving a second code stream within a time period specified by the time packet, wherein whether the second code stream is incomplete or not needs to be determined, and if the second code stream is determined to be incomplete, reconstructing a target image by utilizing an interpolation algorithm and complete first code stream information according to incomplete second code stream information; or the target image can be reconstructed by utilizing an interpolation algorithm according to the incomplete second code stream information and at least one of the reference frame image and the non-reference frame image corresponding to the complete first code stream information and the adjacent frame target image of the target image.

Optionally, the decoder 111 receives the second code stream sent by the encoding end within a preset time period after receiving the first code stream; if the second code stream is complete, performing image decoding on the second code stream to obtain the corresponding non-reference frame image; reconstructing the target image according to the reference frame image includes: and splicing the reference frame image and the non-reference frame image to reconstruct the target image. For example: a time packet may be marked to the code stream when the image coding is performed; after receiving the first code stream, receiving a second code stream within a time period specified by the time packet, and if the second code stream is determined to be complete, directly splicing a reference frame image decoded by the first code stream and a non-reference frame image decoded by the second code stream into a target image.

When the method and the device are implemented, the image coding and decoding algorithm of the embodiment of the application can quickly track and adapt to the change of the channel environment of the wireless channel, so that the delay and the image quality of a receiving end are kept at an acceptable level. Firstly, a coding end adjusts coding parameters during image coding according to channel information of a current wireless channel, so that the code rate of a generated code stream is lower than the channel capacity of the current wireless channel, and further, when the code stream is transmitted in a wireless mode, the code stream cannot lose part of code stream information when the channel capacity is too low to transmit a complete code stream or the channel capacity is jittered or the channel capacity is reduced to be lower than an initial code rate (for example, the initial code rate can be the code rate or a preset code rate when a target image is compressed according to a traditional coding method). Secondly, the code stream received by the encoding end is obtained by down-sampling the target image and then encoding the image. And because the target image can be divided into a plurality of duplicate images with lower resolution ratio by downsampling, the size of a code stream obtained by downsampling and recoding the target image is smaller than that obtained when the target image is directly subjected to image coding, so that the code stream after downsampling can better adapt to the change of a wireless channel, the probability of information loss is reduced in the transmission process, and the problem that a receiving end cannot reconstruct a complete video image due to information loss is avoided. Meanwhile, the adjustment of the sending parameters and the allocation of the channel resources also reduce the probability of information loss in the transmission process, avoid the situation that the receiving end cannot reconstruct a complete video image due to the information loss, reduce the phenomena of mosaic, blockage, blurring and the like of the video information, and improve the watching experience of a user.

The method of the embodiments of the present application is explained in detail above, and the related apparatus of the embodiments of the present application is provided below.

Referring to fig. 4A, fig. 4A is a schematic structural diagram of a low-delay source-channel joint coding apparatus according to an embodiment of the present application, where the low-delay source-channel joint coding apparatus 10 may include a first sampling unit 401, a first coding unit 402, and a first transmitting unit 403, and may further include: a first encoding parameter unit 404, a first acquiring unit 405, a second transmitting unit 406, a second acquiring unit 407, and a third transmitting unit 408, wherein the details of each unit are described below.

A first sampling unit 401 is configured to perform downsampling on the target image to obtain a reference frame image and a non-reference frame image.

The first encoding unit 402 is configured to perform image encoding on the reference frame image and the non-reference frame image, respectively, to obtain a first code stream after encoding the reference frame image and a second code stream after encoding the non-reference frame image.

A first sending unit 403, configured to determine a first channel resource and a second channel resource based on a channel environment of a current wireless channel, send a first code stream using the first channel resource, and send a second code stream using the second channel resource, respectively, where the first channel resource is better than the second channel resource.

In a possible implementation manner, the first encoding unit 402 is specifically configured to perform intra-frame compression on each reference frame image included in the reference frame image to obtain the first code stream; and performing intra-frame compression on the residual error between the non-reference frame image and the reference frame image to obtain the corresponding second code stream.

In one possible implementation, the channel environment further includes a channel capacity; the device further comprises: a first encoding parameter unit 404, configured to determine, based on the channel capacity, encoding parameters corresponding to the reference frame image and the non-reference frame image before the reference frame image and the non-reference frame image are respectively subjected to image encoding to obtain a first code stream after the reference frame image is encoded and a second code stream after the non-reference frame image is encoded, where the encoding parameters are used to control a corresponding image to generate a corresponding code stream according to a target code rate during image encoding, and the target code rates corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity.

In one possible implementation, the channel environment includes one or more of a bandwidth of a channel, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty cycle, and a bit rate; a first sending unit 403, configured to determine, based on the current channel environment, sending parameters corresponding to the first code stream and/or the second code stream, where the sending parameters are used to send the code stream according to target modulation and coding policy information and/or target transmission power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream; and sending the first code stream according to the sending parameters corresponding to the first code stream, and sending the second code stream according to the sending parameters corresponding to the second code stream.

In one possible implementation, the wireless channel includes a first wireless channel and a second wireless channel; the first sending unit 403 is specifically configured to: and determining the first wireless channel and the second wireless channel based on the current channel environment, transmitting the first code stream through the first wireless channel, and transmitting the second code stream through the second wireless channel, wherein the service quality guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the above-mentioned device still includes: a first obtaining unit 405, configured to obtain a code stream of each frame of target image corresponding to the reference frame image in multiple frames of target images included in the target video; acquiring audio information in the target video; a second sending unit 406, configured to send, through the first wireless channel, the code stream and the audio information, where each frame image in the multiple frame images included in the target video corresponds to the reference frame image.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the above-mentioned device still includes: a second obtaining unit 407, configured to obtain a code stream of each frame of target image in multiple frames of target images included in the target video, where the target image corresponds to the non-reference frame image; a third sending unit 408, configured to send, through the second wireless channel, a code stream in which each frame image in the multiple frame images included in the target video corresponds to the non-reference frame image.

It should be noted that, for the functions of the functional units in the low-latency source-channel joint encoding device 20 described in the embodiment of the present application, reference may be made to the related description in the method embodiment described in fig. 2A to fig. 2C, and details are not repeated here.

Referring to fig. 4B, fig. 4B is a schematic structural diagram of another low-latency source channel joint coding apparatus according to an embodiment of the present application, where the low-latency source channel joint coding apparatus 20 may include a second sampling unit 411, a second coding parameter unit 412, a second coding unit 413, and a fourth sending unit 414, and may further include: a third acquiring unit 415, a fifth transmitting unit 416, a fourth acquiring unit 417 and a sixth transmitting unit 418, wherein the details of each unit are as follows.

The second sampling unit 411 is configured to perform downsampling on the target image to obtain a reference frame image and a non-reference frame image.

The second encoding parameter unit 412 is configured to determine, according to a channel environment of the current wireless channel, encoding parameters corresponding to the reference frame image and the non-reference frame image.

And a second encoding unit 413, configured to perform image encoding on the reference frame image and the non-reference frame image according to the encoding parameters, to obtain a first code stream after the reference frame image is encoded and a second code stream after the non-reference frame image is encoded.

A fourth sending unit 414, configured to send the first code stream and the second code stream, respectively.

In a possible implementation manner, the coding parameters are used for controlling the corresponding image to generate a corresponding code stream according to a target code rate when the image coding is performed; the second encoding unit 413 is specifically configured to: performing intraframe compression on each reference frame image included in the reference frame images according to the target code rate to obtain the first code stream; and performing intra-frame compression on the residual error between the non-reference frame image and the reference frame image according to the target code rate to obtain the corresponding second code stream.

In a possible implementation, the channel environment includes one or more of a bandwidth of a channel, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty cycle, and a bit rate; the fourth sending unit 414, before sending the first code stream and the second code stream, is further configured to: and determining sending parameters respectively corresponding to the first code stream and/or the second code stream according to the channel environment, wherein the sending parameters are used for sending the code streams according to target modulation and coding strategy information and/or target transmitting power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream.

In one possible implementation, the wireless channel includes a first wireless channel and a second wireless channel; the fourth sending unit 414 is specifically configured to: and sending the first code stream through a first wireless channel, and sending the second code stream through a second wireless channel, wherein the service quality guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the above-mentioned device still includes: a third obtaining unit 415, configured to obtain a code stream of each frame of target image corresponding to the reference frame image in multiple frames of target images included in the target video; acquiring audio information in the target video; a fifth sending unit 416, configured to send, through the first wireless channel, the code stream and the audio information, where each frame image in the multiple frame images included in the target video corresponds to the reference frame image.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the above-mentioned device still includes: a fourth obtaining unit 417, configured to obtain a code stream of each frame of target image in multiple frames of target images included in the target video, where the target image corresponds to the non-reference frame image; a sixth sending unit 418, configured to send, through the second wireless channel, a code stream in which each frame image in the multiple frame images included in the target video corresponds to the non-reference frame image.

It should be noted that, for the functions of the functional units in the low-latency source-channel joint encoding device 20 described in the embodiment of the present application, reference may be made to the related description in the method embodiment described in fig. 3A to fig. 3C, and details are not repeated here.

Referring to fig. 4C, fig. 4C is a schematic structural diagram of a low-latency source channel joint decoding apparatus according to an embodiment of the present application, where the low-latency source channel joint decoding apparatus 30 may include a receiving unit 421, a decoding unit 422, and an image unit 423, and may further include: a fifth acquisition unit 424 and a sixth acquisition unit 425, wherein the details of each unit are as follows.

The receiving unit 421 is configured to receive a first code stream sent by an encoding end, where the first code stream is a code stream obtained by performing image encoding on a reference frame image, and the reference frame image includes a reference frame image obtained by performing downsampling on a target image.

The decoding unit 422 is configured to perform image decoding on the first code stream to obtain the reference frame image corresponding to the first code stream.

An image unit 423 for reconstructing the target image from the reference frame image.

In a possible implementation manner, the target image is any one of multiple frames of target images included in the target video; the image unit 423 is specifically configured to: and reconstructing the target image through an interpolation algorithm according to at least one of a reference frame image and a non-reference frame image corresponding to one frame of target image adjacent to the target image in the reference frame image.

In one possible implementation, the apparatus further includes: a fifth obtaining unit 424, configured to receive, in a preset time period after receiving the first code stream, a second code stream sent by the encoding end, where the second code stream is a code stream obtained after performing the image encoding on a non-reference frame image, and the non-reference frame image includes a residual image obtained after performing down-sampling on the target image, except for the reference frame image; if the second code stream is incomplete, performing image decoding on the second code stream to obtain a corresponding incomplete non-reference frame image; the image unit 423 is specifically configured to: and reconstructing the target image through an interpolation algorithm according to the peripheral pixels of the reference frame image and the incomplete non-reference frame image.

In one possible implementation, the apparatus further includes: a sixth obtaining unit 425, configured to, if the second code stream is complete, perform image decoding on the second code stream to obtain the corresponding non-reference frame image; the image unit 423 is specifically configured to: and splicing the reference frame image and the non-reference frame image to reconstruct the target image.

As shown in fig. 5, fig. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present disclosure, where the terminal device 40 is an intelligent terminal capable of performing wireless image transmission, such as a projector, a video recorder, a mobile phone, a tablet computer, a vehicle-mounted terminal, and the like, and the device includes at least one processor 501, at least one memory 502, and at least one communication interface 503. In addition, the device may also include common components such as an antenna, which will not be described in detail herein.

The processor 501 may be a general purpose Central Processing Unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a Field Programmable Gate Array (FPGA) circuit, or one or more integrated circuits for controlling the execution of programs according to the above schemes.

Communication interface 503 is used for communicating with other devices or communication Networks, such as ethernet, Radio Access Network (RAN), core network, Wireless Local Area Networks (WLAN), etc.

The Memory 502 may be, but is not limited to, a Read-Only Memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that can store information and instructions, an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical Disc storage, optical Disc storage (including Compact Disc, laser Disc, optical Disc, digital versatile Disc, blu-ray Disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory may be self-contained and coupled to the processor via a bus. The memory may also be integral to the processor.

The memory 502 is used for storing application program codes for executing the above scheme, and is controlled by the processor 501 for execution. The processor 501 is used to execute application program code stored in the memory 502.

The memory 502 stores code that may perform the low latency source-channel joint coding method provided in fig. 2A above, such as: down-sampling the target image to obtain a reference frame image and a non-reference frame image; respectively carrying out image coding on the reference frame image and the non-reference frame image to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded; determining a first channel resource and a second channel resource based on the channel environment of the current wireless channel, and respectively transmitting a first code stream by using the first channel resource and transmitting a second code stream by using the second channel resource, wherein the first channel resource is superior to the second channel resource.

The code stored in the memory 502 may also perform the low latency joint source-channel coding method provided in fig. 3A above, such as: down-sampling the target image to obtain a reference frame image and a non-reference frame image; determining coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel; respectively carrying out image coding on the reference frame image and the non-reference frame image according to the coding parameters to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded; and respectively transmitting the first code stream and the second code stream.

It should be noted that, for the functions of the functional units in the low-latency source-channel joint encoding device 30 described in the embodiment of the present application, reference may be made to the related description in the method embodiment described in fig. 2A to fig. 3C, and details are not repeated here.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the above-described division of the units is only one type of division of logical functions, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of some interfaces, devices or units, and may be an electric or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit may be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as a separate product. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a storage medium, and including several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, and may specifically be a processor in the computer device) to execute all or part of the steps of the above-mentioned method of the embodiments of the present application. The storage medium may include: a U-disk, a removable hard disk, a magnetic disk, an optical disk, a Read-Only Memory (ROM) or a Random Access Memory (RAM), and the like.

The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

Claims

A low-delay source-channel joint coding method, comprising:

down-sampling the target image to obtain a reference frame image and a non-reference frame image;

respectively carrying out image coding on the reference frame image and the non-reference frame image to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded;

determining a first channel resource and a second channel resource based on a channel environment of a current wireless channel, and respectively transmitting the first code stream by using the first channel resource and transmitting the second code stream by using the second channel resource, wherein the first channel resource is superior to the second channel resource.
The method according to claim 1, wherein said performing image coding on the reference frame image and the non-reference frame image respectively to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded comprises:

performing intra-frame compression on each reference frame image included in the reference frame images to obtain the first code stream;

and carrying out intra-frame compression on the residual errors of the non-reference frame image and the reference frame image to obtain the corresponding second code stream.
The method according to claim 1 or 2, wherein the channel environment further comprises a channel capacity; before the step of performing image coding on the reference frame image and the non-reference frame image respectively to obtain the first code stream after the reference frame image is coded and the second code stream after the non-reference frame image is coded, the method further includes:

and determining coding parameters corresponding to the reference frame image and the non-reference frame image based on the channel capacity, wherein the coding parameters are used for controlling the corresponding images to generate corresponding code streams according to target code rates during image coding, and the target code rates respectively corresponding to the first code stream and the second code stream are both less than or equal to the channel capacity.
The method of claim 3, wherein the channel environment comprises one or more of a bandwidth of a channel, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty cycle, a bit rate;

the determining a first channel resource and a second channel resource based on the current channel environment, and respectively transmitting the first code stream by using the first channel resource and the second code stream by using the second channel resource, includes:

determining sending parameters respectively corresponding to the first code stream and/or the second code stream based on the current channel environment, wherein the sending parameters are used for sending the code streams according to target modulation and coding strategy information and/or target transmitting power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream;

and sending the first code stream according to the sending parameters corresponding to the first code stream, and sending the second code stream according to the sending parameters corresponding to the second code stream.
The method of any of claims 1-4, wherein the wireless channel comprises a first wireless channel and a second wireless channel; the determining a first channel resource and a second channel resource based on the current channel environment, and respectively transmitting the first code stream by using the first channel resource and the second code stream by using the second channel resource, includes:

and determining the first wireless channel and the second wireless channel based on the current channel environment, sending the first code stream through the first wireless channel, and sending the second code stream through the second wireless channel, wherein the service quality guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
The method according to claim 5, wherein the target image is any one of a plurality of frames of target images included in the target video; the method further comprises the following steps:

acquiring a code stream of each frame of target image corresponding to the reference frame image in a plurality of frames of target images included in the target video;

acquiring audio information in the target video;

and transmitting the code stream and the audio information of each frame image in the multi-frame images included in the target video corresponding to the reference frame image through the first wireless channel.
The method according to claim 5, wherein the target image is any one of a plurality of frames of target images included in the target video; the method further comprises the following steps:

acquiring a code stream of each frame of target image corresponding to the non-reference frame image in a plurality of frames of target images included in the target video;

and transmitting a code stream of each frame image in the multi-frame images included in the target video corresponding to the non-reference frame image through the second wireless channel.
A low-delay source-channel joint coding method, comprising:

down-sampling the target image to obtain a reference frame image and a non-reference frame image;

determining coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel;

respectively carrying out image coding on the reference frame image and the non-reference frame image according to the coding parameters to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded;

and respectively transmitting the first code stream and the second code stream.
The method of claim 8, wherein the coding parameters are used to control the corresponding image to generate a corresponding code stream according to a target code rate when the image is coded;

the image coding is respectively carried out on the reference frame image and the non-reference frame image according to the coding parameters to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded, and the method comprises the following steps:

performing intraframe compression on each reference frame image included in the reference frame images according to the target code rate to obtain the first code stream;

and carrying out intra-frame compression on the residual errors of the non-reference frame image and the reference frame image according to the target code rate to obtain the corresponding second code stream.
The method according to claim 8 or 9, wherein the channel environment comprises one or more of a bandwidth, a signal-to-interference-plus-noise ratio, a signal-to-noise ratio, a received signal strength indication, a duty cycle, a bit rate of a channel;

before the sending the first code stream and the second code stream respectively, the method further includes:

and determining sending parameters respectively corresponding to the first code stream and/or the second code stream according to the channel environment, wherein the sending parameters are used for sending the code streams according to target modulation and coding strategy information and/or target transmitting power, and the sending parameters of the first code stream are superior to the sending parameters of the second code stream.
The method according to any of claims 8-10, wherein the wireless channel comprises a first wireless channel and a second wireless channel; the respectively sending the first code stream and the second code stream includes:

and sending the first code stream through a first wireless channel, and sending the second code stream through a second wireless channel, wherein the service quality guarantee mechanism of the first wireless channel is higher than that of the second wireless channel.
The method according to claim 11, wherein the target image is any one of a plurality of frames of target images included in the target video; the method further comprises the following steps:

acquiring a code stream of each frame of target image corresponding to the reference frame image in a plurality of frames of target images included in the target video;

acquiring audio information in the target video;

and transmitting the code stream and the audio information of each frame image in the multi-frame images included in the target video corresponding to the reference frame image through the first wireless channel.
The method according to claim 11, wherein the target image is any one of a plurality of frames of target images included in the target video; the method further comprises the following steps:

acquiring a code stream of each frame of target image corresponding to the non-reference frame image in a plurality of frames of target images included in the target video;

and transmitting a code stream of each frame image in the multi-frame images included in the target video corresponding to the non-reference frame image through the second wireless channel.
A low-latency source-channel joint decoding method, comprising:

receiving a first code stream sent by an encoding end, wherein the first code stream is obtained after image encoding is carried out on a reference frame image, and the reference frame image comprises one or more images obtained after down-sampling is carried out on a target image;

performing image decoding on the first code stream to obtain the reference frame image corresponding to the first code stream;

and reconstructing the target image according to the reference frame image.
The method according to claim 14, wherein the target image is any one of a plurality of frames of target images included in the target video;

reconstructing the target image according to the reference frame image includes:

and reconstructing the target image through an interpolation algorithm according to at least one of a reference frame image and a non-reference frame image corresponding to one frame of target image adjacent to the target image in the reference frame image.
The method of claim 14, further comprising:

receiving a second code stream sent by the encoding end within a preset time period after receiving the first code stream, wherein the second code stream is a code stream obtained after image encoding is carried out on a non-reference frame image, and the non-reference frame image comprises a residual image except the reference frame image, which is obtained after down-sampling is carried out on the target image;

if the second code stream is incomplete, performing image decoding on the second code stream to obtain a corresponding incomplete non-reference frame image;

determining peripheral pixels of the incomplete non-reference frame image according to the incomplete non-reference frame image;

reconstructing the target image according to the reference frame image includes:

and reconstructing the target image through an interpolation algorithm according to the reference frame image and the peripheral pixels.
The method of claim 16, further comprising:

if the second code stream is complete, performing image decoding on the second code stream to obtain the corresponding non-reference frame image;

reconstructing the target image according to the reference frame image includes:

and splicing the reference frame image and the non-reference frame image to reconstruct the target image.
A low-delay source-channel joint coding apparatus, comprising: an encoder and a transmitter, wherein,

the encoder is configured to:

down-sampling the target image to obtain a reference frame image and a non-reference frame image;

respectively carrying out image coding on the reference frame image and the non-reference frame image to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded;

the transmitter is configured to:

determining a first channel resource and a second channel resource based on a channel environment of a current wireless channel, and respectively transmitting the first code stream by using the first channel resource and transmitting the second code stream by using the second channel resource, wherein the first channel resource is superior to the second channel resource.
A low-delay source-channel joint coding apparatus, comprising: an encoder and a transmitter, wherein,

the encoder is configured to:

down-sampling the target image to obtain a reference frame image and a non-reference frame image;

determining coding parameters corresponding to the reference frame image and the non-reference frame image according to the channel environment of the current wireless channel;

respectively carrying out image coding on the reference frame image and the non-reference frame image according to the coding parameters to obtain a first code stream after the reference frame image is coded and a second code stream after the non-reference frame image is coded;

the transmitter is configured to: and respectively transmitting the first code stream and the second code stream.
A computer storage medium, characterized in that it stores a computer program which, when executed by a processor, implements the method of any of the preceding claims 1-7 or 8-13.
A computer program, characterized in that the computer program comprises instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1-7 or 8-13.
A low-delay source-channel joint coding system, comprising an encoding side and a decoding side, wherein the encoding side is configured to perform the method according to claims 1-7 or 8-13, and the decoding side is configured to perform the method according to claims 14-17.