TW202218428A - Image encoding method, image decoding method, and related apparatuses - Google Patents

Abstract

Disclosed in embodiments of the present application are an image encoding method, an image decoding method, and related apparatuses. The image decoding method comprises: obtaining an original residual block of a current coding block, the current coding block comprising a currently processed video frame or coding units obtained by dividing the currently processed video frame; obtaining transform features of the current coding block according to the original residual block and a pre-trained feature prediction model; quantizing the transform features of the current coding block to obtain quantized features of the current coding block; determining, by means of a pre-trained probability prediction model, the probability of each pixel in the quantized features of the current coding block; and generating a binary code stream of the current coding block using the probability of each pixel. According to the embodiments of the present application, adaptive dynamic residual compensation is implemented, and different forms of inter-frame residual information can be effectively encoded.

Description

Image coding method, image decoding method and related device

The present application relates to the technical field of electronic devices, and in particular, to an image encoding method, an image decoding method, and related apparatuses.

Digital imaging capabilities can be incorporated into a wide range of devices, including digital televisions, digital live broadcasting systems, wireless broadcasting systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-books Readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, mobile or satellite radio phones, video conferencing devices, video streaming devices, etc.

Digital video devices implement video compression techniques, such as those provided by MPEG-2, MPEG-4, ITU-TH.263, ITU-TH.264/MPEG-4 Part 10 Advanced Video Coding (AVC), ITU- The standard defined by the TH.265 high efficiency video coding (HEVC) standard and those video compression techniques described in extensions to the standard to transmit and receive digital video messages more efficiently. Video devices can transmit, receive, encode, decode and/or store digital video information more efficiently by implementing these video codec techniques.

With the proliferation of online video, despite the continuous evolution of digital video compression technology, it still puts forward higher requirements for video compression ratio.

The embodiments of the present application provide an image encoding method, an image decoding method, and a related device, so as to realize self-adjusting dynamic residual compensation, and can effectively encode different forms of inter-frame residual information.

In a first aspect, an embodiment of the present application provides an image encoding method, including:

obtaining an original residual block of a current coding block, where the current coding block includes a currently processed image frame or a coding unit obtained by dividing the currently processed image frame;

Obtain the transform feature of the current coding block according to the original residual block and the pre-trained feature prediction model;

quantizing the transform feature of the current coding block to obtain the quantization feature of the current coding block;

Determine the probability of each pixel in the quantization feature of the current coding block through a pre-trained probability prediction model;

A binary bit stream for the current coding block is generated using the probability of each pixel.

Compared with the prior art, the solution of the present application performs self-adjusting dynamic residual compensation on the current prediction frame and obtains the final inter-frame reconstruction, which can effectively encode different forms of inter-frame residual information.

In a second aspect, an embodiment of the present application provides an image decoding method, including:

obtaining a binary bit stream of a current decoding block, where the current decoding block includes a bit stream of a currently processed image frame or a decoding unit obtained by dividing the currently processed image frame;

Transforming the binary bit stream into a quantized feature of the current decoding block through a pre-trained probabilistic prediction model;

Determine the residual block of the current decoding block according to the quantized feature and the pre-trained residual prediction model;

A reconstructed block of the current decoding block is determined according to the residual block and the prediction block of the current decoding block.

Compared with the prior art, the solution of the present application performs self-adjusting dynamic residual compensation on the current prediction frame and obtains the final inter-frame reconstruction, which can effectively encode different forms of inter-frame residual information.

In a third aspect, an embodiment of the present application provides an image encoding apparatus, including:

an obtaining unit, configured to obtain an original residual block of a current coding block, where the current coding block includes a currently processed image frame or a coding unit obtained by dividing the currently processed image frame;

The first prediction unit is used to obtain the transform feature of the current coding block according to the original residual block and the pre-trained feature prediction model;

a quantization unit, configured to quantize the transform feature of the current coding block to obtain the quantization feature of the current coding block;

The second prediction unit is used to determine the probability of each pixel in the quantization feature of the current coding block through a pre-trained probability prediction model;

A generating unit, configured to generate a binary bit stream of the current coding block by using the probability of each pixel.

In a fourth aspect, an embodiment of the present application provides an image decoding apparatus, including:

an acquisition unit, configured to acquire a binary bit stream of a current decoding block, where the current decoding block includes a bit stream of a currently processed image frame or a decoding unit obtained by dividing the currently processed image frame;

a first prediction unit for converting the binary bit stream into a quantized feature of the current decoding block through a pre-trained probability prediction model;

a second prediction unit, configured to determine the residual block of the current decoding block according to the quantized feature and the pre-trained residual prediction model;

A determination unit, configured to determine the reconstructed block of the current decoding block according to the residual block and the prediction block of the current decoding block.

In a fifth aspect, an embodiment of the present application provides an encoder, including: a processor and a memory coupled to the processor; the processor is configured to execute the method described in the first aspect.

In a sixth aspect, an embodiment of the present application provides a decoder, including: a processor and a memory coupled to the processor; the processor is configured to execute the method described in the second aspect above.

In a seventh aspect, an embodiment of the present application provides a terminal, the terminal includes: one or more processors, a memory, and a communication interface; the memory and the communication interface are connected to the one or more processors ; the terminal communicates with other devices through the communication interface, the memory is used to store computer program code, the computer program code includes instructions, when the one or more processors execute the instructions, the The terminal executes the method according to the first aspect or the second aspect.

In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, where an instruction is stored in the computer-readable storage medium, and when the instruction is executed on a computer, the computer is made to execute the above-mentioned first aspect or the second aspect the method described.

In a ninth aspect, an embodiment of the present application provides a computer program product including instructions, which, when the instructions are run on a computer, cause the computer to execute the method described in the first aspect or the second aspect.

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

It will be understood that the terms "first", "second", etc., as used herein, may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, a first user terminal may be referred to as a second user terminal, and similarly, a second user terminal may be referred to as a first user terminal without departing from the scope of the present invention. Both the first client and the second client are clients, but they are not the same client.

First, terms and related technologies used in the embodiments of the present application are introduced.

A complete image in a video is usually called a "frame", and a video composed of many frames in time sequence is also called a video sequence. There are a series of redundant information in the image sequence, such as spatial redundancy, temporal redundancy, visual redundancy, information entropy redundancy, structural redundancy, knowledge redundancy, and importance redundancy. In order to remove redundant information in the image sequence as much as possible and reduce the amount of data representing the image, a video coding (Video Coding) technology is proposed to achieve the effect of reducing storage space and saving transmission bandwidth. Video coding technology is also called video compression technology.

As far as the current technological development is concerned, image coding technologies mainly include intra-frame prediction, inter-frame prediction, transform and quantization, entropy coding, and deblocking filtering. In the international scope, video compression coding standards, such as: MPEG-2 and MPEG-4 Part 10 Advanced Video Coding (AVC) formulated by Motion Picture Experts Group (MPEG) , in the H.263, H.264 and H.265 High Efficiency Video Coding standard (HEVC) formulated by the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T), There are four main compression coding methods: chroma sampling, predictive coding, transform coding and quantization coding.

Predictive coding: Use the data information of previously coded frames to predict the current frame to be coded. The encoder obtains a predicted value through prediction, and there is a certain residual value between the predicted value and the actual value. If the prediction is more suitable, the predicted value will be closer to the actual value, and the residual value will be smaller, so that the encoding end can greatly reduce the amount of data by encoding the residual value. When decoding, the decoding end uses the residual value plus the prediction value to restore and reconstruct the original image. In mainstream coding standards, predictive coding is divided into two basic types: intra-frame prediction and inter-frame prediction.

Inter prediction is a prediction technology based on motion compensation. The main processing is to determine the motion information of the current block, obtain a reference image block from a reference frame of the current block according to the motion information, and generate a prediction image of the current block. The current block is performed using one of forward prediction, backward prediction or bidirectional prediction, the prediction direction is indicated by the inter-frame prediction direction in the motion information, and the reference image block used to predict the current block in the reference frame is relative to the current block. The displacement vector of is indicated by the motion vector in the motion information, and one motion vector corresponds to one reference frame. The inter prediction of an image block can use only one motion vector to generate a predicted image using pixels in one reference frame, which is called unidirectional prediction; it can also use two motion vectors to use pixels in two reference frames. to combine to generate predicted images, called bidirectional prediction. That is, an image block may generally contain one or two motion vectors. For some multi-hypothesis inter prediction techniques, an image block may contain more than two motion vectors.

The inter-frame prediction indicates the reference frame (reference frame) through the reference frame index (reference index, ref_idx), and indicates the position offset of the reference block (reference block) of the current block in the reference frame relative to the current block through the motion vector (motion vector, MV). shift. An MV is a two-dimensional vector, including a horizontal displacement component and a vertical displacement component; an MV corresponds to two frames, and each frame has a picture order count (POC), which is used to indicate that the image is in the The numbers are displayed in order, so an MV also corresponds to a POC difference. The POC difference has a linear relationship with the time interval. The scaling of the motion vector usually adopts the scaling method based on the POC difference, and converts the motion vector between one pair of images into the motion vector between the other pair of images.

There are two commonly used inter prediction modes.

1) Advanced motion vector prediction (AMVP) mode: identifies the inter prediction direction (forward, backward or bidirectional), reference frame index (reference index), motion vector used by the current block in the bitstream Predictor index (motion vector predictor index, MVP index), motion vector residual value (motion vector difference, MVD); the reference frame queue used is determined by the inter prediction direction, and the reference frame pointed to by the current block MV is determined by the reference frame index frame, an MVP in the MVP list is indicated by the motion vector predictor index as the predictor of the MV of the current block, and an MVP is added to an MVD to obtain an MV.

2) Merge/skip mode: the merge index is identified in the bit stream, and a merge candidate is selected from the merge candidate list according to the merge index. ), the motion information (including prediction direction, reference frame, and motion vector) of the current block is determined by this merge candidate. The main difference between the merge mode and the skip mode is that the merge mode implies that the current block has residual information, while the skip mode implies that the current block has no residual information (or the residual is 0); these two modes export motion information. The way is the same.

The fusion candidate is specifically a motion information data structure, including various information such as inter-frame prediction direction, reference frame, and motion vector. The current block can select the corresponding fusion candidate from the merge candidate list according to the merge index, and use the motion information of the fusion candidate as the motion information of the current block, or use the motion information of the fusion candidate as the motion information of the fusion candidate. After scaling, it is used as the motion information of the current block. In the HEVC standard, a fusion candidate can be the motion information of image blocks adjacent to the current block, which is called a spatial merge candidate; it can also be a position image corresponding to the current block in another coded image The motion information of the block is called a temporal merge candidate. In addition, the fusion candidate may also be a bi-predictive merge candidate composed of forward motion information of one fusion candidate and backward motion information of another fusion candidate, or a motion vector forced A zero motion vector merge candidate that is a 0 vector.

The division of the inter-frame prediction unit includes a 2N×2N division method (as shown in A in FIG. 4 ), an N×N division method (as shown in B in FIG. 4 ), and an N×2N division method (as shown in FIG. 4 ). C in Figure 4), 2N×N division (as shown in D in Figure 4), 2N×nD division (as shown in E in Figure 4), 2N×nU division (as shown in Figure 4 shown in F), nL×2N division method (as shown in G in FIG. 4 ), and nR×2N division method (as shown in H in FIG. 4 ). Among them, N is any positive integer, n=x×N, 0≤x≤1.

The 2N×2N division method is to not divide the image block; the N×N division method is to divide the image block into four sub-image blocks of equal size; the N×2N division method is to divide the image block into two left and right sub-blocks. sub-image blocks of equal size; the 2N×N division method is to divide the image block into two equal-sized sub-image blocks; the 2N×nD division method is to divide the image block into two upper and lower sub-image blocks block, and the image dividing line moves down by n relative to the vertical bisector of the image block, where D indicates that the image dividing line moves down relative to the vertical bisector of the image block; the 2N×nU division method is to divide the image block It is divided into upper and lower sub-image blocks, and the image division line is shifted relative to the vertical bisector of the image block by n, where U means that the image division line is moved relative to the vertical bisector of the image block; nL×2N division The method is to divide the image block into left and right sub-image blocks, and the image division line is shifted to the left by n relative to the vertical bisector of the image block, where L represents the vertical bisector of the image division line relative to the image block. The line is shifted to the left; the nR×2N division method is to divide the image block into left and right sub-image blocks, and the image division line is shifted to the right relative to the vertical bisector of the image block by n, where R represents the image division line Shifts right relative to the vertical bisector of the image block.

For the division of images, in order to represent the video content more flexibly, the High Efficiency Video Coding standard (HEVC) technology defines a coding tree unit (CTU), a coding block (Coding Unit, CU) ), prediction unit (Prediction Unit, PU) and transform unit (Transform Unit, TU). CTU, CU, PU and TU are all image blocks.

Coding tree unit CTU, an image is composed of multiple CTUs, and a CTU usually corresponds to a square image area, including luma pixels and chroma pixels in this image area (or can only contain luma pixels, or also can only contain chroma pixels); the CTU also contains syntax elements that indicate how to divide the CTU into at least one coding block (coding unit, CU), and the method for decoding each coding block to obtain a reconstructed image. As shown in FIG. 1 , an image 1 is composed of a plurality of CTUs (including CTU A, CTU B, CTU C, etc.). The encoded information corresponding to a certain CTU contains the luminance and/or chrominance values of the pixels in the square image area corresponding to the CTU. In addition, the encoded information corresponding to a certain CTU may also include syntax elements indicating how to divide the CTU into at least one CU, and a method of decoding each CU to obtain a reconstructed image. An image area corresponding to one CTU may include 64×64, 128×128 or 256×256 pixels. In one example, a 64x64 pixel CTU contains a rectangular pixel lattice of 64 columns of 64 pixels, each pixel containing a luma component and/or a chrominance component. A CTU can also correspond to a rectangular image area or an image area of other shapes, and an image area corresponding to a CTU can also be an image area with a different number of pixels in the horizontal direction and the number of pixels in the vertical direction, for example, including 64 ×128 pixels.

Coding block CU, usually corresponding to an A×B rectangular area in the image, including A×B luminance pixels or/and its corresponding chrominance pixels, A is the width of the rectangle, B is the height of the rectangle, A and B can be The same or different, the values of A and B are usually integer powers of 2, such as 128, 64, 32, 16, 8, and 4. The width involved in the embodiments of the present application refers to the length along the X-axis direction (horizontal direction) in the two-dimensional rectangular coordinate system XoY shown in FIG. 1 , and the height refers to the two-dimensional rectangular coordinate system XoY shown in FIG. 1 . The length along the Y-axis direction (vertical direction) in the middle. The reconstructed image of a CU can be obtained by adding the predicted image and the residual image. The predicted image is generated through intra-frame prediction or inter-frame prediction, and can be specifically composed of one or more prediction blocks (PB), The residual image is generated by performing inverse quantization and inverse transform processing on the transform coefficients, and may be specifically composed of one or more transform blocks (transform blocks, TB). Specifically, a CU contains encoded information, and the encoded information includes information such as prediction mode and transform coefficients. According to the encoded information, the CU is subjected to corresponding decoding processes such as prediction, inverse quantization, and inverse transformation, and a reconstructed image corresponding to the CU is generated. The relationship between the coding tree unit CTU and the coding block CU is shown in FIG. 2 .

Digital image compression technology works on image sequences whose color coding method is YCbCr, also known as YUV, and whose color format is 4:2:0, 4:2:2 or 4:4:4. Among them, Y represents the brightness (Luminance or Luma), that is, the grayscale value, Cb represents the blue chrominance component, Cr represents the red chrominance component, and U and V represent the chrominance (Chrominance or Chroma), which is used to describe color and saturation. Spend. In the color format, 4:2:0 means that every 4 pixels has 4 luminance components and 2 chrominance components (YYYYCbCr), and 4:2:2 means that every 4 pixels has 4 luminance components and 4 chrominance components. component (YYYYCbCrCbCr), and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr), Figure 3 shows the distribution of each component in different color formats, where the circle is the Y component and the triangle is the UV component.

The prediction unit PU is the basic unit of intra prediction and inter prediction. The motion information that defines the image block includes inter prediction direction, reference frame, motion vector, etc. The image block that is being encoded is called the current coding block (CCB), and the image block that is being decoded is called the current coding block (CCB). is the current decoding block (CDB), for example, when prediction processing is being performed on an image block, the current coding block or the current decoding block is the prediction block; when residual processing is being performed on an image block, the current coding block Or the current decoding block is a transform block. The picture in which the current coding block or the current decoding block is located is called the current frame. In the current frame, the image blocks located on the left or upper side of the current block may be inside the current frame and have completed the encoding/decoding process to obtain a reconstructed image, which are called reconstructed blocks; the encoding mode of the reconstructed block, reconstruction Information such as pixels is available. A frame that has completed encoding/decoding before the current frame is encoded/decoded is called a reconstructed frame. When the current frame is a unidirectional prediction frame (P frame) or a bidirectional prediction frame (B frame), it has one or two reference frame lists respectively, and the two lists are called L0 and L1 respectively, and each list contains at least one reconstructed frame. frame, referred to as the reference frame for the current frame. The reference frame provides reference pixels for inter prediction of the current frame.

The transform unit TU processes the residuals of the original image block and the predicted image block.

Pixels (also known as pixel points) refer to pixels in an image, such as pixels in coding blocks, pixels in luminance component pixel blocks (also known as luminance pixels), and pixels in chrominance component pixel blocks points (aka chroma pixels), etc.

The sample (also known as pixel value, sample value) refers to the pixel value of a pixel point. The pixel value in the luminance component domain specifically refers to the brightness (ie grayscale value), and the pixel value in the chrominance component domain specifically refers to the color. The degree value (ie color and saturation), according to the different processing stages, the sample of a pixel specifically includes the original sample, the predicted sample and the reconstructed sample.

At present, with the development and maturity of deep learning, image processing and coding based on deep learning have been widely studied. Through a data-driven approach and end-to-end learning, deep neural networks can optimize the entire system end-to-end based on bit rate distortion. The convolutional neural network adopts learnable feature transformation, can be differentiated and quantified, and the dynamic probability distribution estimation can more efficiently remove the redundancy between image images and obtain a more compact image image feature space representation. A higher reconstruction quality can be obtained at low rates. At the same time, hardware acceleration and development based on a specific neural network is conducive to further promoting the acceleration and implementation of learning-based encoding and decoding systems. However, due to the complexity of image encoding and decoding, implementing a complete end-to-end learning-based image encoding method is still an urgent problem to be solved in this field. The optimization and analysis of each specific module and its impact on the entire end-to-end system There is still great uncertainty and research value. The standard work for the end-to-end image coding system based on learning has just started at home and abroad, and MPEG and AVS are basically in the stage of call for evidence for the standardization of smart coding.

In the existing end-to-end system solutions, end-to-end intra-frame coding is directly used to process residual information, without considering the particularity of residual information and the uneven distribution after prediction, and there is no embedded residual sparsification method to approximate traditional coding. skip mode in the method.

In response to the above problems, the embodiments of the present application provide an image encoding method, an encoding method, and a related apparatus. The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

FIG. 5 is a block diagram of an example encoding/decoding system 1 described in the embodiments of the application. The encoding/decoding system 1 includes an image encoder 100 and an image decoder 200. The image encoder 100 and the image decoder 200 are used to implement the present invention. This paper proposes a learning-based end-to-end self-adjusting inter-frame residual coding method.

As shown in FIG. 5 , the codec system 1 includes a source device 10 and a destination device 20 . Source device 10 generates encoded video data. Accordingly, the source device 10 may be referred to as an image encoding device. Destination device 20 may decode the encoded image data generated by source device 10 . Therefore, the destination device 20 may be referred to as an image decoding device. Various implementations of source device 10, destination device 20, or both may include one or more processors and memory coupled to the one or more processors. The memory may include, but is not limited to, RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store the desired code in the form of instructions or data structures that can be accessed by a computer, as described herein.

Source device 10 and destination device 20 may include a variety of devices, including desktop computers, mobile computing devices, notebook (eg, laptop) computers, tablet computers, set-top boxes, telephone handhelds such as so-called "smart" phones, etc. computer, television, camera, display device, digital media player, video game console, car computer or the like.

Destination device 20 may receive encoded image data from source device 10 via link 30 . Link 30 may include one or more media or devices capable of moving encoded image data from source device 10 to destination device 20 . In one example, link 30 may include one or more communication media that enable source device 10 to transmit encoded image data directly to destination device 20 in real time. In this example, source device 10 may modulate the encoded image data according to a communication standard, such as a wireless communication protocol, and may transmit the modulated image data to destination device 20 . The one or more communication media may include wireless and/or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (eg, the Internet). The one or more communication media may include routers, switches, base stations, or other equipment that facilitates communication from source device 10 to destination device 20 . In another example, the encoded data may be output from output interface 140 to storage device 40 .

The image encoding and decoding techniques of the present application can be applied to image encoding and decoding to support a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, streaming image transmission (eg, via the Internet), storage in Encoding of image data on a data storage medium, decoding of image data stored on a data storage medium, or other applications. In some examples, the codec system 1 may be used to support one-way or two-way video transmission to support applications such as video data streaming, video playback, video broadcasting, and/or video telephony.

The codec system 1 illustrated in FIG. 5 is merely an example, and the techniques of this application may be applicable to image decoding setups (eg, image encoding or image decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from local memory, data is streamed over a network, and so on. The image encoding device may encode the data and store the data to memory, and/or the image decoding device may retrieve the data from the memory and decode the data. In many instances, encoding and decoding is performed by devices that do not communicate with each other but only encode data to and/or retrieve data from memory and decode the data.

In the example of FIG. 5 , source device 10 includes video source 120 , video encoder 100 , and output interface 140 . In some examples, output interface 140 may include a modulator/demodulator (modem) and/or a transmitter. Image source 120 may include an image capture device (eg, a camera), an image archive containing previously captured image data, an image feed interface for receiving image data from image content providers, and/or a computer for generating image data A graphics system, or a combination of such sources of image data.

Image encoder 100 may encode image data from image source 120 . In some examples, source device 10 transmits the encoded image data directly to destination device 20 via output interface 140 . In other examples, the encoded image data may also be stored on storage device 40 for later access by destination device 20 for decoding and/or playback.

In the example of FIG. 5 , destination device 20 includes input interface 240 , video decoder 200 and display device 220 . In some examples, input interface 240 includes a receiver and/or modem. Input interface 240 may receive encoded image data via link 30 and/or from storage device 40 . The display device 220 may be integrated with the destination device 20 or may be external to the destination device 20 . Generally, display device 220 displays decoded image data. The display device 220 may include various display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

Although not shown in FIG. 5, in some aspects video encoder 100 and video decoder 200 may each be integrated with an audio encoder and decoder, and may include appropriate multiplexer-demultiplexer units or other Hardware and software to handle the encoding of both audio and video in a common data flow or separate data flows.

Image encoder 100 and image decoder 200 may each be implemented as any of a variety of circuits such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), Field Programmable Gate Array (FPGA), discrete logic, hardware, or any combination thereof. If the application is implemented in part in software, a device may store instructions for the software in a suitable non-volatile computer-readable storage medium and execute the instructions in hardware using one or more processors Thus, the technology of the present application is implemented. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors. Each of image encoder 100 and image decoder 200 may be included in one or more encoders or decoders, either of which may be integrated as a combined encoder in the respective device /decoder (codec) part.

FIG. 6 is an exemplary block diagram of an image encoder 100 described in an embodiment of the present application. The video encoder 100 is used to output the video to the post-processing entity 41 . Post-processing entity 41 represents an example of an image entity that can process encoded image data from image encoder 100, such as a media aware network element (MANE) or a stitching/editing device. In some cases, post-processing entity 41 may be an instance of a network entity. In some image encoding systems, post-processing entity 41 and image encoder 100 may be parts of separate devices, while in other cases, the functionality described with respect to post-processing entity 41 may be implemented by the same device that includes image encoder 100 implement. In an example, post-processing entity 41 is an example of storage device 40 of FIG. 1 .

In the example of FIG. 6 , image encoder 100 includes prediction processing unit 108 , filter unit 106 , memory 107 , summer 112 , transformer 101 , quantizer 102 , and entropy encoder 103 . The prediction processing unit 108 includes an inter predictor 110 and an intra predictor 109 . For image block reconstruction, the image encoder 100 further includes an inverse quantizer 104 , an inverse transformer 105 and a summer 111 . Filter unit 106 represents one or more in-loop filters, such as deblocking filters, adaptive in-loop filters (ALF), and sample adaptive offset (SAO) filters. Although the filter unit 106 is shown in FIG. 6 as an in-loop filter, in other implementations, the filter unit 106 may be implemented as a post-loop filter. In an example, the image encoder 100 may further include an image data memory and a division unit (not shown in the figure).

FIG. 7 is an exemplary block diagram of an image decoder 200 described in an embodiment of the present application. In the example of FIG. 7 , image decoder 200 includes entropy decoder 203 , prediction processing unit 208 , inverse quantizer 204 , inverse transformer 205 , summer 211 , filter unit 206 , and memory 207 . Prediction processing unit 208 may include inter predictor 210 and intra predictor 209 . In some examples, image decoder 200 may perform a decoding process that is substantially the reciprocal of the encoding process described with respect to image encoder 100 from FIG. 6 .

During the decoding process, image decoder 200 receives from image encoder 100 an encoded image bitstream representing image blocks of an encoded image slice and associated syntax elements. The image decoder 200 may receive image data from the network entity 42, and optionally, may also store the image data in an image data memory (not shown in the figure). Image data memory may store image data, such as an encoded image bitstream, to be decoded by the components of image decoder 200 . The image data stored in the image data memory can be obtained, for example, from the storage device 40, from a local image source such as a camera, through wired or wireless network communication of the image data, or by accessing a physical data storage medium. The video data memory may act as a decoded picture buffer (CPB) for storing encoded video data from the encoded video bitstream.

Network entity 42 may be, for example, a server, a MANE, a video editor/splicer, or other such device for implementing one or more of the techniques described above. Network entity 42 may or may not include a video encoder, such as video encoder 100 . Before network entity 42 sends the encoded video bitstream to video decoder 200, network entity 42 may implement portions of the techniques described in this application. In some video decoding systems, network entity 42 and video decoder 200 may be part of separate devices, while in other cases, functionality described with respect to network entity 42 may be performed by the same device that includes video decoder 200 .

It should be understood that other structural variations of the video decoder 200 may be used to decode the encoded video bitstream. For example, the image decoder 200 may generate an output image stream without being processed by the filter unit 206; or, for some image blocks or image frames, the entropy decoder 203 of the image decoder 200 does not decode quantized coefficients, Accordingly, processing by inverse quantizer 204 and inverse transformer 205 is not required.

FIG. 8A is a schematic flowchart of an image encoding method in an embodiment of the present application, and the image encoding method may be applied to the source device 10 in the encoding/decoding system 1 shown in FIG. 5 or the image encoder 100 shown in FIG. 6 . The flow shown in FIG. 8A is described by taking the execution subject as the video encoder 100 shown in FIG. 6 as an example. As shown in FIG. 8A , the image coding method provided by the embodiment of the present application includes:

Step S110: Obtain an original residual block of a current coding block, where the current coding block includes a currently processed image frame or a coding unit obtained by dividing the currently processed image frame.

Wherein, the division manner of the coding unit includes various division manners as shown in FIG. 4 , which is not uniquely limited here.

In the specific implementation, for the case where the current coding block is the currently processed image frame, since the minimum data processing object is a single frame image, the processing efficiency of this method is higher, but the accuracy and performance are lost to a certain extent.

For the case where the current coding block is a coding unit obtained by dividing the currently processed image frame, since the minimum data processing granularity is the coding unit after the division, the overall algorithm processing complexity becomes higher and the processing time becomes longer , but with relatively high accuracy and performance.

Step S120: Obtain the transform feature of the current coding block according to the original residual block and the pre-trained feature prediction model.

Specifically, the feature prediction model can realize data processing through the image processor GPU of the local device, and can adopt any commonly used neural network architecture, such as deep neural network (DNN), support vector machine, etc. The model inputs are residual blocks and outputs are transformed features.

Step S130: Quantize the transform feature of the current coding block to obtain the quantized feature of the current coding block.

Step S140: Determine the probability of each pixel in the quantized feature of the current coding block through a pre-trained probability prediction model.

Among them, in the arithmetic coding process, for each pixel to be encoded, it is necessary to predict the probability of the corresponding pixel (a value between 0 and 1), and its probability can represent the frequency that the current pixel prediction may occur, and the predicted probability The higher it is, the higher the frequency it may appear, and the smaller the bit stream generated during arithmetic coding.

Step S150, using the probability of each pixel to generate a binary bit stream of the current coding block.

In this possible example, the obtaining the original residual block of the current coding block includes: determining the prediction block of the current coding block; comparing the prediction block of the current coding block with the original image of the current coding block The block is differenced to obtain the original residual block.

進行數值變換並量化，從原（0，1）的連續浮點分佈，生成（0，255）的離散分佈

，與當前編碼塊

做差得到整數訊號殘差

，

。 In the specific implementation, the prediction block based on the current coding block

Perform numerical transformation and quantization to generate (0, 255) discrete distribution from the original (0, 1) continuous floating point distribution

, with the current encoding block

Do the difference to get the integer signal residual

,

.

In this possible example, the performing the difference between the prediction block of the current coding block and the original image block of the current coding block to obtain the original residual block includes: according to the prediction of the current coding block The block is numerically transformed and quantized to generate a discrete distribution of the predicted block; the discrete distribution of the predicted block is differentiated from the original image block of the current coding block to obtain the original residual block of integer signals.

In this possible example, obtaining the transform feature of the current coding block according to the original residual block and the pre-trained feature prediction model includes: renormalizing the original residual block, obtaining a normalized first residual block; performing sparse processing on the first residual block to obtain a processed second residual block; inputting the second residual block into the pre-trained feature prediction model to obtain the transform feature of the current coding block.

In the specific implementation, the energy-based renormalization is used to uniformly normalize the residuals of different distributions after prediction to be between (-1, 1). For different image sequences, the energy-based normalization can unify the data. The distribution makes training more stable.

In addition, energy-based renormalization can use other normalization methods, such as 0-1 normalization, linear function normalization, etc., the goal is to unify the residual distribution with large variance after prediction, speed up Model training and convergence speed.

It can be seen that in this example, the threshold sparseness can allocate more code rates in the moving boundary, occlusion and other areas in the end-to-end encoding under the same code rate constraint, saving more code rates required for the background area. In addition, , the energy-based renormalization can speed up the training and convergence of the model, making the model more robust to different residual distributions.

In this possible example, the re-normalizing the original residual block to obtain a normalized first residual block includes: according to an energy unification mechanism, The residual distribution converges to the same distribution space, and the normalized first residual block is obtained.

In this possible example, according to the energy unification mechanism, the different residual distributions of the original residual blocks are converged to the same distribution space to obtain a normalized first residual block, including:

extracting the minimum pixel value x _min and the maximum pixel value x _max in the original residual block;

Normalize the original residual block to the interval (0, 1) by the following formula;

表示初次變換後的像素值，

表示歸一化前的像素值； in,

represents the pixel value after the initial transformation,

Represents the pixel value before normalization;

進行二次變換，得到處於區間（-1，1）的殘差連續分佈，即歸一化後的第一殘差塊，

by the following formula

Perform secondary transformation to obtain a continuous distribution of residuals in the interval (-1, 1), that is, the normalized first residual block,

表示歸一化後的像素值。 in,

Indicates the normalized pixel value.

In this possible example, the performing thinning processing on the first residual block to obtain the processed second residual block includes: acquiring a preset threshold set, where the preset threshold set includes multiple thresholds ; Screen the target threshold for adapting the current coding block from the preset threshold set; traverse the residual samples of each pixel in the first residual block, and compare the residual samples of pixels whose residual samples are less than the target threshold. The residual samples are set to zero to obtain the processed second residual block.

表示歸一化前的像素值，m1表示預設閾值集合中的第一個閾值，m _n表示表示預設閾值集合中的第n個閾值，不同閾值處理後，生成的殘差圖有著不同的稀疏性，閾值越大得到的殘差越稀疏，同時表示需要編碼的殘差空間區間越小。透過遍歷預設閾值集合，可以準確篩選出最適合當前幀殘差編碼的閾值，提高編碼效率。 In the specific implementation, the target threshold can be obtained by the following methods: starting from the smallest threshold in the preset threshold set, performing bit rate distortion optimization for each threshold at the encoding end to obtain the corresponding result, and selecting the optimal one from the results The threshold corresponding to the result is used as the most suitable threshold for residual coding of the current frame. The performing bit rate distortion optimization on each threshold means that each time a threshold is selected, one encoding and decoding is required to obtain a corresponding result, and the optimal result is selected from the final result. As shown in Figure 8B,

Represents the pixel value before normalization, m1 represents the first threshold in the preset threshold set, and m _n represents the nth threshold in the preset threshold set. After different thresholds are processed, the generated residual maps have different Sparsity, the larger the threshold, the sparser the residual, and the smaller the residual space interval that needs to be encoded. By traversing the preset threshold set, the most suitable threshold for residual coding of the current frame can be accurately screened, thereby improving coding efficiency.

In the specific implementation, different thresholds are set, and the normalized residuals are sparsed, so that more effective information can be allocated to the effective pixels.

It should be noted that the threshold-based sparsification is based on the traditional mode selection method. The skip mode is implemented to adjust the coding residual information by itself. The threshold sparsification here can be directly operated on the quantized features.

It can be seen that in this example, threshold sparseness can allocate more code rates in the moving boundary, occlusion and other areas in the end-to-end encoding under the same code rate constraint, saving more code rates required for background areas.

In this possible example, each of the plurality of thresholds is obtained by uniformly sampling the pixels of the current coding block according to a preset sampling interval.

The value range of the sampling interval is determined by: generating a residual histogram of the numerical distribution according to the residual distribution of the current frame, and obtaining the interval corresponding to the peak part of 1/α of the residual distribution.

Wherein, the value of α can be 4, 6, 8, etc., which is not uniquely limited here.

In addition, in other possible examples, each of the plurality of thresholds is obtained by non-uniformly sampling the pixels of the current coding block according to a preset sampling interval, and under normal conditions, no more than 4 thresholds can better Balancing complexity and performance.

In this possible example, the quantizing the transform feature of the current coding block to obtain the quantization feature of the current coding block includes: using a differentiable quantization mechanism for the transform feature of the current coding block, The point feature is transformed into a quantized integer feature to obtain the quantized feature of the current coding block.

In the specific implementation, a differentiable quantization method is used for the extracted features, and the floating-point (floating32) features are transformed into quantized integer features; the specific method is forward calculation.

為四捨五入函數，

為正負

的均值雜訊分佈；反向傳播把此函數近似為線性函數，用1作為反向求導的梯度。 here,

is the rounding function,

positive or negative

The mean noise distribution of ; backpropagation approximates this function as a linear function, using 1 as the gradient of the reverse derivation.

In this possible example, as shown in FIG. 8C , the feature prediction model includes a first branch and a second branch, and the first branch and the second branch are connected in parallel; the first branch It includes three cascaded residual extraction modules and one downsampling module; the second branch includes three cascaded residual extraction modules, one downsampling module and one startup module.

Among them, the residual extraction module can use any mainstream neural network module, such as residual block, dense connection block, etc., the downsampling module uses a convolution kernel with stride; the other branch uses a cascaded volume The stacked layer extracts features and starts with the sigmoid function to obtain a spatial-channel wise self-tuning mask, and performs self-tuning on the extracted features. The upsampling module can be implemented by transposed convolution.

In a specific implementation, the residual extraction module is used for feature extraction for the input residual block, and multiple residual extraction modules are used to extract multiple features for stacking, thereby realizing cascade feature extraction.

It should be noted that the first branch is the main feature extraction module, the module after the sigmoid of the second branch is the self-attention mapping module, and the outputs of the two branches are multiplied to generate the final transformation feature. .

In addition, in the training process of the feature prediction model, the code rate and the loss function can be determined in the following manner.

得到，R為碼率約束的損失，P為所述量化後的變換特徵中每個像素的概率； The code rate is estimated through the formula

Obtain, R is the loss of the code rate constraint, and P is the probability of each pixel in the quantized transform feature;

，D(.)為均方誤差MSE函數或者L2損失函數，

為當前編碼塊的預測塊，

為前編碼塊，整數訊號殘差

，

為當前編碼塊的預測塊的離散分佈； loss function

, D(.) is the mean square error MSE function or L2 loss function,

is the prediction block of the current coding block,

is the pre-coding block, the integer signal residual

,

is the discrete distribution of the prediction block of the current coding block;

，L為每一幀的重建損失，R為碼率約束的損失，透過調整

，訓練得到不同碼率的特徵預測模型。 use rate-distortion optimization for the bit rate and the loss function

, L is the reconstruction loss of each frame, R is the loss of the rate constraint, by adjusting

, and trained to obtain feature prediction models with different bit rates.

In the specific implementation, the feature prediction model can use a self-attention mechanism, which can flexibly adjust the number of residual extraction modules used in the two channels as needed, or can use simple convolution to replace the residual extraction module. group, which is suitable for codec acceleration and simplification.

For example, the first branch and the second branch may respectively include four residual extraction modules, or respectively include four convolution modules.

It can be seen that, in the embodiment of the present application, the pre-trained neural network model is used to encode the residual information, so that the neural network model can implicitly learn residuals with different distortions, which is compared with the general end-to-end residuals. Coding, this method can self-adjust coding and perform inter-frame compensation. Under the same bit rate, it can allocate the residual information in space more efficiently, and obtain higher-quality reconstructed image frames.

Corresponding to the image encoding method described in FIG. 8A , FIG. 9A is a schematic flowchart of the image encoding method in the embodiment of the present application, and the image encoding method can be applied to the purpose of the encoding and decoding system 1 shown in FIG. 5 . The device 20 or the video decoder 200 shown in FIG. 7 . The flow shown in FIG. 9A is described by taking the execution subject as the video encoder 200 shown in FIG. 7 as an example. As shown in FIG. 9A , the image decoding method provided by the embodiment of the present application includes:

Step S210: Acquire a binary bit stream of a current decoding block, where the current decoding block includes a bit stream of a currently processed image frame or a decoding unit obtained by dividing the currently processed image frame.

Wherein, the division manner of the decoding unit includes various division manners as shown in FIG. 4 , which is not uniquely limited here.

Wherein, the decoding block corresponds to the encoding block involved in the foregoing encoding method embodiments, and may specifically be represented as having the same size.

In the specific implementation, for the current decoding block is the bit stream of the currently processed image frame, since the minimum data processing object is the bit stream of a single frame image, the processing efficiency of this method is higher, but the accuracy and performance have a certain loss. .

For the case where the current coding block is the bit stream of the coding unit obtained by dividing the currently processed image frame, since the minimum data processing granularity is the divided coding unit, the overall algorithm processing complexity becomes high, and the processing The duration becomes longer, but the accuracy and performance are relatively high.

Step S220 , transform the binary bit stream into the quantized feature of the current decoding block through a pre-trained probability prediction model.

Wherein, the transformation is a lossless transformation.

Among them, in the arithmetic coding process, for each pixel to be encoded, it is necessary to predict the probability of the corresponding pixel (a value between 0 and 1), and its probability can represent the frequency that the current pixel prediction may occur, and the predicted probability The higher it is, the higher the frequency it may appear, and the smaller the bit stream generated during arithmetic coding.

Step S230: Determine the residual block of the current decoding block according to the quantized feature and the pre-trained residual prediction model.

The residual prediction model can specifically realize data processing through the image processor GPU of the local device, and can adopt any commonly used neural network architecture, such as deep neural network DNN, recurrent neural network (Recurrent Neural Network, RNN), convolutional neural network (Convolutional Neural Network, CNN), etc., the input of this model is quantized features, and the output is residual block.

Step S240: Determine the reconstructed block of the current decoding block according to the residual block and the prediction block of the current decoding block.

In this possible example, the determining the reconstructed block of the current decoding block according to the original residual block and the prediction block of the current decoding block includes: determining the prediction block of the current decoding block; using the The original residual block performs residual compensation on the prediction block of the current decoding block to obtain a reconstructed block of the current decoding block.

The image decoding method in the embodiment of the present application can be specifically explained as the following steps.

First, obtain a bit stream, which corresponds to the secondary bit stream of the current decoding block, which may specifically include the public parameter set of the current decoding block and the encoding information of the image of the current decoding block,

Secondly, starting with the initialized all-zero feature, the value read from the binary bit stream is the input of the pre-trained probability prediction model, and the model is run to output the quantized feature of the current decoding block;

Again, take the quantized features predicted by the model as the input of the pre-trained residual prediction model, run the model to output the corresponding residual block,

Finally, according to the residual block predicted by the model and the predicted block of the current decoding block, the reconstructed block or reconstructed image is calculated.

The prediction block can be obtained by predicting the current decoding block according to the inter-frame prediction mode carried in the decoding message.

In this possible example, the determining the prediction block of the current decoding block includes: performing entropy decoding on the current decoding block to generate a syntax element; determining the inter frame for decoding the current decoding block according to the syntax element a prediction mode; according to the determined inter prediction mode, perform inter prediction on the current decoding block to obtain a prediction block of the current decoding block.

In this possible example, as shown in FIG. 9B , the residual prediction model includes a first branch and a second branch, and the first branch and the second branch are connected in parallel; the first branch The circuit includes three cascaded residual extraction modules and one upsampling module; the second branch includes three cascaded residual extraction modules, one upsampling module and one startup module.

In addition, during the training process of the residual prediction model, the code rate and the loss function can be determined in the following manner.

得到，R為碼率約束的損失，P為所述量化後的變換特徵中每個像素的概率； The code rate is estimated through the formula

Obtain, R is the loss of the code rate constraint, and P is the probability of each pixel in the quantized transform feature;

，D(.)為均方誤差MSE函數或者L2損失函數，

為當前編碼塊的預測塊，

為前編碼塊，整數訊號殘差

，

為當前編碼塊的預測塊的離散分佈； loss function

, D(.) is the mean square error MSE function or L2 loss function,

is the prediction block of the current coding block,

is the pre-coding block, the integer signal residual

,

is the discrete distribution of the prediction block of the current coding block;

，L為每一幀的重建損失，R為碼率約束的損失，透過調整

，訓練得到不同碼率的殘差預測模型。 use rate-distortion optimization for the bit rate and the loss function

, L is the reconstruction loss of each frame, R is the loss of the rate constraint, by adjusting

, and train the residual prediction models with different bit rates.

In the specific implementation, the residual prediction model can adopt a self-attention mechanism, which can flexibly adjust the number of residual extraction modules used in the two channels according to the needs, and can also use simple convolution to replace the residual extraction module. Decoding acceleration and simplification.

In the specific implementation, the residual prediction model is used to extract features for the input residual blocks, and multiple residual extraction modules are used to extract multiple features for stacking, thereby realizing cascade feature extraction.

It should be noted that the first branch is the main feature extraction module, the module after the sigmoid of the second branch is the self-attention mapping module, and the outputs of the two branches are multiplied to generate the final residual. yuan.

It can be seen that, in the embodiment of the present application, the pre-trained neural network model is used to encode the residual information, so that the neural network model can implicitly learn residuals with different distortions, which is compared with the general end-to-end residuals. Coding, this method can self-adjust coding and perform inter-frame compensation. Under the same bit rate, it can allocate the residual information in space more efficiently, and obtain higher-quality reconstructed image frames.

An embodiment of the present application provides an image encoding apparatus, and the image encoding apparatus may be an image decoder or an image encoder. Specifically, the image encoding device is configured to perform the steps performed by the image decoder in the above decoding method. The image encoding apparatus provided by the embodiments of the present application may include modules corresponding to corresponding steps.

In this embodiment of the present application, the image coding apparatus can be divided into functional modules according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. middle. The above-mentioned integrated modules can be implemented in the form of hardware, or can be implemented in the form of software function modules. The division of modules in the embodiments of the present application is schematic, and is only a logical function division, and other division methods may be used in actual implementation.

In the case where each functional module is divided according to each function, FIG. 10 shows a possible schematic structural diagram of the image coding apparatus involved in the above embodiment. As shown in FIG. 10 , the image coding apparatus 1000 includes an obtaining unit 1001 for obtaining an original residual block of a current coding block, where the current coding block includes a currently processed image frame or is obtained by dividing the currently processed image frame The first prediction unit 1002 is used for obtaining the transform feature of the current coding block according to the original residual block and the pre-trained feature prediction model; the quantization unit 1003 is used for the current coding block. Quantize the transform feature of the current coding block to obtain the quantized feature of the current coding block; the second prediction unit 1004 is used to determine the probability of each pixel in the quantized feature of the current coding block through a pre-trained probability prediction model; Unit 1005, configured to generate a binary bit stream of the current coding block by using the probability of each pixel.

In this possible example, in terms of obtaining the original residual block of the current coding block, the obtaining unit 1001 is specifically configured to: determine the prediction block of the current coding block; and compare the prediction block of the current coding block with the prediction block of the current coding block. The original image block of the current coding block is subjected to difference to obtain the original residual block.

In this possible example, in the aspect of obtaining the original residual block by making a difference between the prediction block of the current coding block and the original image block of the current coding block, the obtaining unit 1001 is specifically configured to: : perform numerical transformation and quantization according to the prediction block of the current coding block to generate a discrete distribution of the prediction block; make a difference between the discrete distribution of the prediction block and the original image block of the current coding block to obtain all the integer signals. the original residual block.

In this possible example, in the aspect of obtaining the transform feature of the current coding block according to the original residual block and the pre-trained feature prediction model, the In terms of obtaining the transform feature of the current coding block, the first prediction unit 1002 is specifically used to: re-normalize the original residual block to obtain the normalized first residual block. difference block; perform sparse processing on the first residual block to obtain a processed second residual block; input the second residual block into a pre-trained feature prediction model to obtain the current coding block Transform features.

In this possible example, in terms of renormalizing the original residual block to obtain a normalized first residual block, the first prediction unit 1002 is specifically configured to: unify according to energy A mechanism is used to converge the different residual distributions of the original residual blocks to the same distribution space to obtain a normalized first residual block.

In this possible example, in the aspect of converging different residual distributions of the original residual blocks into the same distribution space according to the energy unification mechanism to obtain a normalized first residual block, the first residual block is The prediction unit 1002 is specifically configured to: extract the minimum pixel value x _min and the maximum pixel value x _max in the original residual block; normalize the original residual block to the interval (0, 1) through the following formula;

表示初次變換後的像素值，

表示歸一化前的像素值； in,

represents the pixel value after the initial transformation,

Represents the pixel value before normalization;

進行二次變換，得到處於區間（-1，1）的殘差連續分佈，即歸一化後的第一殘差塊，

by the following formula

Perform secondary transformation to obtain a continuous distribution of residuals in the interval (-1, 1), that is, the normalized first residual block,

表示歸一化後的像素值。 in,

Indicates the normalized pixel value.

In this possible example, in the aspect of performing thinning processing on the first residual block to obtain a processed second residual block, the first prediction model 101 is specifically used to: obtain a preset threshold set , the preset threshold value set includes a plurality of threshold values; the target threshold value adapted to the current coding block is screened from the preset threshold value set; the pixel value of each pixel in the first residual block is traversed, and the pixel value is The pixel value of the pixel whose value is less than the target threshold is set to zero to obtain the processed second residual block.

In this possible example, each of the plurality of thresholds is obtained by uniformly sampling the pixels of the current coding block according to a preset sampling interval.

In this possible example, in terms of quantizing the transform feature of the current coding block to obtain the quantization feature of the current coding block, the quantization unit 1003 is specifically configured to: transform the current coding block The feature adopts a differentiable quantization mechanism to transform the floating-point feature into a quantized integer feature to obtain the quantized feature of the current coding block.

In this possible example, the feature prediction model includes a first branch and a second branch, and the first branch and the second branch are connected in parallel; the first branch includes three cascaded A residual error extraction module and a downsampling module; the second branch includes three cascaded residual error extraction modules, a downsampling module and a startup module.

Wherein, all relevant contents of the steps involved in the above method embodiments can be cited in the functional descriptions of the corresponding functional modules, which will not be repeated here. Of course, the image encoding apparatus 1000 provided in the embodiment of the present application includes but is not limited to the above modules. For example, the image encoding apparatus 1000 may further include a storage unit. The storage unit can be used to store the code and data of the image encoding device.

In the case of using an integrated unit, a schematic structural diagram of the image encoding apparatus provided by the embodiment of the present application is shown in FIG. 11 . In FIG. 11 , the image encoding device 11 includes: a processing module 1102 and a communication module 1101 . The processing module 1102 is used to control and manage the actions of the image encoding device, for example, to execute the steps performed by the acquisition unit 1001, the first prediction unit 1002, the quantization unit 1003, the second prediction unit 1004, and the generation unit 1005, and/or Other processes for performing the techniques described herein. The communication module 1101 is used to support the interaction between the image encoding device and other devices. As shown in FIG. 11 , the image encoding apparatus may further include a storage module 1103, and the storage module 1103 is used to store the program codes and data of the image encoding apparatus, for example, to store the content stored in the above-mentioned storage unit.

The processing module 1102 may be a processor or a controller, such as a central processing unit (Central Processing Unit, CPU), a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), ASIC, FPGA or other Program logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules and circuits described in connection with this disclosure. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 1101 can be a transceiver, an RF circuit, or a communication interface. The storage module 1103 may be a memory.

Wherein, all relevant contents of the scenarios involved in the above method embodiments can be cited in the functional descriptions of the corresponding functional modules, which will not be repeated here. The image encoding device 1000 and the image encoding device 11 can both execute the image encoding method shown in FIG. 8A . functional device.

The present application also provides an image encoder, including a non-volatile storage medium, and a central processing unit, wherein the non-volatile storage medium stores an executable program, the central processing unit is connected to the non-volatile storage medium, and The executable program is executed to implement the image encoding method of the embodiment of the present application.

An embodiment of the present application provides an image decoding apparatus, and the image decoding apparatus may be an image decoder or an image decoder. Specifically, the image decoding apparatus is configured to perform the steps performed by the image decoder in the above decoding method. The image decoding apparatus provided by the embodiments of the present application may include modules corresponding to corresponding steps.

In this embodiment of the present application, the image decoding apparatus may be divided into functional modules according to the above method examples. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. middle. The above-mentioned integrated modules can be implemented in the form of hardware, or can be implemented in the form of software function modules. The division of modules in the embodiments of the present application is schematic, and is only a logical function division, and other division methods may be used in actual implementation.

In the case where each functional module is divided according to each function, FIG. 12 shows a possible schematic structural diagram of the image decoding apparatus involved in the above embodiment. As shown in FIG. 12, the image decoding device 12 includes:

an obtaining unit 124, configured to obtain a binary bit stream of a current decoding block, where the current decoding block includes a bit stream of a currently processed image frame or a decoding unit obtained by dividing the currently processed image frame;

The first prediction unit 121 is used to transform the binary bit stream into a quantized feature of the current decoding block through a pre-trained probability prediction model;

The second prediction unit 122 is configured to determine the residual block of the current decoding block according to the quantized feature and the pre-trained residual prediction model;

The determining unit 123 is configured to determine the reconstructed block of the current decoding block according to the residual block and the prediction block of the current decoding block.

In a possible example, in the aspect of determining the reconstructed block of the current decoding block according to the original residual block and the prediction block of the current decoding block, the determining unit 123 is specifically configured to: determine the The prediction block of the current decoding block; using the original residual block to perform residual compensation on the prediction block of the current decoding block to obtain the reconstructed block of the current decoding block.

In a possible example, in the aspect of determining the prediction block of the current decoding block, the determining unit 123 is specifically configured to: perform entropy decoding on the current decoding block to generate a syntax element; according to the determined inter prediction mode, performing inter prediction on the current decoding block to obtain the prediction block of the current decoding block.

In a possible example, the residual prediction model includes a first branch and a second branch, the first branch and the second branch are connected in parallel; the first branch includes a cascade of three a residual extraction module and an upsampling module; the second branch includes three cascaded residual extraction modules, an upsampling module and a startup module.

Wherein, all relevant contents of the steps involved in the above method embodiments can be cited in the functional descriptions of the corresponding functional modules, which will not be repeated here. Of course, the image decoding apparatus provided in the embodiment of the present application includes but is not limited to the above-mentioned modules. For example, the image decoding apparatus may further include a storage unit. The storage unit can be used to store the code and data of the image decoding device.

In the case of using an integrated unit, a schematic structural diagram of an image decoding apparatus provided by an embodiment of the present application is shown in FIG. 13 . In FIG. 13 , the image decoding device 13 includes: a processing module 130 and a communication module 131 . The processing module 130 is used to control and manage the actions of the image decoding device, for example, to perform the steps performed by the acquisition unit 124, the first prediction unit 121, the second prediction unit 122, and the determination unit 123, and/or to perform the steps described herein. other procedures of the described techniques. The communication module 131 is used to support the interaction between the image decoding apparatus and other devices. As shown in FIG. 13 , the image decoding apparatus may further include a storage module 132 , and the storage module 132 is used to store the program codes and data of the image decoding apparatus, for example, to store the contents stored in the above-mentioned storage unit 123 .

The processing module 130 may be a processor or a controller, such as a central processing unit (Central Processing Unit, CPU), a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), ASIC, FPGA or other Program logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules and circuits described in connection with this disclosure. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 131 can be a transceiver, an RF circuit, a communication interface, or the like. The storage module 132 may be a memory.

Wherein, all relevant contents of the scenarios involved in the above method embodiments can be cited in the functional descriptions of the corresponding functional modules, which will not be repeated here. Both the image decoding device 12 and the image decoding device 13 described above can perform the image decoding method shown in FIG. 9A . functional device.

The present application also provides an image decoder, including a non-volatile storage medium, and a central processing unit, wherein the non-volatile storage medium stores an executable program, the central processing unit is connected to the non-volatile storage medium, and The executable program is executed to implement the image decoding method of the embodiment of the present application.

The present application also provides a terminal, where the terminal includes: one or more processors, a memory, and a communication interface. The memory and the communication interface are coupled with one or more processors; the memory is used to store computer program codes, and the computer program codes include instructions. When one or more processors execute the instructions, the terminal executes the images of the embodiments of the present application. Encoding and/or image decoding methods. The terminal here can be an image display device, a smart phone, a portable computer, and other devices that can process images or play images.

Another embodiment of the present application further provides a computer-readable storage medium, the computer-readable storage medium includes one or more program codes, and the one or more programs include instructions, when the processor in the decoding device executes the program code , the decoding device executes the image encoding method and the image decoding method of the embodiments of the present application.

In another embodiment of the present application, there is also provided a computer program product, the computer program product includes computer-executable instructions, and the computer-executable instructions are stored in a computer-readable storage medium; at least one processor of the decoding device can be obtained from the computer The read storage medium reads the computer-executed instruction, and at least one processor executes the computer-executed instruction so that the terminal implements the image encoding method and the image decoding method of the embodiments of the present application.

In the above embodiments, it may be implemented in whole or in part through software, hardware, firmware or any combination thereof. When implemented using a software program, it may be in the form of a computer program product, in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, all or part of the processes or functions described in the embodiments of the present application are generated.

The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions may be stored on or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be downloaded from a website, computer, server or data center Transmission to another website, computer, server or data center via wired (eg coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg infrared, wireless, microwave, etc.).

The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, etc., which is integrated with one or more available media. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), and the like.

Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. The allocation is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or elements may be Incorporation may either be integrated into another device, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical or other forms.

The unit described as a separate component may or may not be physically separated, and a component shown as a unit may be one entity unit or multiple entity units, that is, it may be located in one place, or may be distributed to multiple different places. . Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist independently, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, which are stored in a storage medium , which includes several instructions for causing a device (which may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: flash drives, mobile hard drives, Read-Only Memory (ROM), Random Access Memory (RAM), disks or CDs, etc. that can store program codes medium.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this, and any changes or substitutions within the technical scope disclosed in the present application should be covered within the protection scope of the present application. . Therefore, the protection scope of the present application shall be subject to the protection scope of the claimed item.

1: Image 10: Source device 11: Image coding device 12: Image decoding device 13: Image decoding device 20: Purpose Device 30: Link 40: Storage device 41: Postprocessing entities 42: Network entities 100: Video encoder 101: Transformer 102: Quantizer 103: Entropy Encoder 104: Inverse Quantizer 105: Inverse Transformer 106: Filter unit 107: Memory 108: Prediction processing unit 109: Intra Predictor 110: Inter predictor 111: Summation 112: Summation 120: Image source 121: First prediction unit 122: Second prediction unit 123: Determine unit 124: Get Unit 130: Processing Modules 131: Communication module 132: Storage Module 140: Output interface 200: Video decoder 203: Entropy Decoder 204: Inverse Quantizer 205: Inverse Transformer 206: Filter unit 207: Memory 208: Prediction Processing Unit 209: Intra Predictor 210: Inter predictor 211: Summation 220: Display device 240: Input interface 1000: Image coding device 1001: Get unit 1002: First prediction unit 1003: Quantization unit 1004: Second prediction unit 1005: Generate unit 1101: Communication module 1102: Processing Modules 1103: Storage Module S110~S150: Steps S210~S240: Steps

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the For some embodiments of the invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a schematic block diagram of a coding tree unit in an embodiment of the present application;

2 is a schematic block diagram of a CTU and a coding block CU in an embodiment of the present application;

3 is a schematic block diagram of a color format in an embodiment of the application;

FIG. 4 is a schematic diagram of an image division manner in an embodiment of the present application;

5 is a schematic block diagram of an encoding and decoding system in an embodiment of the present application;

6 is a schematic block diagram of an image encoder in an embodiment of the present application;

7 is a schematic block diagram of an image decoder in an embodiment of the present application;

8A is a schematic flowchart of an image encoding method in an embodiment of the present application;

8B is a schematic diagram of a residual map generated after processing with different thresholds in an embodiment of the present application;

8C is a structural diagram of a feature prediction model in an embodiment of the present application;

9A is a schematic flowchart of an image decoding method in an embodiment of the present application;

9B is a structural diagram of a residual prediction model in an embodiment of the present application;

10 is a block diagram of a functional unit of an image encoding apparatus in an embodiment of the application;

11 is a block diagram of another functional unit of the image coding apparatus in the embodiment of the application;

12 is a block diagram of a functional unit of an image decoding apparatus in an embodiment of the application;

FIG. 13 is a block diagram of another functional unit of the image decoding apparatus according to the embodiment of the present application.

S110~S150: Steps

Claims (21)

An image encoding method, comprising: obtaining an original residual block of a current coding block, where the current coding block includes a currently processed image frame or a coding unit obtained by dividing the currently processed image frame; Obtain the transform feature of the current coding block according to the original residual block and the pre-trained feature prediction model; quantizing the transform feature of the current coding block to obtain the quantization feature of the current coding block; Determine the probability of each pixel in the quantization feature of the current coding block through a pre-trained probability prediction model; A binary bit stream for the current coding block is generated using the probability of each pixel. The method according to claim 1, wherein the obtaining the original residual block of the current coding block includes: determining a prediction block for the current coding block; The original residual block is obtained by making a difference between the prediction block of the current coding block and the original image block of the current coding block. The method according to claim 2, wherein the obtaining the original residual block by performing a difference between the prediction block of the current coding block and the original image block of the current coding block includes: Perform numerical transformation and quantization according to the prediction block of the current coding block to generate a discrete distribution of the prediction block; Differences between the discrete distribution of the prediction block and the original image block of the current coding block are performed to obtain the original residual block of integer signals. The method according to claim 1, wherein obtaining the transform feature of the current coding block according to the original residual block and the pre-trained feature prediction model, comprising: renormalizing the original residual block to obtain a normalized first residual block; performing sparse processing on the first residual block to obtain a processed second residual block; Inputting the second residual block into a pre-trained feature prediction model to obtain the transform feature of the current coding block. The method according to claim 4, wherein the re-normalization of the original residual block to obtain the normalized first residual block includes: According to the energy unification mechanism, the different residual distributions of the original residual blocks are converged to the same distribution space to obtain a normalized first residual block. The method according to claim 5, wherein, according to the energy unification mechanism, different residual distributions of the original residual blocks are converged to the same distribution space to obtain a normalized first residual block, including: Extract the minimum pixel value x _min and the maximum pixel value x _max in the original residual block; normalize the original residual block to the interval (0, 1) through the following formula;

in,

represents the pixel value after the initial transformation,

Indicates the pixel value before normalization; through the following formula

Perform secondary transformation to obtain a continuous distribution of residuals in the interval (-1, 1), that is, the normalized first residual block;

in,

Indicates the normalized pixel value. The method according to any one of claims 4-6, wherein the performing thinning processing on the first residual block to obtain a processed second residual block, comprising: acquiring a preset threshold set, where the preset threshold set includes multiple thresholds; Screening the target threshold for adapting the current coding block from the preset threshold set; The pixel value of each pixel in the first residual block is traversed, and the pixel value of the pixel whose pixel value is less than the target threshold is set to zero to obtain a processed second residual block. The method according to claim 7, wherein each of the plurality of thresholds is obtained by uniformly sampling the pixels of the current coding block according to a preset sampling interval. The method according to claim 1, wherein the quantizing the transform feature of the current coding block to obtain the quantization feature of the current coding block includes: A differentiable quantization mechanism is used for the transform feature of the current coding block, and the floating-point feature is transformed into a quantized integer feature to obtain the quantized feature of the current coding block. The method according to any one of claims 1-9, wherein the feature prediction model includes a first branch and a second branch, and the first branch and the second branch are connected in parallel; The first branch includes three cascaded residual extraction modules and a downsampling module; The second branch includes three cascaded residual extraction modules, a downsampling module and a startup module. An image decoding method, comprising: obtaining a binary bit stream of a current decoding block, where the current decoding block includes a bit stream of a currently processed image frame or a decoding unit obtained by dividing the currently processed image frame; Transforming the binary bit stream into a quantized feature of the current decoding block through a pre-trained probabilistic prediction model; Determine the residual block of the current decoding block according to the quantized feature and the pre-trained residual prediction model; A reconstructed block of the current decoding block is determined according to the residual block and the prediction block of the current decoding block. The method according to claim 11, wherein the determining the reconstructed block of the current decoding block according to the original residual block and the prediction block of the current decoding block comprises: determining a prediction block for the current decoded block; Perform residual compensation on the prediction block of the current decoding block by using the original residual block to obtain the reconstructed block of the current decoding block. The method according to claim 12, wherein the determining the prediction block of the current decoding block comprises: entropy decoding the current decoded block to generate syntax elements; determining an inter prediction mode for decoding the current decoding block according to the syntax element; According to the determined inter prediction mode, inter prediction is performed on the current decoding block to obtain a prediction block of the current decoding block. The method according to claim 11, wherein the residual prediction model includes a first branch and a second branch, and the first branch and the second branch are connected in parallel; The first branch includes three cascaded residual extraction modules and an upsampling module; The second branch includes three cascaded residual extraction modules, an upsampling module and a startup module. An image encoding device, comprising: an obtaining unit, configured to obtain an original residual block of a current coding block, where the current coding block includes a currently processed image frame or a coding unit obtained by dividing the currently processed image frame; The first prediction unit is used to obtain the transform feature of the current coding block according to the original residual block and the pre-trained feature prediction model; a quantization unit, configured to quantize the transform feature of the current coding block to obtain the quantization feature of the current coding block; The second prediction unit is used to determine the probability of each pixel in the quantization feature of the current coding block through a pre-trained probability prediction model; A generating unit, configured to generate a binary bit stream of the current coding block by using the probability of each pixel. An image decoding device, comprising: an acquisition unit, configured to acquire a binary bit stream of a current decoding block, where the current decoding block includes a bit stream of a currently processed image frame or a decoding unit obtained by dividing the currently processed image frame; a first prediction unit for converting the binary bit stream into a quantized feature of the current decoding block through a pre-trained probability prediction model; a second prediction unit, configured to determine the residual block of the current decoding block according to the quantized feature and the pre-trained residual prediction model; A determination unit, configured to determine the reconstructed block of the current decoding block according to the residual block and the prediction block of the current decoding block. An encoder includes a non-volatile storage medium and a central processing unit, the non-volatile storage medium stores an executable program, the central processing unit is connected to the non-volatile storage medium, and when the central processing unit executes When the executable program is executed, the encoder performs the bidirectional inter-frame prediction method as described in any one of claims 1-10. A decoder includes a non-volatile storage medium and a central processing unit, the non-volatile storage medium stores an executable program, the central processing unit is connected to the non-volatile storage medium, and when the central processing unit executes When the executable program is executed, the decoder performs the bidirectional inter prediction method as described in any one of claims 11-14. A terminal, the terminal comprises: one or more processors, a memory and a communication interface; the memory, the communication interface are connected with the one or more processors; the terminal communicates with the one or more processors through the communication interface other equipment communication, the memory is used to store computer program code, the computer program code includes instructions, When the one or more processors execute the instructions, the terminal performs the method as described in any one of claim items 1-10 or 11-14. A computer program product comprising instructions which, when run on a terminal, cause the terminal to perform the method of any one of claims 1-10 or 11-14. A computer-readable storage medium comprising instructions that, when executed on a terminal, cause the terminal to perform the method of any one of claim 1-10 or 11-14.

Images