CN116998148A

CN116998148A - Coding method and device

Info

Publication number: CN116998148A
Application number: CN202180095762.2A
Authority: CN
Inventors: 王菁; 于建华
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-03-23
Filing date: 2021-03-23
Publication date: 2023-11-03
Also published as: WO2022198465A1

Abstract

A coding method and a device relate to the technical field of video image coding and solve the problem that the existing code rate control is over or insufficient in bits distributed to image frames. The specific scheme comprises the following steps: the encoding apparatus determines target information including the change information, acquires reference bits, and determines target bits allocated to an image frame to be encoded based on the target information and the reference bits. Thereafter, the encoding apparatus performs encoding of the image frame to be encoded, and quantization in the encoding is performed based on the target bits. Wherein the change information is used to characterize the change in the content of the image frame to be encoded relative to the reference frame. The reference bit is obtained according to a preset target code rate and a preset frame rate.

Description

Coding method and device

Technical Field

The present application relates to the field of video image coding technology, and in particular, to a coding method and apparatus.

Background

In video coding, a target code rate may be set to a video in advance. The coding end can enable the code rate after video coding to be as close to the target code rate as possible by carrying out code rate control.

The code rate control process comprises the following steps: the encoding end encodes the first image frame of the video to obtain the actual bits consumed by encoding the first image frame. Then, for the image frames to be encoded except for the first image frame, the encoding end obtains the reference bit of the image frame according to the target code rate and the frame rate, and determines the estimated bit allocated to the image frame to be encoded according to the reference bit and the actual bit of the previous image frame of the image frame to be encoded. The encoding end determines quantization parameters (quantization parameter, QP) of the image frame to be encoded according to the estimated bit, and quantizes the image frame to be encoded by QP. After the encoding end finishes encoding the image frames to be encoded, the actual bits consumed by encoding the image frames to be encoded are obtained. Repeating the above process until the video coding is completed.

However, the encoding end determines the estimated bits allocated to the image frame to be encoded according to the reference bits and the actual bits of the previous image frame, which may cause excessive or insufficient allocated estimated bits, and excessive allocation may cause resource waste, and insufficient allocation may cause poor quality of the encoded image frame to be encoded.

Disclosure of Invention

The application provides a coding method and a device, which solve the problem that the bits distributed to image frames in the existing code rate control are too much or too insufficient.

In order to achieve the above purpose, the application adopts the following technical scheme:

in a first aspect, the present application provides an encoding method, an encoding apparatus determining target information, the target information including change information, acquiring reference bits, and determining target bits allocated to an image frame to be encoded based on the target information and the reference bits. Thereafter, the encoding apparatus performs encoding of the image frame to be encoded, and quantization in the encoding is performed based on the target bits. Wherein the change information is used to characterize the change in the content of the image frame to be encoded relative to the reference frame. The reference bit is obtained according to a preset target code rate and a preset frame rate.

The change information can reflect the change of the content of the image frame to be encoded relative to the reference frame, and the encoding end determines the target bit of the image frame to be encoded by taking the reference bit as a base line according to the content of the image frame to be encoded, so that the target bits of all the image frames to be encoded float up and down on the base line, and the code rate after video encoding can reach the target code rate. For example, if the change of the content of the image frame to be encoded relative to the reference frame is large, the bit is added as the target bit on the basis of the reference bit to improve the image quality. If the change of the content of the image frame to be encoded relative to the reference frame is small, the bit is reduced on the basis of the reference bit as a target bit, so that the resource waste is reduced. Compared with the mode of determining the target bit by taking the reference bit as the base line in the prior art and determining the target bit according to the information of the reference frame, the method and the device for determining the target bit in the embodiment of the application have the advantages that the determined target bit is more accurate, so that the resource waste can be reduced or the image quality can be improved.

Optionally, in a possible implementation manner of the present application, the method for determining the change information may include: the encoding device determines the similarity between the image frame to be encoded and the reference frame, and determines the change signal and the change signal according to the similarity.

Alternatively, in another possible implementation manner of the present application, the method for determining the similarity between the image frame to be encoded and the reference frame may include: the encoding device acquires the homography matrix and determines the similarity according to the homography matrix and the identity matrix. The homography matrix is used for representing the mapping relation of the position of the target object in the image frame to be encoded and the reference frame.

In the case where the acquisition of the homography matrix depends on the anti-shake technique, no additional computational burden is imposed on the encoding apparatus.

Optionally, in another possible implementation of the present application, the target information further includes a first bit, where the first bit is a bit consumed for encoding a reference frame, and the reference frame is a forward encoded image frame or a backward encoded image frame of the image frame to be encoded.

When determining the target bit of the image frame to be encoded, the encoding end uses the reference bit as a base line, so that not only the change information is referenced, but also the actual bit consumed by encoding the reference frame, namely the first bit. And when the first bit is larger than the reference bit, the bit number is reduced, and when the first bit is smaller than the reference bit, the bit number is increased, so that the accuracy of the determined target bit is further improved.

Alternatively, in another possible implementation manner of the present application, the method of determining the target bits allocated to the image frame to be encoded according to the reference bits and the target information may include: the encoding device determines a second bit according to the first bit and the reference bit, and determines an adjustment amount of the second bit according to the change information, thereby obtaining a target bit.

Optionally, in another possible implementation manner of the present application, the encoding method provided by the present application further includes: the encoding device determines a motion vector of the image frame to be encoded relative to the reference frame according to the homography matrix and the position of the first pixel point, and determines an image block bit of each image block in the image frame to be encoded according to the motion vector and the target bit. The first pixel point is a pixel point at the center of the image frame to be encoded.

The encoding device enables an optimization of block-level bit allocation of the image frames to be encoded.

Alternatively, in another possible implementation manner of the present application, the method of determining an image block bit of each image block in an image frame to be encoded according to a motion vector and a target bit may include: the encoding device determines a target image block in the image frame to be encoded according to the motion vector, determines an average bit of the image block in the image frame to be encoded according to the target bit, and determines an image block bit of the target image block according to the average bit. Wherein the reference frame does not include the contents of the target image block.

In this way, the image quality can be further improved by optimizing the block-level code rate allocation and compensating the target image block.

Optionally, in another possible implementation manner of the present application, the encoding method provided by the present application further includes: the encoding device determines a prediction mode corresponding to an image block of the image frame to be encoded according to the change information, wherein the prediction mode is intra-frame prediction or inter-frame prediction. In this case, the above-described method of "encoding an image frame to be encoded" may include: the encoding device performs encoding on the image frame to be encoded according to the target bit and the prediction mode.

By referring to the change information when determining the prediction mode, the computational complexity can be reduced.

In a second aspect, the present application provides an encoding apparatus comprising respective modules for performing the encoding method of the first aspect or any one of the possible implementations of the first aspect.

In a third aspect, the present application provides an encoding apparatus comprising a memory and a processor. The memory is coupled to the processor. The memory is used to store computer program code, which includes computer instructions. When the processor executes computer instructions, the encoding apparatus performs the encoding method as in the first aspect and any one of its possible implementations.

In a fourth aspect, the present application provides a chip system, which is applied to an encoding apparatus. The system-on-chip includes one or more interface circuits, and one or more processors. The interface circuit and the processor are interconnected through a circuit; the interface circuit is for receiving signals from the memory of the encoding device and for sending signals to the processor, the signals including computer instructions stored in the memory. When the processor executes computer instructions, the encoding apparatus performs the encoding method as in the first aspect and any one of its possible implementations.

In a fifth aspect, the present application provides a computer readable storage medium comprising computer instructions which, when run on an encoding apparatus, cause the encoding apparatus to perform the encoding method as in the first aspect and any one of its possible implementations.

In a sixth aspect, the present application provides a computer program product comprising computer instructions which, when run on an encoding apparatus, cause the encoding apparatus to perform the encoding method as in the first aspect and any one of its possible implementations.

For a detailed description of the second to sixth aspects of the present application and various implementations thereof, reference may be made to the detailed description of the first aspect and various implementations thereof; moreover, the advantages of the second aspect and the various implementations thereof may be referred to as analyzing the advantages of the first aspect and the various implementations thereof, and will not be described herein.

Drawings

Fig. 1 is a schematic diagram of an image encoding and decoding process according to an embodiment of the present application;

fig. 2 is a schematic diagram of LCU partitioning according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of a codec system according to an embodiment of the present application;

FIG. 4 is a schematic flow chart of an encoding method according to an embodiment of the present application;

fig. 5 is a schematic diagram of a scenario of rate control provided by an embodiment of the present application;

fig. 6 is a schematic structural diagram of an encoding device according to an embodiment of the present application.

Detailed Description

In embodiments of the application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g." in an embodiment should not be taken as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

The terms "first" and "second" are used below for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the embodiments of the present application, unless otherwise indicated, the meaning of "a plurality" is two or more.

In order to facilitate understanding of embodiments of the present application, terms related to the embodiments of the present application will be briefly described.

1. Image coding and decoding

Image encoding (image encoding): a process of compressing a sequence of images into a code stream.

Image decoding (image decoding): and recovering the code stream into a processing procedure of reconstructing the image according to a specific grammar rule and a processing method.

Fig. 1 is a schematic diagram of a process of image encoding and decoding. As shown in fig. 1, the coding process of the image is: the coding end firstly divides a frame of original image into a plurality of parts which are not overlapped with each other, and each part can be used as an image block. Then, the encoding end performs operations such as prediction (prediction), transform (transform), quantization (quantization), and entropy coding (entropy coding) for each image block to obtain actual bits consumed for encoding each image block. The sum of the actual bits consumed by all image blocks included in an image frame is the actual bits consumed to encode the image frame. The code stream output after entropy encoding in fig. 1 refers to a data stream composed of actual bits consumed by each image frame of a video after the encoding end encodes the video. In addition, prediction includes intra prediction (intra prediction) and inter prediction (inter prediction) to obtain a prediction block of an image block, so that only a difference (or referred to as a residual or residual block) between the image block and the prediction block thereof, and information contained in the prediction block can be encoded and transmitted, thereby saving transmission overhead. Intra prediction relies on information of the current frame to obtain a prediction block, and inter prediction relies on information of the current frame and a reference frame to obtain a prediction block. And finally, the encoding end sends the code stream corresponding to the image block to the decoding end.

Of course, the encoding end may also be a decoding end, and has a decoding function. As shown in fig. 1, the decoding process of the image is: the decoding end receives the code stream from other encoding end (the code stream refers to the code stream input with entropy coding in fig. 1), and performs operations such as entropy coding, inverse quantization, inverse transformation, prediction, filtering, etc. on the received code stream to obtain a reconstructed image block (or referred to as a reconstructed image block), which is referred to as an image reconstruction process (or an image reconstruction process). Then, the decoding end assembles the reconstruction block of each image block in the original image to obtain a reconstructed image of the original image.

2. Image block

Image encoding/decoding the encoding/decoding process may be performed row by row starting from the upper left corner position of the image from left to right, top to bottom, in units of image blocks. Here, the image block may be a Macroblock (MB) in the video codec standard h.264, or may be a Coding Unit (CU) in the high efficiency video coding (high efficiency video coding, HEVC) standard, which is not particularly limited in the embodiment of the present application.

For example, when the image block is a CU in HEVC, the partitioning procedure of the CU is: the coding end divides the image into maximum coding units (largest coding unit, LCU) with fixed sizes, and then divides each LCU into CUs according to the specific content of the image.

The LCU corresponds to a square image area. An LCU corresponds to an image area comprising 64 x 64 pixels. A 64 x 64 pixel LCU contains a rectangular pixel lattice of 64 columns of 64 pixels each.

The CU corresponds to a rectangular area of a×b. Here, the values of a and B may be the same. The values of a and B are integers raised to the power of 2, for example: 64. 32, 16, 8. The smallest partition unit of a CU is 8×8.

When dividing an LCU into CUs, there may be four division depths, e.g., 64×64, 32×32, 16×16, 8×8. As shown in fig. 2 (a), when the partition depth is 64×64, one LCU is one CU. As shown in (b) of fig. 2, when the division depth is 32×32, the LCU may be divided into four CUs, a, b, c, d respectively. As shown in (c) of fig. 2, a, b, c, or d may be divided into four CUs when the division depth is 16×16, respectively, and (c) of fig. 2 is exemplified by dividing a into a0, a1, a2, a 3. As shown in (d) of fig. 2, a0, a1, a2, or a3 may be divided into four CUs when the division depth is 8×8, respectively, and (d) of fig. 2 is exemplified by division of a0 into a00, a01, a02, a 03.

The optimal division depth of the LCU is determined according to the rate-distortion cost when different CUs are combined with different prediction modes under different division depths by traversing all selectable intra-frame prediction modes or inter-frame prediction modes of each CU in the prediction process. Thus, fast prediction algorithms are derived, for example, by early termination of traversal conditions based on the characteristics of local blocks of the image, but they still consume computational resources. According to the coding method provided by the embodiment of the application, the coding equipment can determine the rate distortion cost of the prediction unit according to the change information and determine the prediction mode of the image block according to the rate distortion cost, so that the calculation complexity can be reduced.

3. Motion Vector (MV)

Motion vectors are an important parameter in the inter prediction process that represents the spatial displacement of an encoded image block relative to the current image block. In general, motion vectors may be obtained using methods of motion estimation (Motion Estimation, ME), such as motion search. In the initial inter-frame prediction technology, the coding end transmits the motion vector of the current image block in the code stream, so that the decoding end reproduces the predicted pixels of the current image block, and further a reconstructed block is obtained.

4. Frame rate

The frame rate refers to the number of frames of an image transmitted per second.

5. Code rate control

The code rate control is applied to the coding process of the image, and is a coding optimization algorithm for realizing the control of the video code rate. The target code rate can be preset, and the code rate after video coding is close to the target code rate by adjusting the quantization parameters in the quantization process.

The code rate refers to the number of bits transmitted per second. The unit generally used is kbps.

At present, a common code rate control strategy is average code rate (ABR), the average code rate is required to reach a target code rate within a period of time, such as 3-5 seconds, and the strategy can well consider the stability of the code rate and the image quality.

Illustratively, in connection with fig. 1, the average code rate control process is: the code rate control module of the coding end calculates the number of bits which can be averagely allocated to each frame of image according to the set target code rate, frame rate and image resolution, and takes the number of bits as a reference bit. When the video is required to be encoded, the encoding end can predict and transform the first image frame of the video, determine the corresponding quantization parameter according to the reference bit, and quantize the first image frame by adopting the determined quantization parameter. Then, the encoding end performs entropy encoding on the first image frame to obtain the actual bits consumed by encoding the first image frame, and inputs the actual bits of the first image frame to the code rate control module. And then, for each image frame to be encoded after the first image frame, the encoding end predicts and transforms the image frame to be encoded, determines the estimated bit allocated to the image frame to be encoded according to the reference bit and the actual bit of the previous image frame of the image frame to be encoded, determines the quantization parameter of the image frame to be encoded according to the estimated bit, and quantizes the image frame to be encoded by adopting the quantization parameter. Then, the encoding end carries out entropy encoding on the image frame to be encoded so as to obtain actual bits consumed by encoding the image frame to be encoded, and the actual bits of the image frame to be encoded are input into the code rate control module. Repeating the above process until the video coding is completed.

However, since the similarity difference between the image frame to be encoded and the previous image frame may be larger or smaller, the encoding end determines the estimated bit allocated to the image frame to be encoded according to the reference bit and the actual bit of the previous image frame, which may result in excessive or insufficient estimated bit allocation, and excessive allocation may cause resource waste, and insufficient allocation may result in poor quality of the encoded image frame to be encoded.

In order to solve the above technical problems, an embodiment of the present application provides an encoding method, where an encoding end can determine a target bit of an image frame to be encoded according to a reference bit and change information, where the change information can reflect a change of the content of the image frame to be encoded relative to a reference frame. That is, the encoding end determines the target bit of the image frame to be encoded by taking the reference bit as the baseline according to the content of the image frame to be encoded, so that the target bits of all the image frames to be encoded float up and down on the baseline, and the code rate after video encoding can reach the target code rate. For example, if the change of the content of the image frame to be encoded relative to the reference frame is large, the bit is added as the target bit on the basis of the reference bit to improve the image quality. If the change of the content of the image frame to be encoded relative to the reference frame is small, the bit is reduced on the basis of the reference bit as a target bit, so that the resource waste is reduced. Compared with the mode of determining the target bit by taking the reference bit as the base line in the prior art and determining the target bit according to the information of the reference frame, the method and the device for determining the target bit in the embodiment of the application have the advantages that the determined target bit is more accurate, so that the resource waste can be reduced or the image quality can be improved.

The coding method provided by the embodiment of the application is suitable for a video coding and decoding system. Fig. 3 shows a structure of the video codec system. As shown in fig. 3, the video codec system includes an encoding device 31 and a decoding device 32.

The encoding device 31 generates encoded video data, and the encoding device 31 may also be referred to as a source device or an encoding apparatus. The decoding device 32 may decode the encoded video data generated by the encoding device 31, and the decoding device 32 may also be referred to as a destination device or a decoding apparatus. The encoding device 31 and/or the decoding device 32 may include at least one processor and a memory coupled to the at least one processor. The memory may include, but is not limited to, read-only memory (ROM), random access memory (random access memory) RAM, charged erasable programmable read-only memory (EEPROM), flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures that can be accessed by a computer, as the application is not limited in detail.

Encoding device 31 and decoding device 32 may include a variety of devices including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as "smart" phones, televisions, cameras, display devices, digital media players, video game consoles, vehicle mount computers, or the like.

In fig. 3, the encoding device 31 includes a video source 301, a video encoder 302, and an output interface 303. In some examples, output interface 303 may include a regulator/demodulator (modem) and/or a transmitter. Video source 301 may include a video capture device (e.g., a video camera), a video archive containing previously captured video data, a video input interface to receive video data from a video content provider, and/or a computer graphics system for generating video data, or a combination of such sources of video data.

Video encoder 302 may encode video data from video source 301. In some examples, encoding device 31 transmits the encoded video data directly to decoding device 32 via output interface 303. In other examples, the encoded video data may also be stored on storage device 33 for later access by decoding device 32 for decoding and/or playback.

Optionally, the encoding device 31 may further comprise an image signal processor (image signal processor, ISP) 304.ISP 304 is used to perform multi-stage processing, such as noise reduction, anti-shake, and distortion correction, on video data acquired by video source 301. The processed video data is video compressed by video encoder 302.

Alternatively, the video encoder 302 may be integrated with an audio encoder and may include an appropriate multiplexer-demultiplexer unit or other hardware and software to handle encoding of both audio and video in a common data stream or separate data streams.

The video encoder 302 may include at least one microprocessor, digital signal processor (digital signal processor, DSP), application-specific integrated circuit (ASIC), field programmable gate array (field programmable gate array, FPGA), discrete logic, hardware, or any combination thereof.

Based on the description of the video encoding and decoding system and the encoding device, an embodiment of the present application provides an encoding method, and the encoding method provided by the embodiment of the present application is described below with reference to the accompanying drawings.

As shown in fig. 4, the encoding method provided by the embodiment of the present application may include the following steps 401 to 404.

401. The encoding device determines target information.

Wherein the target information may include change information. When an image frame to be encoded is required, the encoding apparatus may acquire a reference frame of the image frame to be encoded and determine a change in the content of the image frame to be encoded with respect to the reference frame, that is, determine change information. The reference frame may be a forward coded image frame or a backward coded image frame of the image frame to be coded.

It will be appreciated that if the content of the image frame to be encoded does not change, or changes to a lesser extent, it is indicated that the content of the image frame to be encoded is approximately the same as the content of the reference frame. That is, the image frame to be encoded is a scene still frame relative to the reference frame. If the content of the image frame to be encoded changes to a greater extent, it is indicated that the content of the image frame to be encoded is substantially different from the content of the reference frame. That is, the image frame to be encoded is a scene change frame with respect to the reference frame.

Optionally, in the embodiment of the present application, the process of determining the change information by the encoding device specifically includes: the encoding device determines the similarity between the image frame to be encoded and the reference frame, and then determines the change information according to the determined similarity. The similarity can characterize whether the content of the image frame to be encoded changes relative to the reference frame, as well as the magnitude of the degree of change. When the similarity is large, it indicates that the content of the image frame to be encoded is unchanged or changed to a small extent. When the similarity is small, it indicates that the content of the image frame to be encoded is changed, and the degree of change is large.

Optionally, in the embodiment of the present application, the process of determining, by the encoding device, the similarity between the image frame to be encoded and the reference frame may specifically be: the encoding device may obtain a homography matrix (homography matrix) capable of characterizing a mapping relationship of a position of the same target object in an image frame to be encoded and a reference frame, and determine a similarity between the image frame to be encoded and the reference frame according to the homography matrix and the identity matrix. It will be appreciated that when the homography matrix is an identity matrix, it is indicative of the same position of the target object in the image frame to be encoded as in the reference frame.

In some embodiments, the encoding device determines the similarity by comparing the homography matrix to coefficients at the same location in the identity matrix. The encoding apparatus may determine that the degree of similarity is large, i.e., the content of the image frame to be encoded is approximately the same as that of the reference frame, if the absolute value of the difference value of the coefficients of each identical position is less than or equal to the first threshold. If the absolute value of the difference value of the coefficients at each identical position is greater than the first threshold value and less than or equal to the second threshold value, the encoding apparatus determines that the similarity is in a middle range, that is, the content of the image frame to be encoded and the content of the reference frame are identical and the proportions of the difference are approximately identical. If the absolute value of the difference value of the coefficients of each identical position is greater than the second threshold, the encoding apparatus determines that the similarity is small, i.e., the content of the image frame to be encoded is substantially different from that of the reference frame. The first threshold and the second threshold may be preset, and the first threshold is smaller than the second threshold.

Homography matrices are typically 3 x 3 matrices, including nine coefficients. For example, the homography matrix H is as follows:

each coefficient in the homography matrix can be divided by h ₂₂ To normalize the coefficients of the homography matrix. The encoding device can normalize the homography matrix and the identity matrix after coefficientAnd comparing the coefficients at the same position one by one to obtain the similarity between the image frame to be encoded and the reference frame.

It should be noted that, in the embodiment of the present application, the homography matrix may be obtained during the process of performing anti-shake and distortion correction on the image acquired by the video source by the ISP. Of course, the homography matrix may also be obtained by other methods, and the method for obtaining the homography matrix in the embodiment of the present application is not limited herein.

In the case where the acquisition of the homography matrix depends on the anti-shake technique of the ISP, no additional computational burden is imposed on the encoding apparatus.

Further alternatively, the target information may include the first bit in addition to the change information. The first bit is the bit consumed to encode the reference frame. After determining the reference frame, the encoding device may obtain the bits consumed for encoding the reference frame, i.e. obtain the first bits. The first bit is obtained after the reference frame is encoded by the encoding method provided by the embodiment of the application.

402. The encoding device acquires the reference bits.

The reference bit is obtained according to a preset target code rate and a preset frame rate, and the reference bit is obtained by dividing the target code rate by the frame rate.

For example, assuming a target code rate of 1200kbps and a frame rate of 30 frames per second, the reference bit is 40kb.

It should be noted that, in the embodiment of the present application, the execution sequence of the step 401 and the step 402 is not related. The execution sequence of steps 401 and 402 is not limited in this embodiment of the present application.

403. The encoding device determines target bits allocated to the image frames to be encoded based on the reference bits and the target information.

After determining the target information and acquiring the reference bit, the encoding device may determine the target bit allocated to the image frame to be encoded according to the reference bit and the target information.

Optionally, in the embodiment of the present application, when the target information includes only the change information, the process of determining, by the encoding device, the target bit of the image frame to be encoded is: if the change information is used for representing that the content of the image frame to be encoded is unchanged or the change degree is small relative to the content of the reference frame, the encoding device can adjust the reference bit by a preset value so as to obtain the target bit. If the change information is used for representing that the content of the image frame to be encoded changes relative to the content of the reference frame, and the degree of the change is large, the encoding device can increase the reference bit by a preset value, so that the target bit is obtained. If the change information is used to characterize that the content of the image frame to be encoded changes relative to the reference frame, and the degree of the change is intermediate, i.e. between a smaller degree and a larger degree, it indicates that the change information has less influence on the bit allocation of the image frame to be encoded, the encoding device may determine the reference bit as the target bit.

Optionally, in the embodiment of the present application, when the target information includes the change information and the first bit, the specific process of determining, by the encoding device, the target bit of the image frame to be encoded is: the encoding device determines a second bit based on the reference bit and the first bit, and determines an adjustment amount of the second bit based on the change information, thereby obtaining a target bit. Of course, in another implementation manner, the encoding end may determine the second bit according to the reference bit and the change information, and then determine the adjustment amount of the second bit according to the first bit, so as to obtain the target bit. In the embodiment of the application, the second bit is determined according to the reference bit and the first bit, and then the adjustment amount of the second bit is determined according to the change information.

In some embodiments, the encoding device determines the second bit from the reference bit and the first bit specifically as: and if the first bit is larger than the reference bit, the first bit is reduced to obtain a second bit. Therefore, the average code rate for a period of time can reach the target code rate, and the problem of poor image quality caused by severe fluctuation of the code rate is avoided. And if the first bit is smaller than the reference bit, the first bit is regulated to be larger, and the second bit is obtained. If the first bit is equal to the reference bit, the second bit is determined to be the reference bit.

In some embodiments, the encoding device determining the adjustment amount of the second bit according to the change information, and obtaining the target bit may specifically include: if the change information is used to characterize that the content of the image frame to be encoded has not changed or has changed to a small extent relative to the reference frame, the encoding device may adjust the second bit by a preset value, thereby obtaining the target bit. If the change information is used to characterize that the content of the image frame to be encoded changes relative to the reference frame to a great extent, the encoding device may increase the second bit by a preset value to obtain the target bit in order to ensure the image quality. If the change information is used to characterize that the content of the image frame to be encoded changes relative to the reference frame, and the degree of the change is intermediate, i.e. between a smaller degree and a larger degree, it indicates that the change information has less influence on the bit allocation of the image frame to be encoded, the encoding device may determine the second bit as the target bit.

For example, assume that the reference bit is 40kb and the adjustment amount of each bit is 1kb. If the first bit is 45kb and 45kb is greater than 40kb, the encoding apparatus determines that the second bit is 44kb in order to achieve the target code rate while ensuring the image quality. If the change information is used to characterize that the content of the image frame to be encoded is unchanged or changed to a small extent relative to the reference frame, the second bit may be further adjusted downward to determine that the target bit is 43kb. If the change information is used to characterize that the content of the image frame to be encoded changes relative to the reference frame, and the degree of the change is large, in order to ensure the image quality, the second bit can be enlarged, and the target bit is determined to be 45kb. If the change information is used for representing that the content of the image frame to be encoded changes relative to the reference frame, and the degree of the change is in the middle degree, the change information has smaller influence on bit allocation of the image frame to be encoded, and the target bit can be determined to be 44kb.

For another example, assume that the reference bit is 40kb and the adjustment amount of each bit is 1kb. If the first bit is 38kb and 38kb is less than 40kb, the encoding device determines the second bit to be 39kb. If the change information is used to characterize that the content of the image frame to be encoded has not changed or has changed to a small extent relative to the reference frame, the target bit is determined to be 38kb. If the change information is used to characterize that the content of the image frame to be encoded changes relative to the reference frame, and the degree of the change is large, the target bit is determined to be 40kb. If the change information is used to characterize the content of the image frame to be encoded relative to the reference frame, and the degree of change is at an intermediate level, the target bit is determined to be 39kb.

404. The encoding device performs encoding of an image frame to be encoded, quantization in the encoding being performed according to the target bits.

After determining the target bit of the image frame to be encoded, the encoding device may determine the adjustment amount of the quantization parameter according to the difference between the target bit of the image frame to be encoded and the target bit of the reference frame. Then, the encoding device determines the quantization parameter of the image frame to be encoded according to the determined adjustment amount and the quantization parameter of the reference frame. The encoding apparatus may perform quantization on the image frame to be encoded according to quantization parameters of the image frame to be encoded. The more bits allocated to an image frame, the smaller the quantization parameter of that image frame.

For example, assuming that the reference frame is the previous image frame to the image frame to be encoded, as shown in fig. 5, a scene diagram of the encoding apparatus adjusting the quantization parameter of the image frame to be encoded is shown.

According to the coding method provided by the embodiment of the application, the coding equipment determines the target information, acquires the reference bit, determines the target bit of the image frame to be coded according to the reference bit and the target information, and codes the image frame to be coded according to the target bit. The change information can reflect the change of the content of the image frame to be encoded relative to the reference frame, and the encoding end determines the target bit of the image frame to be encoded by taking the reference bit as a base line according to the content of the image frame to be encoded, so that the target bits of all the image frames to be encoded float up and down on the base line, and the code rate after video encoding can reach the target code rate. For example, if the change of the content of the image frame to be encoded relative to the reference frame is large, the bit is added as the target bit on the basis of the reference bit to improve the image quality. If the change of the content of the image frame to be encoded relative to the reference frame is small, the bit is reduced on the basis of the reference bit as a target bit, so that the resource waste is reduced. Compared with the mode of determining the target bit by taking the reference bit as the base line in the prior art and determining the target bit according to the information of the reference frame, the method and the device for determining the target bit in the embodiment of the application have the advantages that the determined target bit is more accurate, so that the resource waste can be reduced or the image quality can be improved.

For example, assuming that the frames 0 to 4 of the video sequence are stable, the content of the image changes at the 5 th frame, and assuming that the reference frame is the previous frame, if the rate control is performed according to the method in the prior art, the target bit of the 5 th frame is determined according to the reference bit and the actual bit consumed by the 4 th frame, which may result in insufficient bit allocation of the 5 th frame and poor image quality of the 5 th frame. If the code rate control is performed according to the method in the embodiment of the application, more bits than the reference bits are allocated to the 5 th frame according to the reference bits and the change information, so that the image quality of the 5 th frame is improved.

Furthermore, when determining the target bit of the image frame to be encoded, the encoding end uses the reference bit as a baseline, so that not only the change information, but also the actual bit consumed by encoding the reference frame, namely the first bit, can be referred to. And when the first bit is larger than the reference bit, the bit number is reduced, and when the first bit is smaller than the reference bit, the bit number is increased, so that the accuracy of the determined target bit is further improved.

Optionally, in the embodiment of the present application, it is determined in step 403 that the target bit is a bit allocation between frames, and the encoding device needs to perform block-level bit allocation. In particular, the target bits may be divided by the number of image blocks of the image frame to be encoded to obtain an average bit allocated to each image block.

Further, in case the change information is used to characterize that the content of the image frame to be encoded changes with respect to the reference frame, and the degree of the change is an intermediate degree, the encoding device may optimize the block-level bit allocation of the image frame to be encoded.

Specifically, after the encoding device acquires the homography matrix and determines the target bit, the encoding method may further include: the encoding device determines a motion vector of the image frame to be encoded relative to the reference frame according to the homography matrix and the position of the first pixel point, and determines an image block bit of each image block in the image frame to be encoded according to the motion vector and the target bit. The first pixel point is a pixel point at the center of the image frame to be encoded.

It will be appreciated that the above described motion vectors are used to characterize the direction of motion of objects in an image frame to be encoded relative to a reference frame. For example, left or right, up or down.

In some embodiments, the encoding device may determine the motion vectors mvx and mvy from the homography matrix and the coordinates (x, y) of the first pixel point. Wherein mvx and mvy satisfy the following formulas, respectively:

in some embodiments, the encoding device determines an image block bit of each image block in the image frame to be encoded according to the motion vector and the target bit, and specifically may include: the encoding device first determines a target image block in an image frame to be encoded based on the motion vector. Wherein the reference frame does not include the content of the target image block, that is, the target image block refers to the image block whose image content appears in the image frame to be encoded and which does not appear in the reference frame. Then, the encoding apparatus determines an average bit of the image blocks in the image frame to be encoded, based on the target bit, the average bit being obtained by dividing the target bit by the number of the image blocks of the image frame to be encoded, and determines the image block bit of the target image block based on the average bit. Specifically, the encoding device may increase the average bit to obtain the image block bit of the target image block, and use the average bit as the image block bit of the other image blocks except the target image block. In this case, the sum of the image block bits of all the image blocks of the image frame to be encoded is larger than the target bit of the image frame to be encoded.

It will be appreciated that since the target image block has no reference image block relative to the reference frame, it is necessary to allocate more bits to the target image block in order to improve the image quality of the image frame to be encoded.

Illustratively, if the encoding apparatus obtains a larger value of the motion vector mvx and a value of mvy is close to 0, the encoding apparatus determines that the motion direction of the image frame to be encoded is horizontal to the right. The encoding apparatus determines, as a target image block, one or more image blocks located on the right side among the image blocks of the image frame to be encoded, according to the motion direction. Assuming that the target bit is 44kb, the number of image blocks of the image frame to be encoded is 11, and the average bit is 4kb. Assuming that the number of target tiles is 3, i.e., the number of other tiles is 8, and the one-time bit adjustment amount is 1kb, the encoding apparatus determines that the tile bit of each of the 3 target tiles is 5kb and the tile bit of each of the 8 other tiles is 4kb. In this case, the allocated bits of the image frame to be encoded are 47kb.

Optionally, in an embodiment of the present application, the encoding method may further include: the encoding device determines a prediction mode corresponding to an image block of the image frame to be encoded according to the change information, wherein the prediction mode is intra-frame prediction or inter-frame prediction. For example, the encoding device may determine the rate-distortion cost of the prediction unit according to the change information, and then determine the prediction mode corresponding to the image block according to the rate-distortion cost. In this case, the step 404 may specifically include: the encoding device performs encoding on the image frame to be encoded according to the target bit and the prediction mode. In this way, since the change information is obtained by analyzing the global content of the image frame to be encoded, the computational complexity of determining the prediction mode can be greatly reduced by determining the prediction mode of the image frame to be encoded based on the change information.

It should be noted that, in the embodiment of the present application, if the change information is used to characterize that the content of the image frame to be encoded changes relative to the reference frame, and the degree of the change is large, the prediction mode tends to be intra-prediction, but whether the final result is intra-prediction needs to be determined according to the rate-distortion cost. If the change information is used to characterize that the content of the image frame to be encoded has not changed, or has changed to a lesser extent, relative to the reference frame, the prediction mode is more prone to inter prediction, but the end result is whether inter prediction needs to be determined according to the rate-distortion cost. If the change information is used to characterize the degree to which the content of the image frame to be encoded changes relative to the reference frame as an intermediate degree, the change information has less effect on the result of the prediction mode.

The foregoing description of the solution provided by the embodiments of the present application has been mainly presented in terms of a method. To achieve the above functions, it includes corresponding hardware structures and/or software modules that perform the respective functions. Those of skill in the art will readily appreciate that the various illustrative algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

As shown in fig. 6, a schematic structural diagram of an encoding apparatus 60 according to an embodiment of the present application is provided, where the encoding apparatus 60 is configured to perform the encoding method shown in fig. 4. The encoding device 60 may include a determination unit 61, an acquisition unit 62, and an encoding unit 63.

A determining unit 61 for determining target information comprising change information for characterizing a change in content of the image frame to be encoded relative to the reference frame. For example, in connection with fig. 4, the determination unit 61 may be used to perform step 401. An acquisition unit 62 for acquiring reference bits, the reference bits being obtained according to a preset target code rate and a preset frame rate. For example, in connection with fig. 4, the acquisition unit 62 may be used to perform step 402. The determining unit 61 is further configured to determine target bits allocated to the image frame to be encoded based on the target information and the reference bits acquired by the acquiring unit 62. For example, in connection with fig. 4, the determination unit 61 may also be used to perform step 403. An encoding unit 63 for performing encoding on the image frame to be encoded, the quantization in the encoding being performed according to the target bit determined by the determining unit 61. For example, in connection with fig. 4, the encoding unit 63 may be used to perform step 404.

Optionally, the determining unit 61 is specifically configured to: and determining the similarity between the image frame to be encoded and the reference frame, and determining the change information according to the similarity.

Optionally, the determining unit 61 is specifically configured to: and acquiring a homography matrix, and determining the similarity according to the homography matrix and the identity matrix. The homography matrix is used for representing the mapping relation of the position of the target object in the image frame to be encoded and the reference frame.

Optionally, the target information further comprises a first bit, the first bit being a bit consumed for encoding a reference frame, the reference frame being a forward encoded image frame or a backward encoded image frame of the image frame to be encoded.

Optionally, the determining unit 61 is specifically configured to: and determining a second bit according to the first bit and the reference bit, and determining the adjustment quantity of the second bit according to the change information to obtain a target bit.

Optionally, the determining unit 61 is further configured to determine a motion vector of the image frame to be encoded relative to the reference frame according to the homography matrix and a position of a first pixel, where the first pixel is a pixel at a center position of the image frame to be encoded; image block bits for each image block in the image frame to be encoded are determined based on the motion vector and the target bits.

Optionally, the determining unit 61 is specifically configured to: determining a target image block in an image frame to be encoded according to the motion vector, wherein the reference frame does not comprise the content of the target image block; determining average bits of image blocks in an image frame to be encoded according to the target bits; the image block bits of the target image block are determined from the average bits.

Optionally, the determining unit 61 is further configured to determine, according to the change information, a prediction mode corresponding to an image block of the image frame to be encoded, where the prediction mode is intra prediction or inter prediction. The encoding unit 63 is specifically configured to perform encoding on the image frame to be encoded according to the target bit and the prediction mode.

Of course, the encoding device 60 provided in the embodiment of the present application includes, but is not limited to, the above-described modules.

In actual implementation, the determining unit 61, the acquiring unit 62, and the encoding unit 63 may be implemented by a video encoder of the encoding apparatus shown in fig. 3. The specific implementation process may refer to the description of the coding method shown in fig. 4, and will not be repeated here.

Another embodiment of the present application also provides a computer readable storage medium, where computer instructions are stored, where the computer instructions, when executed on an encoding device, cause the encoding device to perform the steps performed by the encoding device in the method flow shown in the foregoing method embodiment.

Another embodiment of the present application also provides a chip system, which is applied to an encoding apparatus. The system-on-chip includes one or more interface circuits, and one or more processors. The interface circuit and the processor are interconnected by a wire. The interface circuit is for receiving signals from the memory of the encoding device and for sending signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the encoding apparatus performs the steps performed by the encoding apparatus in the method flow shown in the method embodiment described above.

In another embodiment of the present application, there is also provided a computer program product including computer instructions which, when run on an encoding apparatus, cause the encoding apparatus to perform the steps performed by the encoding apparatus in the method flow shown in the method embodiment described above.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer-executable instructions are loaded and executed on a computer, the processes or functions in accordance with embodiments of the present application are fully or partially produced. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, a website, computer, server, or data center via a wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. Computer readable storage media can be any available media that can be accessed by a computer or data storage devices including one or more servers, data centers, etc. that can be integrated with the media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

The foregoing is only a specific embodiment of the present application. Variations and alternatives will occur to those skilled in the art based on the detailed description provided herein and are intended to be included within the scope of the application.

Claims

A method of encoding, comprising:

determining target information, wherein the target information comprises change information, and the change information is used for representing the change of the content of an image frame to be encoded relative to a reference frame;

obtaining reference bits, wherein the reference bits are obtained according to a preset target code rate and a preset frame rate;

determining target bits allocated to the image frame to be encoded according to the reference bits and the target information;

and performing encoding on the image frame to be encoded, wherein quantization in the encoding is completed according to the target bit.
The encoding method according to claim 1, wherein determining the change information comprises:

determining the similarity between the image frame to be encoded and the reference frame;

and determining the change information according to the similarity.
The encoding method according to claim 2, wherein the determining the similarity between the image frame to be encoded and the reference frame comprises:

Acquiring a homography matrix, wherein the homography matrix is used for representing the mapping relation of the position of a target object in the image frame to be encoded and the reference frame;

and determining the similarity according to the homography matrix and the identity matrix.
A coding method according to any of claims 1-3, characterized in that the target information further comprises a first bit, which is a bit consumed for coding the reference frame, which is a forward coded image frame or a backward coded image frame of the image frame to be coded.
The encoding method according to claim 4, wherein the determining the target bits allocated to the image frame to be encoded based on the reference bits and the target information comprises:

determining a second bit according to the reference bit and the first bit;

and determining the adjustment quantity of the second bit according to the change information to obtain the target bit.
A coding method according to claim 3, characterized in that the coding method further comprises:

determining a motion vector of the image frame to be encoded relative to the reference frame according to the homography matrix and the position of a first pixel point, wherein the first pixel point is a pixel point at the central position of the image frame to be encoded;

And determining the image block bit of each image block in the image frame to be encoded according to the motion vector and the target bit.
The encoding method according to claim 6, wherein the determining the image block bit of each image block in the image frame to be encoded based on the motion vector and the target bit comprises:

determining a target image block in the image frame to be encoded according to the motion vector, wherein the reference frame does not comprise the content of the target image block;

determining average bits of image blocks in the image frame to be encoded according to the target bits;

and determining the image block bit of the target image block according to the average bit.
The encoding method according to any one of claims 1 to 7, characterized in that the encoding method further comprises:

determining a prediction mode corresponding to the image block of the image frame to be encoded according to the change information, wherein the prediction mode is intra-frame prediction or inter-frame prediction;

the encoding of the image frame to be encoded comprises:

and performing encoding on the image frame to be encoded according to the target bit and the prediction mode.
An encoding device, comprising:

A determining unit configured to determine target information, the target information including change information for characterizing a change in content of an image frame to be encoded relative to a reference frame;

the acquisition unit is used for acquiring reference bits, wherein the reference bits are obtained according to a preset target code rate and a preset frame rate;

the determining unit is further configured to determine a target bit allocated to the image frame to be encoded according to the target information and the reference bit acquired by the acquiring unit;

and an encoding unit configured to perform encoding on the image frame to be encoded, wherein quantization in the encoding is performed according to the target bit determined by the determining unit.
The encoding device according to claim 9, wherein the determining unit is specifically configured to:

determining the similarity between the image frame to be encoded and the reference frame;

and determining the change information according to the similarity.
The encoding device according to claim 10, wherein the determining unit is specifically configured to:

acquiring a homography matrix, wherein the homography matrix is used for representing the mapping relation of the position of a target object in the image frame to be encoded and the reference frame;

And determining the similarity according to the homography matrix and the identity matrix.
The encoding device according to any one of claims 9-11, wherein the target information further comprises a first bit, the first bit being a bit consumed for encoding the reference frame, the reference frame being a forward encoded image frame or a backward encoded image frame of the image frame to be encoded.
The encoding device according to claim 12, wherein the determining unit is specifically configured to:

determining a second bit according to the reference bit and the first bit;

and determining the adjustment quantity of the second bit according to the change information to obtain the target bit.
The coding apparatus of claim 11, wherein the code means comprises,

the determining unit is further configured to determine a motion vector of the image frame to be encoded relative to the reference frame according to the homography matrix and a position of a first pixel, where the first pixel is a pixel at a center position of the image frame to be encoded; and determining the image block bit of each image block in the image frame to be encoded according to the motion vector and the target bit.
The encoding device according to claim 14, wherein the determining unit is specifically configured to:

determining a target image block in the image frame to be encoded according to the motion vector, wherein the reference frame does not comprise the content of the target image block;

determining average bits of image blocks in the image frame to be encoded according to the target bits;

and determining the image block bit of the target image block according to the average bit.
The coding device according to any of the claims 9-15, wherein,

the determining unit is further configured to determine a prediction mode corresponding to an image block of the image frame to be encoded according to the change information, where the prediction mode is intra-frame prediction or inter-frame prediction;

the coding unit is specifically configured to perform coding on the image frame to be coded according to the target bit and the prediction mode.
An encoding device, wherein the encoding device comprises a memory and a processor; the memory is coupled to the processor; the memory is used for storing computer program codes, and the computer program codes comprise computer instructions; the encoding apparatus performs the encoding method of any one of claims 1-8 when the processor executes the computer instructions.
A computer readable storage medium comprising computer instructions which, when run on an encoding apparatus, cause the encoding apparatus to perform the encoding method of any one of claims 1-8.