WO2023061129A1 - Video encoding method and apparatus, device, and storage medium - Google Patents

Abstract

A video encoding method and apparatus, a device, and a storage medium, relating to the technical field of digital signal processing. The method comprises: obtaining a target image frame to be encoded (210); in the case that the number of reference image frames of the target image frame does not meet an analysis condition, determining a frame type and an encoding parameter of the target image frame according to a first mode, wherein the first mode is a mode in which the frame type and the encoding parameter are determined not according to the reference image frames (220); and encoding the target image frame according to the frame type and the encoding parameter of the target image frame (230). According to the present method, the waiting process that an encoding device obtains the enough number of reference image frames in the case that the number of the reference image frames of an image frame to be encoded does not meet analysis is eliminated, the waiting time period of the encoding device is shortened, excessive parameters and complex calculation modes do not need to be introduced, such that the encoding time delay can be effectively reduced.

Description

Video coding method, device, equipment and storage medium

This application claims the priority of the Chinese patent application with application number 202111188246.2 and titled "Video Coding Method, Device, Equipment, and Storage Medium" filed on October 12, 2021, the entire contents of which are incorporated by reference in this application middle.

technical field

The present application relates to the technical field of digital signal processing, and in particular to a video encoding method, device, equipment and storage medium.

Background technique

Before video data transmission, in order to reduce the amount of transmitted data and shorten the transmission time, the encoding device needs to encode the video image frame, compress the data amount of the video image frame, and increase the transmission speed.

In related technologies, before encoding a certain target image frame to be encoded, it is necessary to predict the frame type and encoding parameters of the target image frame according to other multiple reference image frames, by calculating the target image frame using different encoding methods The encoding loss that will be brought, determine the most suitable frame type and encoding parameters for the target image frame. The encoding device encodes the target image frame according to the determined frame type and encoding parameters to obtain a code stream corresponding to the target image frame. The frame type and encoding parameters of the target image frame are predicted, and the encoding is performed according to the prediction result, so that stable video playback quality can be obtained.

However, in some cases, when encoding an image frame to be encoded according to other multiple reference image frames, it is necessary to wait for these reference image frames to be input into the encoding device, which will cause a large encoding delay.

Contents of the invention

Embodiments of the present application provide a video encoding method, device, equipment, and storage medium. The technical solution is as follows:

According to an aspect of an embodiment of the present application, a video coding method is provided, the method is executed by a computer device, and the method includes:

Obtain the target image frame to be encoded;

In the case that the number of reference image frames of the target image frame does not satisfy the analysis condition, determine the frame type and encoding parameters of the target image frame according to the first method; wherein, the first method refers to not according to the The method of determining the frame type and encoding parameters with reference to the image frame;

Encoding the target image frame according to the frame type and encoding parameters of the target image frame.

According to an aspect of an embodiment of the present application, a video encoding device is provided, and the device includes:

An image frame acquisition module, configured to acquire a target image frame to be encoded;

The first encoding determination module is configured to determine the frame type and encoding parameters of the target image frame in a first manner when the number of reference image frames of the target image frame does not meet the analysis condition; wherein, the first One method refers to a method of determining the frame type and encoding parameters not according to the reference image frame;

An image frame encoding module, configured to encode the target image frame according to the frame type and encoding parameters of the target image frame.

According to an aspect of an embodiment of the present application, there is provided a computer device, the computer device includes a processor and a memory, a computer program is stored in the memory, and the processor executes the computer program to implement the above video coding method .

According to an aspect of the embodiments of the present application, a computer-readable storage medium is provided, where a computer program is stored in the storage medium, and the computer program is used to be executed by a processor, so as to implement the above video encoding method.

According to one aspect of the present application, a computer program product is provided, which, when the computer program product is run on a computer device, causes the computer device to execute the above video encoding method.

The technical solutions provided in the embodiments of the present application can bring the following beneficial effects:

When the target image frame to be encoded does not meet the analysis conditions, the target image frame is directly encoded without waiting for the number of reference image frames of the target image frame to meet the analysis requirements, eliminating multiple other references waiting for the target image frame The waiting time caused by the image frame reduces the encoding delay.

Description of drawings

FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

FIG. 2 is a flowchart of a video coding method provided by an embodiment of the present application;

FIG. 3 is a flowchart of a video encoding method provided by another embodiment of the present application;

FIG. 4 is a schematic diagram of a video encoding method provided by another embodiment of the present application;

FIG. 5 is a schematic diagram of a video encoding method provided by another embodiment of the present application;

FIG. 6 is a schematic diagram of a video encoding process provided by another embodiment of the present application;

FIG. 7 is a block diagram of a video encoding device provided by an embodiment of the present application;

Fig. 8 is a block diagram of a video encoding device provided by another embodiment of the present application.

Detailed ways

Before introducing the technical solution of this application, some background technical knowledge involved in this application will be introduced. The following related technologies may be optionally combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them belong to the protection scope of the embodiments of the present application. The embodiment of the present application includes at least part of the following contents.

First, the nouns that may appear in this application are introduced.

A frame refers to a still picture in a video.

FPS (Frames Per Second, frame rate) refers to the number of frames to refresh the picture per second, or the number of times the graphics processor refreshes per second. The higher the frame rate, the more pictures are refreshed in one second, the more realistic the video picture, and the smoother the action in the video.

DR (Data Rate, code stream) refers to the data flow used by a video file per unit time. The larger the code stream of a video file, the smaller the compression ratio and the higher the picture quality. The larger the code stream, the larger the sampling rate per unit time, the higher the accuracy of the data stream, the closer the processed file is to the original file, the better the image quality, the clearer the picture quality, and the higher the decoding ability of the playback device is required. high.

Video coding refers to the process of removing redundant information in video images through specific compression techniques.

GOP (Group of Picture, key frame period) refers to the distance between two adjacent I frames (Intra Picture, internal picture) frames, that is, the maximum number of frames in a frame group, and the decoding process is decoded in units of GOP .

QP (Quantizer Parameter, quantization parameter), used to reflect the compression of spatial details. The smaller the value of QP, the finer the quantization, the greater the degree of detail retention in the image, the higher the quality of the picture, the longer the code stream generated, that is, the greater the amount of data; the larger the QP, the greater the loss of some details in the picture , the lower the code rate, the greater the degree of image distortion.

The Y (Lumina, brightness) U (Chrominance, chroma) V (Chroma, chroma) signal is introduced and explained below.

YUV is a digital representation of color, Y is used to represent the grayscale value of the pixel, U and V are used to represent the color of the pixel, including the color and saturation of the pixel. Since the sensitivity of the human eye to changes in hue is lower than that of the human eye to brightness changes, greater compression can be performed in the chrominance dimension during the video encoding process, so that the code stream obtained after video encoding is smaller.

The sampling formats of YUV images include:

(1) 4:4:4, that is, the sampling rate of the three channels of YUV is the same. In the generated image, the three component information of each pixel is complete (each component is usually 8 bits), and each uncompressed A pixel takes 3 bytes. For example, there are four pixels: [Y0U0V0], [Y1U1V1], [Y2U2V2] and [Y3U3V3], and the corresponding code stream after sampling is: Y0U0V0; Y1U1V1; Y2U2V2; Y3U3V3.

(2) 4:2:2, that is, the sampling rate of each chroma channel is half of that of the luma channel, therefore, the chroma sampling rate in the horizontal direction is half of 4:4:4. For an uncompressed 8-bit quantized image, each macropixel consisting of two horizontally adjacent pixels requires 4 bytes of memory (2 bytes for brightness and 1 byte for each of the two chroma). For example, there are four pixels: [Y0U0V0], [Y1U1V1], [Y2U2V2] and [Y3U3V3], and the corresponding code stream after sampling is: Y0U0; Y1V1; Y2U2; Y3V3.

(3) 4:2:0, each scan line includes a chrominance component stored at a sampling rate of 2:1. Adjacent scan lines store different chroma components, for example, if the first line is 4:2:0, the second line is 4:0:2, the third line is 4:2:0, and so on. For each chroma component, the sampling rate in the horizontal and vertical directions is 2:1, which is equivalent to a chroma sampling rate of 4:1. For uncompressed 8bit quantized video, each macropixel consisting of 2x2 adjacent pixels in 2 rows and 2 columns requires 6 bytes of memory (4 bytes for brightness, 1 byte for each of the two chroma ). For example, there are eight pixels as: [Y0 U0 V0], [Y1 U1 V1], [Y2 U2 V2], [Y3 U3 V3], [Y5 U5 V5], [Y6 U6 V6], [Y7 U7 V7] and [Y8 U8 V8], the code stream after sampling is: Y0 U0; Y1; Y2 U2; Y3; Y5V5; Y6; Y7V7; Y8.

The frame types are introduced and explained below.

Frame types mainly include the following categories:

(1) I frame: also known as a key frame, the I frame is used to represent the details of the image background and the moving subject, and the amount of information occupied by the data is relatively large. In the encoding process, the compression rate of the I frame is the smallest. When decoding, the I frame does not depend on other frames for decoding, and can be used as a reference image frame in the decoding process of other frame types. The I frame is the basic frame of the GOP (the first frame in the GOP), and one I frame is included in a group of GOPs.

(2) IDR (Instantaneous Decoding Refresh, instant decoding refresh) frame: IDR frame is a kind of I frame, before decoding the IDR frame, the decoding device will clear the forward and backward reference buffer (the reference buffer is used to store in An image frame that has a reference role in the decoding process), so that any frame after the IDR frame cannot be decoded with reference to any frame before the IDR appears. As a random access point for video security, the IDR frame can prevent the transmission of wrong image frames from affecting the decoding results of other frames for a long time.

(3) P (Predictive-coded Picture, forward predictive coded image) frame: The difference information between the current image frame and the previous frames is recorded in the P frame. During the encoding process, the compression rate of the P frame is relatively high. For the decoding process of the P frame, it is necessary to refer to part of the information in the previously buffered picture to decode and generate a picture.

(4) B (Bidirectionally Predicted Picture, bidirectional predictive coding image) frame: B frame records the difference between the current frame and the preceding and following frames, and the maximum amount of compression in the encoding process. When decoding a B frame, it is necessary to refer to a forward or/and backward I frame or P frame, and the forward direction refers to an image frame that appears earlier than the current frame on the time axis.

Next, the video coding method is introduced and explained.

Video coding includes predictive coding, transform coding, quantization and entropy coding, wherein predictive coding includes intra-frame predictive coding and inter-frame predictive coding. Intra-frame predictive coding uses the correlation of the video space domain to compare adjacent macroblocks in the same frame. Inter-frame predictive coding exploits temporal-spatial correlation of video. The inter-frame predictive coding uses the pixels of the adjacent coded image frame to predict the corresponding pixel of the current image frame after coding, and obtains the residual signal. Inter-frame predictive coding includes: motion estimation, MV prediction and weighted prediction, motion compensation, etc. Among them, motion estimation is used to match the macroblock of the current image frame with the closest macroblock in the reference image frame, and motion estimation is used to find Referring to the macroblocks in the image frame, the macroblocks in the reference image frame that are most similar to the macroblocks in the current image frame are determined using matching criteria such as minimum mean square error and minimum mean absolute error.

Quantization refers to the processing of time-discrete signals to make them discrete in amplitude. Quantization can be divided into uniform quantization and non-uniform quantization, among which non-uniform quantization is better for small signals.

Please refer to FIG. 1 , which shows a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment may include an encoding device 10 and a decoding device 20 .

The coding device 10 is a device having functions such as video coding, video data storage, and video data transceiving. The encoding device 10 may be a device such as a computer, a mobile phone, a tablet computer, a smart TV, a video camera, or a vehicle system. The encoding device 10 encodes a target image frame to be encoded in a certain frame. Optionally, the encoding device 10 directly specifies the encoding parameters of the target image frame, and encodes the target image frame according to the specified encoding parameters; optionally, The encoding device 10 first predicts the frame type and encoding parameters of the target image frame according to other reference image frames, and then encodes the target image frame according to the encoding parameters. The encoding device 10 encodes the target image frame and sends the code stream to the decoding device 20 in the form of a video data stream.

The decoding device 20 is a device having functions such as video encoding and video data sending and receiving. The decoding device may be a server or a terminal device, and the server may be a background server running a target application program. The target application program includes a video application program, a live broadcast application program, a social application program with a video communication function, etc., and the type of the target application program is not limited here. The decoding device 20 may be a server, or a server cluster composed of multiple servers, or a cloud computing service center. There is a target application program running on the terminal device, and besides the function of decoding video data stream, the terminal device has at least one other function, such as video playing function, information sending function and so on.

Please refer to FIG. 2, which shows a flow chart of a video coding method provided by an embodiment of the present application. The execution subject of each step of the method may be a coding device, and the method may include the following steps (210-230) At least one step of:

Step 210, acquiring a target image frame to be encoded.

An image frame is a still picture, and multiple image frames are switched continuously to form a video. In some embodiments, the image frame is a signal in YUV format. Coding refers to the process in which a coding device compresses a picture in one format to obtain a picture in another format. The data volume of the encoded picture is smaller than the data volume of the original picture before encoding.

The encoding device acquires the target image frame to be encoded, and encodes the acquired target image frame. Optionally, the encoding device acquires the target image frame from a certain video file in the video database; optionally, the encoding device acquires the target image frame through a device with a video frame acquisition function such as a camera. The target image frame can be a frame in a video file, or a frame of a scene captured by a screen capture device during a live video broadcast or a video call. limited. In an example, in the application scenario of video playback, the encoding device can be set on the background server of the target application program, and the encoding device obtains a certain target image frame to be encoded from the database of the server, wherein the target image frame comes from the received to the video file corresponding to the play instruction command. In another example, in a live video application scenario, the camera captures a frame of live images every 25 ms, and the encoding device obtains the target image frame to be encoded through the camera.

Step 220, in the case that the number of reference image frames of the target image frame does not meet the analysis conditions, determine the frame type and encoding parameters of the target image frame according to the first method; wherein, the first method refers to determining the frame without referring to the reference image frame Type and way of encoding parameters.

The reference image frame and the target image frame are image frames in the same video file, or image frames collected by the same image acquisition device within a certain time interval. The reference image frame may be an image frame that has been encoded, or an image frame that has not been encoded. In some embodiments, the reference image frame and the target image frame have part of the same features, and the target image frame can be encoded according to the information in the reference image frame. In some embodiments, the reference image frame is stored in the encoding device.

The analysis condition refers to the condition for judging the frame type of the target image frame and the way of determining the coding parameters. The frame type is used to represent the compressibility of the target image frame. The frame type includes I frame, P frame and B frame. For details about the frame type, please refer to the above, and will not be repeated here. The coding parameters include code rate control parameters, and the coding parameters are used to keep the code rate stable during the coding process and reduce the distortion rate of the image frame during the coding process.

In some embodiments, the analysis condition includes that the number of reference image frames of the target image frame is greater than or equal to a first threshold. In some embodiments, after the encoding device acquires the target image frame to be encoded, it determines the encoding method for acquiring the target image frame according to the number of reference image frames in the encoding device. If the number of reference image frames of the target image frame does not meet the analysis condition, Then the encoding device adopts the first method to determine the frame type and encoding parameters of the target image frame; otherwise, if the number of reference image frames of the target image frame satisfies the analysis condition, the encoding device uses other methods except the first method to determine the target image frame frame type and encoding parameters.

In some embodiments, after obtaining the target image frame to be encoded, it also includes: adding the target image frame to the pre-coding queue; when the length of the pre-coding queue is less than the rated length, determining the reference image frame of the target image frame The number of does not meet the analysis condition; when the length of the precoding queue is equal to the rated length, it is determined that the number of reference image frames of the target image frame meets the analysis condition.

The length of the pre-coding queue refers to the number of image frames stored in the pre-coding queue. In some embodiments, the nominal length is equal to the size of the pre-coding queue, and the size of the pre-coding queue refers to the maximum number of image frames that can be accommodated in the predefined queue. In this case, before the number of image frames stored in the pre-encoding queue reaches the size of the pre-encoding queue, the encoding device adopts a first method to determine the frame type and encoding parameters of the acquired target image frame.

In some embodiments, after the encoding device acquires the target image frame to be encoded, it adds the target image frame to a pre-encoding queue, and multiple image frames stored in the pre-encoding queue are used as reference image frames of the target image frame. In some embodiments, the encoding device stores the frame type and encoding parameters of the target image frame to be encoded in the pre-encoding queue, for example, the encoding device stores the target image frame to be encoded, the frame type and encoding parameter corresponding to the target image frame Stored in the same unit of the pre-encoding queue; for another example, the encoding device separately stores the frame type and encoding parameters of a target image frame to be encoded in a unit of the pre-encoding queue, and this unit is used to store the The unit of the target image frame has a mapping relationship, and the encoding device determines the frame type and encoding parameters corresponding to the target image frame through the mapping relationship.

Step 230: Encode the target image frame according to the frame type and encoding parameters of the target image frame.

In some embodiments, the encoding device encodes the target image frame according to the frame type and encoding parameters of the target image frame, obtains a code stream corresponding to the target image frame, and outputs the code stream corresponding to the target image frame. For details about how the encoding device encodes a target image frame according to the frame type and encoding parameters of the target image frame to be encoded, please refer to the relevant background introduction, and details are not repeated here.

To sum up, this method determines the frame type and coding parameters of the target image frame through the first method for the target image frame to be encoded whose number of reference image frames does not meet the analysis conditions, because the first method does not need to refer to the target image frame The image frame determines the frame type and encoding parameters of the target image frame, thus eliminating the waiting time for the encoding device to acquire a sufficient number of reference image frames when the number of reference image frames of the target image frame does not meet the analysis requirements. This method does not require Introducing too many parameters and complex calculation methods can effectively reduce the encoding delay.

The video coding process provided by this method is introduced and described through several embodiments.

Please refer to FIG. 3 , which shows a flowchart of a video encoding method provided by another embodiment of the present application.

Step 310, acquiring a target image frame to be encoded.

In some embodiments, after obtaining the target image frame to be encoded, it also includes: adding the target image frame to the pre-encoding queue; when the length of the pre-encoding queue is less than the rated length, the encoding device determines the reference of the target image frame The number of image frames does not meet the analysis condition; when the length of the pre-coding queue is equal to the rated length, the encoding device determines that the number of reference image frames of the target image frame meets the analysis condition.

Step 320, if the number of reference image frames of the target image frame does not satisfy the analysis condition, determine a predefined frame type as the frame type of the target image frame.

In some embodiments, the predefined frame type is stored in the encoding device, or the predefined frame type is determined according to a picture capture tool or the video to be encoded. The predefined frame type can be determined according to actual application conditions such as network transmission delay, encoding capability of encoding equipment, and video playback quality requirements, and is not limited here in this application. In some embodiments, when the number of reference image frames of the target image frame does not satisfy the analysis condition, the encoding device determines the frame type of the target image frame according to a predefined frame type.

Wherein, step 320, in a possible implementation manner, includes step 322, obtaining a predefined frame type arrangement, and the predefined frame type arrangement refers to a predetermined arrangement of frame types corresponding to a plurality of image frames respectively. ; Determine the frame type of the target image frame from the predefined frame type arrangement based on the sequence number of the target image frame.

In another possible implementation manner, step 320 includes step 324, obtaining predefined frame type configuration information, and the predefined frame type configuration information includes the configuration quantity of at least one frame type; Type and predefined frame type configuration information to determine the frame type of the target image frame.

In some embodiments, a predefined arrangement of frame types is stored in the encoding device. In some embodiments, at least one of I frame, P frame and B frame is included in the predefined frame type configuration information, for example, the predefined frame type arrangement may include: IIIIIIIIII, IPPPIPPPIP, IBPBPIBPP, etc. The arrangement of predefined frame types can be set according to experience, actual video playback requirements and other factors, which is not limited in this application.

In some embodiments, the sequence number of the target image frame is determined according to the order in which the encoding device acquires the target image frames. For example, if a certain target image frame is the ninth image frame acquired by the encoding device, the sequence number of the target image frame is 9. In some embodiments, after the encoding device acquires the target image frame, it acquires a predefined frame type arrangement, and the encoding device determines the frame type of the target image according to the predefined frame type arrangement and the sequence number of the target image. For example, the encoding device acquires a certain target image frame to be encoded, and the target image frame is the first target image frame acquired by the encoding device, then the sequence number of the target image frame is 1, and the reference image frame of the target image frame When the quantity does not meet the analysis conditions, the predefined frame type arrangement stored in the encoding device is IPPBPP, and the encoding device determines that the frame type of the target image frame is I according to the sequence number of the target image frame and the predefined frame type arrangement. frame.

In some embodiments, the length of the predefined frame type arrangement is determined according to the length of the precoding queue. For example, if the size of the precoding queue is 10 frames, the length of the predefined frame type arrangement is greater than or equal to 10. The sequence number of the target image is determined according to the number of image frames stored in the pre-encoding queue. In some embodiments, the encoding device adds the target image frame to the pre-encoding queue after acquiring a certain target image frame to be encoded; By judging that the length of the pre-encoding queue is less than the rated length, the encoding device determines the frame type and encoding parameters of the target image frame through the predefined frame type arrangement and the sequence number of the target image frame. For example, the encoding device obtains the predefined frame type The arrangement is IBPBPIBPP, after the encoding device acquires a target image frame to be analyzed, the number of image frames stored in the pre-encoding queue is 4, and the sequence number of the target image frame is 4, so the frame type of the target image frame is B frame.

The frame type of the encoded image frame refers to the frame type of the image frame whose encoding process is completed by the encoding device. In some embodiments, the encoding device counts the frame types of encoded image frames through variables, for example, a first variable, a second variable and a third variable are set in the encoding device, wherein the first variable is used to record the frame type Be the number of encoded image frames of I frame, the second variable is used to record the number of encoded image frames of P frame as the frame type, and the third variable is used to record the number of encoded image frames of B frame as the frame type; encoding After the device encodes the target image frame, it updates the value of the variable corresponding to the frame type. For example, the current value of the second variable is 3, and the value of the second variable is updated to 4 after the encoding device encodes a target image frame whose frame type is P frame.

In some embodiments, the predefined configuration information is determined by the coding device, and the coding device determines according to requirements such as coding quality requirements and video playback quality. The method of determining the predefined frame type configuration information depends on the actual situation. To limit. In some embodiments, the predefined frame type configuration information includes 5 I frames, and the encoded image frame includes 1 I frame, and the encoding device acquires a target image frame to be encoded and stores the encoding device In the pre-coding queue, when the number of image frames in the pre-coding queue is smaller than the size of the pre-coding queue, the coding device determines that the frame type of the target image frame is an I frame based on preconfigured frame type configuration information. In some embodiments, when the encoding device stores an acquired target image frame in the pre-encoding queue, the encoding device according to the pre-configured frame type configuration information, the frame type of the encoded image frame and the pre-encoding queue Length, to determine the frame type of the target image frame, for example, the pre-configured frame type configuration information includes 3 I frames, the encoded image frame includes 2 I frames, the length of the pre-coding queue is 8, and the pre-coding queue The size is 10, and the encoding device determines that the frame type of the target image frame is an I frame according to the above information.

In some embodiments, the predefined frame type configuration information includes the quantity of at least one frame type and the ratio of two different frame type data, for example, a certain predefined frame type configuration information includes 3 I frames and I frame Quantity/P frame quantity is less than or equal to 4. In some embodiments, the encoding device configures the predefined frame type configuration information to include 3 I frames, and the number of I frames/the number of P frames is less than or equal to 4 in consideration of the compression rate of video encoding, etc. The coded image frame includes 3 I frames and 6 P frames. The coding device acquires a target image frame to be coded. When the number of reference image frames of the target image frame does not meet the analysis conditions, based on the above The information identifies the frame type of the target image frame as a P frame.

Step 330, when the number of reference image frames of the target image frame does not satisfy the analysis condition, determine a predefined frame type as the frame type of the target image frame.

Wherein, step 330 includes the following several sub-steps, step 332, obtains the predefined coding parameter configuration information, and the coding parameter configuration information includes at least one set of correspondence between frame types and coding parameters; step 334, from the coding parameter configuration information In, the encoding parameters corresponding to the frame type of the target image frame are determined as the encoding parameters of the target image frame.

Encoding parameters include bit rate, QP, etc.

In some embodiments, the encoding device acquires a certain target image frame to be encoded, and stores the target image frame in a pre-encoding queue. When the length of the pre-encoding queue is less than the rated length, the encoding device passes a predefined The frame type arrangement IPPIPP and the sequence number 5 of the target image frame determine the frame type of the target image frame, and the encoding parameters of the target image frame are determined through predefined encoding parameters.

In some embodiments, the encoding device acquires a target image frame to be encoded, and stores the target image frame in a pre-encoding queue. The encoding device judges whether the reference image frame of the target image frame satisfies the encoding condition, that is, determines whether the number of image frames stored in the pre-encoding queue reaches the size of the pre-encoding queue. When the number of reference image frames of the target image frame does not meet the encoding conditions, that is, when the number of image frames in the pre-encoding queue is smaller than the size of the pre-encoding queue, the encoding device determines the predefined encoding parameters as the target image Encoding parameters for the frame.

In some embodiments, the encoding device acquires the encoding parameters of the target image frame through a rate control model. For details about how the encoding device determines the encoding parameters of the target image frame through the rate control model, please refer to the following embodiment of encoding the target image frame in the second manner.

Step 340, in the case where the frame type of the target image frame is a key frame, use the intra prediction mode to encode the target image frame according to the encoding parameters of the target image frame; or, when the frame type of the target image frame is a non-key frame In this case, the target image frame is encoded according to the encoding parameters of the target image frame by using an inter-frame prediction mode.

In some embodiments, the encoding device encodes the target image frame according to the frame type and encoding parameters of the target image frame. For the specific encoding process, please refer to the detailed introduction in the background technology, and details are not repeated here. Different encoding methods may be used to encode the target image frame according to the signal type of the target image frame and the encoding characteristics of the encoding device, and the specific encoding method is not limited in this application.

Please refer to FIG. 4 , which shows a schematic diagram of a video encoding method provided by another embodiment of the present application. In some embodiments, the encoding device obtains a target image frame to be encoded, and determines that the number of reference image frames of the target image frame does not meet the analysis conditions by judging. As shown in FIG. 4, the image frame to be encoded acquired by the encoding device Be the 3rd frame image frame, the quantity of the reference image frame of the 3rd frame image frame is 2 (the reference image frame of the 3rd frame image frame includes the 1st frame and the 2nd frame), the analysis condition is 30, the encoding device passes the first The method determines that the frame type of the third image frame is an I frame (key frame), and obtains the encoding parameters corresponding to the third image frame through the code rate control model. The encoding device performs intra-frame predictive encoding on the third image frame to obtain the residual value, and after encoding processes such as quantization and entropy encoding, finally obtains the code stream corresponding to the third image frame, and the encoding device outputs the corresponding code stream of the target image frame. Code stream, and prepare to acquire a new target image frame.

In some embodiments, after the encoding device acquires a target image frame to be encoded, the target image frame is added to the pre-encoding queue; when the pre-encoding queue does not meet the rated length, the encoding device uses the first method to determine The frame type of the target image frame is a P frame, and the encoding parameters corresponding to the target image frame are determined through a code rate prediction model; the encoding device adds the frame type and encoding parameters of the target image frame to the pre-encoding queue; The target image frame is subjected to inter-frame prediction to obtain a residual value, and after encoding processing such as quantization and entropy encoding, the code stream corresponding to the key frame is finally obtained, and the encoding device outputs the code stream corresponding to the target image frame. Optionally, In the case that the length of the pre-encoding queue is less than the rated length, the encoded target image frame is stored in the encoding queue.

When the number of reference image frames of the target image frame does not meet the analysis conditions, the encoding device does not use the image frames of the target image to predict the frame type and encoding parameters of the target image frame, and directly determines the target through the predefined frame type arrangement The frame type of the image frame can save the time for the encoding device to determine the frame type of the target image frame, and avoid the situation that the encoding device obtains a sufficient number of reference image frames when the number of reference image frames of the target image frame does not meet the analysis conditions. The waste of encoding time caused by predicting the frame type and encoding parameters of the target image frame.

The video encoding method under the condition that the number of reference image frames of the target image frame satisfies the analysis condition is introduced below through several embodiments.

In some embodiments, after obtaining the target image frame to be coded, it further includes: when the number of reference image frames satisfies the analysis condition, determining the frame type and encoding parameters of the target image frame in a second manner; wherein, the second The method refers to the method of determining the frame type and encoding parameters according to the reference image frame.

The second method means that before formally encoding the target image frame, the encoding device needs to predict the frame type and encoding parameters of the target image frame according to the target image frame and the reference image frame, so as to obtain a better compression rate and reduce the video encoding loss. The degree of distortion ensures that the code stream obtained after video encoding can produce a video picture with a stable bit rate after decoding.

In some embodiments, after the encoding device acquires the target image frame, it adds the target image frame to the pre-coding queue, and when the length of the pre-coding queue satisfies the rated length, the encoding device determines the frame type of the target image frame according to the second method and encoding parameters. In some embodiments, in addition to the target image frame added by the encoding device in the pre-encoding queue, the other multiple image frames are already encoded image frames. Optionally, the pre-encoding queue includes the frame types of the other multiple image frames and encoding parameters. In some embodiments, all image frames in the precoding queue except the target image frame are reference image frames of the target image frame.

In some embodiments, determining the frame type and encoding parameters of the target image frame according to the second method includes: determining the frame type of the target image frame according to the reference image frame of the target image frame; The encoding parameters of the type target image frame, the configuration parameters are used to constrain the bit rate of the target image frame.

In some embodiments, determining the frame type of the target image frame according to the reference image frame includes: generating a plurality of frame type arrangements, each frame type arrangement including an assumed frame type of the target image frame and an assumed frame type following the assumed frame type at least one frame type; calculate the distortion rates corresponding to multiple frame type arrangements; wherein, the distortion rate corresponding to the frame type arrangement is determined by encoding the target image frame and at least one reference image frame according to the frame type arrangement; from a variety of In the frame type arrangement, determine the target frame type arrangement with the smallest distortion rate; determine the assumed frame type in the target frame type arrangement as the frame type of the target image frame.

In some embodiments, an assumed frame type of the target image frame includes I-frame, P-frame and B-frame. At least one frame type following the assumed frame type refers to the assumed frame type of the reference image frame determined according to the assumed frame type of the target image frame. In the process of determining the frame type of a target image frame to be encoded according to the reference image frame, the encoding device first assumes a frame type of the target image frame, and determines the frame type of the reference image frame one by one according to the assumed frame type of the target image frame. Assume frame type, and finally obtain multiple frame type arrangements, because the assumed frame type of the target image frame will affect the assumed frame determined by the subsequent reference image frame, so the target image frame sets different assumed frame types, resulting in multiple different frames Type arrangement, for example, a target image frame to be coded has 5 reference image frames, the assumed frame type of the target image frame is determined as I frame, according to the picture content of the target image frame and the assumed frame of the target image frame Type, the assumed frame type of the first reference image frame determined is B frame, the second reference image frame is based on the picture content of the target image frame, the assumed frame type of the target image frame, the picture content of the first reference image frame, the first reference The assumed frame type of the image frame, the determined assumed frame type is a B frame, repeat the above steps until the assumed frame types of all reference image frames of the target image frame are determined, and a frame type finally obtained is arranged as IBBPBB; the target image The frame type of the frame is determined as a B frame, and the above steps are repeated to determine the assumed frame types of all reference image frames, and finally a frame type obtained is arranged as BIPBPP.

The encoding device calculates distortion rates corresponding to multiple frame type arrangements, wherein the distortion rates corresponding to the frame type arrangements are determined by the encoding device through rough encoding of the target image frame and the reference image frame based on the frame type arrangement. The smaller the distortion rate corresponding to the frame type arrangement is, the better the encoding quality of this frame type arrangement is. The encoding device selects the frame type arrangement with the smallest distortion rate as the target frame type sequence, and uses the assumed frame type of the target image frame in the target frame type arrangement as the frame type of the target image frame. In some embodiments, the encoding parameters of the target image frame are determined through a rate control model.

Since the conversion frequency between adjacent video image frames is low in video files or real-time live scenes, predicting the frame type of the target image frame through the forward reference image frame of the target image frame can obtain more reliable frame type prediction results.

In some embodiments, the encoding parameters of the target image frame at least include: a code rate and a quantization parameter of the target image frame. According to the configuration parameters and/or the frame type of the target image frame, determining the encoding parameters of the target image frame includes: determining the code rate of the target image frame according to the configuration parameters, and determining the quantization parameter of the target image frame according to the frame type of the target image frame . Or, determine the bit rate and quantization parameters of the target image frame according to the configuration parameters.

The encoding device can set different configuration parameters according to different encoding requirements. In some embodiments, in order to keep the encoded video bit rate stable, the encoding device uses an ABR (Average Bitrate, average bit rate) control model as a configuration parameter. The encoding device can also use rate control models such as CRF (Constant Rate Factor, constant rate factor), CQP (Constant Quantizer Parameter, constant quantization parameter) and R-QP (Rate-Quantization Parameter, rate-quantization) as configuration parameters , the configuration parameters are determined according to conditions such as network communication quality and coding requirements, which are not limited in this application.

In some embodiments, the encoding device determines the configuration parameters based on the R-QP code rate control model, and the configuration parameters include the calculation formula of the code rate R _i of the initial QP and the quadratic model:

Among them, R _i represents the code rate of the target image frame, a ₀ , a ₁ and a _i are used to represent information related to the content of the target image frame, Q _i refers to the distortion obtained after encoding the target image frame according to the initial QP Rate.

After obtaining the code rate of the target frame type, the encoding device determines the QP of the target image frame according to the frame type of the target image frame in the RDO (Rate-Distortion Optimization) process.

In some embodiments, the encoding device determines configuration parameters based on CQP, and according to the CQP method, the encoding device configures a fixed QP in the configuration parameters. Optionally, the encoding device configures different QPs for different frame types, for example, the encoding device configures the QP of the target image frame whose frame type is P frame as the first fixed QP. The encoding device calculates the code rate of the target image frame according to the fixed QP in the configuration parameters.

It should be noted that all methods for determining the bit rate and quantization parameters of the target image frame during the pre-encoding process of the video file can be used in this application.

Please refer to FIG. 5 , which shows a schematic diagram of a video encoding method provided by another embodiment of the method.

After the encoding device acquires the target image frame, it adds the target image frame to the pre-encoding queue, the length of the pre-encoding queue is n, and the rated length is n, where n is a positive integer. During the 0-T1 time period, the number of frames in the pre-coding queue is less than n, and for an acquired target image frame to be encoded, the encoding device uses the first method to determine the frame type and encoding parameters of the target image frame; In one example, the first frame is input into the pre-encoding queue, the number of frames in the pre-encoding queue is 1, less than n, and the analysis condition is not met, and the encoding device determines that the frame type of the target image frame is an I frame through the first method, based on The code rate control model determines the encoding parameters corresponding to the first frame, and encodes the first frame, obtains the code stream corresponding to the first frame after encoding, and outputs the code stream. After time T1, the encoding device acquires the nth frame of the target image frame. At this time, the length of the pre-encoding queue is n, which meets the analysis conditions; the encoding device uses the second method to determine the type of the nth frame of the target image frame and the encoding parameters. In an example, the encoding device adds the n+1th frame to the pre-encoding queue, and the pre-encoding queue includes any frame from the second frame to the n+1th frame. The encoding device generates m frame type arrangements according to the second frame to the n+1th frame, m is a positive integer, calculates the encoding distortion rates corresponding to the m frame type arrangements, and determines the frame type arrangement with the smallest distortion rate as the target frame type arrangement, and determine the frame type of the target image frame of the n+1th frame according to the arrangement of the target image frame type, and the encoding device determines the encoding parameters of the n+1th frame through a code rate control model. The encoding device encodes the n+1th frame according to the predicted frame type and encoding parameters to obtain a code stream corresponding to the n+1th frame, and outputs the code stream. The encoding device removes the second frame from the pre-encoding queue, and waits for the n+2 frame to be encoded to enter the pre-encoding queue.

Please refer to FIG. 6 , which shows a schematic diagram of a video encoding process provided by another embodiment of the present application.

In the application scenario of real-time live broadcast, the encoding device obtains a target image frame to be encoded through the screen acquisition device; the encoding device adds the target image frame to the pre-encoding queue; the encoding device judges whether the length of the pre-encoding queue meets the analysis requirements , in some embodiments, the analysis requirement is that the number of image frames stored in the pre-encoding queue is the same as the size of the encoding queue; when the length of the pre-encoding queue does not meet the analysis requirements, the encoding device adopts the The frame type and encoding parameters of the target image frame, the encoding device adds the frame type and encoding parameters of the target image to the pre-encoding queue, and encodes the target image frame according to the frame type and encoding parameters to obtain the corresponding code stream, And output the code stream; in the case that the length of the pre-encoding queue meets the analysis requirements, the encoding device uses the second method to predict the frame type and encoding parameters of the target image frame, so that the frame type of the target image frame obtained through prediction can be obtained during the encoding process. produces the least amount of distortion. The encoding device adds the frame type and encoding parameters of the target image to the pre-encoding queue, and encodes the target image frame according to the frame type and encoding parameters, obtains the corresponding code stream, and outputs the code stream; the encoding device passes In the second method, after obtaining the frame type and encoding parameters of the target image frame, the image frame added to the pre-encoding queue at the earliest is removed from the pre-encoding queue to ensure that the encoding device can add the target image frame obtained from the picture acquisition device next time to the in the precoding queue. After the encoding device finishes encoding a target image frame, it needs to judge whether to continue to repeat the above encoding process. If the encoding device receives an exit command, such as an exit command triggered by the user exiting the video playback interface on the terminal device, or the end of the screen capture device If the exit command is triggered by the work, the coding device stops the above coding process.

Through the above method, when the number of reference image frames of the target image frame does not meet the analysis conditions, the frame type and encoding parameters of the target image frame are directly determined using the first method, which reduces the waiting time of the encoding device and reduces the encoding delay. When the number of reference image frames of the target image frame satisfies the analysis conditions, using the second method to predict the frame type of the target image through the reference image frames helps to maintain the stability of the encoding process bit rate and avoid inter-frame quality fluctuations. Since the content switching rate of the image frames in the video file or the image frames of the live scene is low, that is, the scene switching is less, and there is a strong correlation between the target image frame and the adjacent reference image frame, therefore, In this method, the second method is adopted, that is, using the forward reference image frame to predict the frame type of the target image frame can achieve the similar effect of using the backward reference image frame to predict the target image frame type, and reduces the coding cost. delay.

Although, when using the first method to directly determine the frame type and coding parameters of the target image frame, it will cause fluctuations in the quality between frames, but in the process of encoding the video stream, the first method is used to determine the frame type and coding parameters. The number of image frames is much smaller than the number of image frames that determine the frame type and encoding parameters according to the second method, so this method is used to establish a reasonable code rate control model, and the frame will be reduced in a short period of time during the initial period of video playback. Inter-frame quality fluctuations, and the stability of the inter-frame quality of video images can be maintained afterwards, and the encoding delay is low, which is highly usable in live broadcast and RTC scenarios.

The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

Please refer to FIG. 7 , which shows a block diagram of a video coding apparatus provided by an embodiment of the present application. The device has the function of realizing the above video coding method, and the function can be realized by hardware, or by hardware executing corresponding software. The device may be the electronic device described above, or may be set in the electronic device. The apparatus 700 may include: an image frame acquisition module 710 , a first encoding determination module 720 and an image frame encoding module 730 .

The image frame acquisition module 710 is configured to acquire the target image frame to be encoded.

The first encoding determination module 720 is configured to determine the frame type and encoding parameters of the target image frame in a first manner when the number of reference image frames of the target image frame does not meet the analysis condition; wherein, the The first manner refers to a manner in which frame types and encoding parameters are not determined according to the reference image frame.

The image frame encoding module 730 is configured to encode the target image frame according to the frame type and encoding parameters of the target image frame.

In some embodiments, as shown in FIG. 8, the first encoding determining module 720 includes: a first frame type determining unit 722 configured to determine a predefined frame type as the frame type of the target image frame; the first The parameter determining unit 724 is configured to determine a predefined encoding parameter as an encoding parameter of the target image frame.

In some embodiments, the first frame type determination unit 722 is configured to: acquire a predefined frame type arrangement, the predefined frame type arrangement refers to the frame types corresponding to the predetermined plurality of image frames respectively Arrangement mode: determine the frame type of the target image frame from the predefined frame type arrangement based on the sequence number of the target image frame; or obtain predefined frame type configuration information, the predefined frame The type configuration information includes configuration quantities of at least one frame type; and the frame type of the target image frame is determined based on the frame type of the encoded image frame and the predefined frame type configuration information.

In some embodiments, the first parameter determination unit 724 is configured to acquire predefined encoding parameter configuration information, where the encoding parameter configuration information includes at least one set of correspondence between frame types and encoding parameters; From the encoding parameter configuration information, determine an encoding parameter corresponding to the frame type of the target image frame as an encoding parameter of the target image frame.

The image frame encoding module 730 is configured to encode the target image frame by using an intra prediction mode according to encoding parameters of the target image frame when the frame type of the target image frame is a key frame; or , if the frame type of the target image frame is a non-key frame, encode the target image frame in an inter-frame prediction mode according to the encoding parameters of the target image frame.

In some embodiments, as shown in FIG. 8 , the apparatus 700 further includes a second encoding determination module 740 configured to determine the encoding in a second manner when the number of the reference image frames satisfies the analysis condition. The frame type and coding parameters of the target image frame; wherein, the second method refers to a method of determining the frame type and coding parameters according to the reference image frame.

In some embodiments, the second encoding determination module 740 includes: a second frame type determination unit 742 configured to determine the frame type of the target image frame according to a reference image frame of the target image frame; a second parameter determination unit 744 configured to The encoding parameters of the target image frame according to the configuration parameters and/or the frame type of the target image frame, the configuration parameters are used to constrain the code rate of the target image frame.

In some embodiments, the second frame type determination unit 742 is configured to generate a plurality of frame type arrangements, each frame type arrangement includes a hypothetical frame type of the target image frame and a frame located in the hypothetical frame At least one frame type after the type; calculate the distortion rates corresponding to the multiple frame type arrangements; wherein, the distortion rate for the frame type arrangement is based on the frame type arrangement for the target image frame and at least one reference The image frame is determined by encoding; from the various frame type arrangements, determine the target frame type arrangement with the smallest distortion rate; determine the assumed frame type in the target frame type arrangement as the frame type of the target image frame .

In some embodiments, as shown in FIG. 8 , the apparatus 700 further includes an image frame adding module 750 configured to add the target image frame to a pre-coding queue; when the length of the pre-coding queue is less than the rated length In the case of , it is determined that the number of reference image frames of the target image frame does not meet the analysis condition; when the length of the precoding queue is equal to the rated length, determine the number of reference image frames of the target image frame meet the analysis conditions.

It should be noted that, when realizing the functions of the device provided by the above-mentioned embodiments, the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional modules according to the needs. The content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and the method embodiment provided by the above embodiment belong to the same idea, and the specific implementation process thereof is detailed in the method embodiment, and will not be repeated here.

In an example embodiment, there is also provided a computer device comprising a processor and a memory in which a computer program is stored. The computer program is configured to be executed by one or more processors to implement the above video encoding method.

A computer device may be referred to as a video encoding device, and is used for encoding a video file, so as to reduce resources consumed in transmitting the video file.

In an exemplary embodiment, there is also provided a computer-readable storage medium, in which a computer program is stored, and the computer program implements the above video encoding method when executed by a processor of a computer device.

Optionally, the above-mentioned computer-readable storage medium may be storage devices such as ROM (Read-Only Memory, read-only memory) and RAM (Random Access Memory, random access memory).

In an exemplary embodiment, a computer program product is also provided, which, when the computer program product is run on a computer device, causes the computer device to execute the above video encoding method.

It should be understood that the "plurality" mentioned herein refers to two or more than two. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. The character "/" generally indicates that the contextual objects are an "or" relationship. In addition, the numbering of the steps described herein only exemplarily shows a possible sequence of execution among the steps. In some other embodiments, the above-mentioned steps may not be executed according to the order of the numbers, such as two different numbers The steps are executed at the same time, or two steps with different numbers are executed in the reverse order as shown in the illustration, which is not limited in this embodiment of the present application.

Claims (13)

1. A video encoding method, the method is executed by a computer device, the method comprising:

   Obtain the target image frame to be encoded;

   In the case that the number of reference image frames of the target image frame does not satisfy the analysis condition, determine the frame type and encoding parameters of the target image frame according to the first method; wherein, the first method refers to not according to the The method of determining the frame type and encoding parameters with reference to the image frame;

   Encoding the target image frame according to the frame type and encoding parameters of the target image frame.
2. The method according to claim 1, wherein said determining the frame type and encoding parameters of the target image frame according to the first method comprises:

   Determining the predefined frame type as the frame type of the target image frame;

   A predefined encoding parameter is determined as an encoding parameter of the target image frame.
3. The method according to claim 2, wherein said determining the predefined frame type as the frame type of the target image frame comprises:

   Acquiring a predefined frame type arrangement, the predefined frame type arrangement refers to the arrangement of frame types corresponding to a plurality of predetermined image frames; In the type arrangement, determine the frame type of the target image frame;

   or,

   Acquire predefined frame type configuration information, the predefined frame type configuration information including the configuration quantity of at least one frame type; based on the frame type of the encoded image frame and the predefined frame type configuration information, determine the Describes the frame type of the target image frame.
4. The method according to claim 2, wherein said determining the predefined encoding parameters as the encoding parameters of the target image frame comprises:

   Obtain predefined encoding parameter configuration information, where the encoding parameter configuration information includes at least one set of correspondence between frame types and encoding parameters;

   From the encoding parameter configuration information, determine an encoding parameter corresponding to the frame type of the target image frame as an encoding parameter of the target image frame.
5. The method according to claim 1, wherein said encoding the target image frame according to the frame type and encoding parameters of the target image frame comprises:

   In the case where the frame type of the target image frame is a key frame, encoding the target image frame by using an intra-frame prediction mode according to the encoding parameters of the target image frame;

   or,

   When the frame type of the target image frame is a non-key frame, the target image frame is encoded according to the encoding parameters of the target image frame by using an inter-frame prediction mode.
6. The method according to claim 1, wherein, after acquiring the target image frame to be encoded, further comprising:

   When the number of the reference image frames satisfies the analysis condition, determine the frame type and coding parameters of the target image frame according to the second method; wherein, the second method refers to determining according to the reference image frame Frame type and way of encoding parameters.
7. The method according to claim 6, wherein said determining frame type and encoding parameters of said target image frame in a second manner comprises:

   determining the frame type of the target image frame according to the reference image frame;

   Determine the encoding parameters of the target image frame according to the configuration parameters and/or the frame type of the target image frame, where the configuration parameters are used to constrain the code rate of the target image frame.
8. The method according to claim 7, wherein said determining the frame type of the target image frame according to the reference image frame comprises;

   generating a plurality of frame type arrangements, each frame type arrangement including a hypothetical frame type of the target image frame and at least one frame type following the hypothetical frame type;

   Calculating the respective distortion rates corresponding to the various frame type arrangements; wherein, the distortion rates for the frame type arrangements are determined by encoding the target image frame and at least one reference image frame according to the frame type arrangement;

   From the various frame type arrangements, determine the target frame type arrangement with the smallest distortion rate;

   The assumed frame type in the target frame type arrangement is determined as the frame type of the target image frame.
9. The method according to any one of claims 1 to 8, wherein, after acquiring the target image frame to be encoded, further comprising:

   adding the target image frame to the precoding queue;

   When the length of the precoding queue is less than a rated length, it is determined that the number of reference image frames of the target image frame does not meet the analysis condition;

   When the length of the precoding queue is equal to the rated length, it is determined that the number of reference image frames of the target image frame meets the analysis condition.
10. A video encoding device, the device comprising:

    An image frame acquisition module configured to acquire a target image frame to be encoded;

    The first encoding determination module is configured to determine the frame type and encoding parameters of the target image frame in a first manner when the number of reference image frames of the target image frame does not meet the analysis condition; wherein, the first One method refers to a method of determining the frame type and encoding parameters not according to the reference image frame;

    An image frame encoding module configured to encode the target image frame according to the frame type and encoding parameters of the target image frame.
11. A computer device, the computer device includes a processor and a memory, and a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method described in any one of claims 1 to 9 method.
12. A computer-readable storage medium, in which a computer program is stored, and the computer program is loaded and executed by a processor to implement the method according to any one of claims 1 to 9.
13. A computer program product, the computer program product runs on a computer device, causing the computer device to execute the method according to any one of claims 1 to 9.

Images