CN113660491B

CN113660491B - Encoding method, encoding device, storage medium, and electronic apparatus

Info

Publication number: CN113660491B
Application number: CN202110914491.0A
Authority: CN
Inventors: 何鸣; 阮良; 陈功; 许迅; 代苑莹; 程玲; 韩庆瑞; 陈丽
Original assignee: Hangzhou Netease Zhiqi Technology Co Ltd
Current assignee: Hangzhou Netease Zhiqi Technology Co Ltd
Priority date: 2021-08-10
Filing date: 2021-08-10
Publication date: 2024-05-07
Anticipated expiration: 2041-08-10
Also published as: CN113660491A

Abstract

The embodiment of the disclosure relates to an encoding method, an encoding device, a storage medium and electronic equipment, and relates to the technical field of image processing. The encoding method comprises the following steps: updating the number of target frames in the current coding period according to the type of the current frame; the type of the current frame comprises a target frame and a non-target frame; responding to the fact that the number of target frames in the current coding period is larger than a first preset frame number and the number of coded frames in the current coding period is larger than a second preset frame number, coding the current frame according to a first code rate, and ending the current coding period; responsive to the number of target frames in the current coding period being greater than a first preset frame number and the number of coded frames in the current coding period not being greater than a second preset frame number, performing coding processing on the current frame according to a second code rate; the first code rate is higher than a reference code rate determined according to the current network performance, and the second code rate is lower than the reference code rate. The method and the device can improve the quality of the coded video while not affecting the video volume.

Description

Encoding method, encoding device, storage medium, and electronic apparatus

Technical Field

Embodiments of the present disclosure relate to the field of image processing technology, and more particularly, to an encoding method, an encoding apparatus, a computer-readable storage medium, and an electronic device.

Background

This section is intended to provide a background or context for the embodiments of the disclosure recited in the claims, which description herein is not admitted to be prior art by inclusion in this section.

The video code rate is the number of data bits transmitted in unit time during data transmission, the sampling rate is understood in a popular way, the larger the sampling rate in unit time is, the higher the precision is, and the processed file is more similar to the original file. Currently, video is generally encoded using CBR (Constant Bit Rate ) mode, where the bit rate remains substantially constant.

Disclosure of Invention

However, when the video picture is complex, the current coding method can cause a large loss of picture quality after coding, blurred pictures and poor video quality.

Therefore, a coding method is highly needed, which can improve the video image quality and the video quality after coding.

In this context, embodiments of the present disclosure desirably provide an encoding method, an encoding apparatus, a computer-readable storage medium, and an electronic device.

According to a first aspect of embodiments of the present disclosure, there is provided an encoding method, including: updating the number of target frames in the current coding period according to the type of the current frame; the type of the current frame comprises a target frame and a non-target frame; responding to the fact that the number of target frames in the current coding period is larger than a first preset frame number and the number of coded frames in the current coding period is larger than a second preset frame number, coding the current frame according to a first code rate, and ending the current coding period; responsive to the number of target frames in the current coding period being greater than the first preset frame number and the number of coded frames in the current coding period not being greater than the second preset frame number, performing coding processing on the current frame according to a second code rate; wherein the first code rate is higher than a reference code rate determined according to current network performance, and the second code rate is lower than the reference code rate determined according to current network performance.

In an alternative embodiment, the method further comprises: responding to the number of target frames in the current coding period not larger than the first preset frame number and the number of coded frames in the current coding period larger than a third preset frame number, coding the current frame according to the first code rate, and ending the current coding period; wherein the third preset frame number is greater than the second preset frame number.

In an alternative embodiment, the method further comprises: and in response to the number of target frames in the current coding period not being greater than the first preset frame number and the number of coded frames in the current coding period not being greater than the third preset frame number, coding the current frame according to the second code rate.

In an alternative embodiment, the second code rate is determined by: and carrying out reduction processing on the reference code rate according to a preset reduction ratio to obtain the second code rate.

In an alternative embodiment, the first code rate is determined by: acquiring the first N coded frames of the current frame in the current coding period; acquiring a code rate difference value between the actual code rate of each coded frame and the reference code rate; determining the first code rate according to the N code rate difference values and the reference code rate; wherein N is a positive integer greater than the second preset number of frames and less than the third preset number of frames.

In an optional embodiment, the determining the first code rate according to the N code rate differences and the reference code rate includes: acquiring N first accumulated values of the code rate difference values; obtaining the product of the first accumulated value and the reference code rate; and acquiring a second accumulated value of the product and the reference code rate, and determining the second accumulated value as the first code rate.

In an alternative embodiment, the type of the current frame is determined by: performing significance detection on the current frame to obtain the area of a significance region; determining that the current frame is the target frame in response to the area of the salient region being greater than a preset area threshold; and determining that the current frame is the non-target frame in response to the area of the salient region being not greater than the preset area threshold.

In an alternative embodiment, the type of the current frame is determined by: performing motion detection on the current frame to obtain whether the current frame contains a moving target or not; determining the current frame as the target frame in response to the current frame containing the moving target; and determining the current frame as the non-target frame in response to the current frame not including the moving target.

In an alternative embodiment, the type of the current frame is determined by: performing background detection on the current frame to obtain a background area and a foreground area in the current frame; detecting whether relative motion exists between the foreground region and the background region; determining that the current frame is the target frame in response to relative motion of the foreground region and the background region; and determining that the current frame is the non-target frame in response to the foreground region and the background region not having relative motion.

According to a second aspect of embodiments of the present disclosure, there is provided an encoding apparatus including: the frame type detection module is used for updating the number of target frames in the current coding period according to the type of the current frame; the type of the current frame comprises a target frame and a non-target frame; the first coding module is used for responding to the fact that the number of target frames in the current coding period is larger than a first preset frame number and the number of coded frames in the current coding period is larger than a second preset frame number, coding the current frame according to a first code rate, and ending the current coding period; the second coding module is used for responding to the fact that the number of target frames in the current coding period is larger than the first preset frame number and the number of coded frames in the current coding period is not larger than the second preset frame number, and coding the current frames according to a second code rate; wherein the first code rate is higher than a reference code rate determined according to current network performance, and the second code rate is lower than the reference code rate determined according to current network performance.

In an alternative embodiment, the first encoding module is configured to: responding to the number of target frames in the current coding period not larger than the first preset frame number and the number of coded frames in the current coding period larger than a third preset frame number, coding the current frame according to the first code rate, and ending the current coding period; wherein the third preset frame number is greater than the second preset frame number.

In an alternative embodiment, the second encoding module is configured to: and in response to the number of target frames in the current coding period not being greater than the first preset frame number and the number of coded frames in the current coding period not being greater than the third preset frame number, coding the current frame according to the second code rate.

In an alternative embodiment, the second encoding module is configured to: and carrying out reduction processing on the reference code rate according to a preset reduction ratio to obtain the second code rate.

In an alternative embodiment, the first encoding module is configured to: acquiring the first N coded frames of the current frame in the current coding period; acquiring a code rate difference value between the actual code rate of each coded frame and the reference code rate; determining the first code rate according to the N code rate difference values and the reference code rate; wherein N is a positive integer greater than the second preset number of frames and less than the third preset number of frames.

In an alternative embodiment, the first encoding module is configured to: acquiring N first accumulated values of the code rate difference values; obtaining the product of the first accumulated value and the reference code rate; and acquiring a second accumulated value of the product and the reference code rate, and determining the second accumulated value as the first code rate.

In an alternative embodiment, the frame type detection module is configured to: performing significance detection on the current frame to obtain the area of a significance region; determining that the current frame is the target frame in response to the area of the salient region being greater than a preset area threshold; and determining that the current frame is the non-target frame in response to the area of the salient region being not greater than the preset area threshold.

In an alternative embodiment, the frame type detection module is configured to: performing motion detection on the current frame to obtain whether the current frame contains a moving target or not; determining the current frame as the target frame in response to the current frame containing the moving target; and determining the current frame as the non-target frame in response to the current frame not including the moving target.

In an alternative embodiment, the frame type detection module is configured to: performing background detection on the current frame to obtain a background area and a foreground area in the current frame; detecting whether relative motion exists between the foreground region and the background region; determining that the current frame is the target frame in response to relative motion of the foreground region and the background region; and determining that the current frame is the non-target frame in response to the foreground region and the background region not having relative motion.

According to a third aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the encoding method of the first aspect described above.

According to a fourth aspect of embodiments of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the encoding method of the first aspect described above via execution of the executable instructions.

According to the encoding method, the encoding device, the computer-readable storage medium and the electronic apparatus of the embodiments of the present disclosure, by determining the complexity of the video picture in each encoding period according to the number of target frames in each encoding period, on the premise that the video picture is sufficiently complex (the number of target frames is greater than the first preset frame number), the encoded frames are required to be satisfied to be greater than the second preset frame number at the same time, the encoded frames are encoded at the first code rate with a higher value, and if the number of encoded frames is not greater than the second preset frame number, the encoded frames are encoded at the second code rate with a smaller value. Therefore, on one hand, the method and the device can solve the technical problems of large loss of the coded picture and poor video quality when the video picture is complex due to coding according to the constant code rate in CBR coding, and improve the video quality after coding. On the other hand, the method and the device can solve the problems of video picture blocking and larger network delay caused by encoding at a high code Rate once a video picture is detected to be more complex in ABR (Average Bit Rate) encoding, and improve the video quality after encoding on the premise of not influencing the video volume.

Drawings

The above, as well as additional purposes, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description when read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which:

FIG. 1 shows a schematic diagram of CBR encoding;

FIG. 2 illustrates a flow chart of an encoding method according to an embodiment of the present disclosure;

FIG. 3 illustrates a flow chart for determining a type of a current frame according to an embodiment of the present disclosure;

FIG. 4 illustrates another flow chart for determining a type of a current frame according to an embodiment of the present disclosure;

FIG. 5 illustrates a flow chart of yet another determination of the type of current frame in accordance with an embodiment of the present disclosure;

FIG. 6 shows a sub-flowchart of an encoding method according to an embodiment of the present disclosure;

Fig. 7 illustrates a flow chart of determining a first code rate according to an embodiment of the present disclosure;

FIG. 8 illustrates an overall flow chart of an encoding method according to an embodiment of the present disclosure;

FIG. 9 illustrates an overall flow chart of another encoding method according to an embodiment of the present disclosure;

FIG. 10 shows a schematic diagram of an encoding method according to an embodiment of the present disclosure;

FIG. 11 shows a schematic diagram of an encoding apparatus according to an embodiment of the present disclosure;

fig. 12 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

Detailed Description

The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are presented merely to enable one skilled in the art to better understand and practice the present disclosure and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Those skilled in the art will appreciate that embodiments of the present disclosure may be implemented as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the following forms, namely: complete hardware, complete software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.

According to an embodiment of the present disclosure, there are provided an encoding method, an encoding apparatus, a computer-readable storage medium, and an electronic device.

Any number of elements in the figures are for illustration and not limitation, and any naming is used for distinction only, and not for any limiting sense.

The principles and spirit of the present disclosure are described in detail below with reference to several representative embodiments thereof.

Summary of The Invention

The inventor discovers that the current coding method only carries out coding according to the set constant code rate, so that when the video picture is complex, the loss of the coded image quality is relatively large, and the video quality is poor.

In view of the above, the basic idea of the present disclosure is that: a coding method, a coding device, a computer readable storage medium and an electronic apparatus are provided, by determining the complexity of a video picture in each coding period according to the number of target frames in each coding period, on the premise that the video picture is sufficiently complex (the number of target frames is greater than a first preset frame number), the first code rate with a higher value is required to be coded while the number of coded frames is greater than a second preset frame number, and if the number of coded frames is not greater than the second preset frame number, the second code rate with a smaller value is required to be coded. Therefore, on one hand, the method and the device can solve the technical problems of large loss of the coded picture and poor video quality when the video picture is complex due to coding according to the constant code rate in CBR coding, and improve the video quality after coding. On the other hand, the method and the device can solve the problems of video picture blocking and larger network delay caused by encoding with high code rate once detecting the video picture in ABR encoding, and improve the video quality after encoding on the premise of not influencing the video volume.

Having described the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are specifically described below.

Application scene overview

It should be noted that the following application scenarios are only shown for facilitating understanding of the spirit and principles of the present disclosure, and embodiments of the present disclosure are not limited in this respect. Rather, embodiments of the present disclosure may be applied to any scenario where applicable.

The embodiment of the disclosure supports detecting the type of each frame of video image in an input video stream in a live video and video communication scene, updating the number of detected target frames in a current coding period when the current frame is detected as the target frame, further, responding to the fact that the number of the target frames in the current coding period is larger than a first preset frame number and the number of the coded frames in the current coding period is larger than a second preset frame number, carrying out coding processing on the current frame according to a first code rate (the first code rate is higher than a reference code rate determined according to the current network performance), ending the current coding period and entering a next coding period. And in response to the number of target frames in the current coding period being greater than the first preset frame number and the number of coded frames in the current coding period not being greater than the second preset frame number, performing coding processing on the current frame according to a second code rate (the second code rate is lower than a reference code rate determined according to current network performance).

Exemplary method

In the related art, some embodiments employ the following two encoding methods:

① CBR encoding. In the live video and video communication scene, the video code rate is often required to be adjusted according to the network condition, and a code rate control mode of CBR is introduced for ensuring stable transmission of video. The CBR is a code mode with code rate priority, and according to a set target code rate, the CBR algorithm can adjust the QP (quantization parameter) of the coded video in real time, quantization is an important step in video coding, the larger the quantization value QP is, the higher the quantization granularity is, the larger the compression rate is, the smaller the code rate is, the lower the video quality is, the larger the mosaic is, the picture is not fine and smooth, and the picture is fuzzy. When the set target code rate is lower than the current code rate, the QP of the frame to be coded is regulated to be larger, the aim of reducing the code rate is fulfilled, and when the set target code rate is higher than the current code rate, the QP of the frame to be coded is regulated to be smaller, and the aim of improving the code rate is fulfilled. Fig. 1 shows a schematic diagram of a CBR encoding scheme, and referring to fig. 1, it can be seen that the code rate in CBR encoding remains substantially constant.

② ABR encoding. In a video on demand scene or a video live scene, an ABR coding mode is adopted, the ABR code rate control mode can maintain constant code rate within a period of time according to a set code rate, different code rates can be distributed according to the complexity of video, more code rates can be distributed for more complex frames, the coding quality of the complex frames is improved, and less code rates can be distributed for simpler frames, so that the overall quality of the video is controllable.

However, the above method has the following drawbacks:

In CBR coding, since the video coding rate is adjusted only according to the set target rate, when the video picture is complex, the loss of picture quality after coding is relatively large, and the subjective effect of the video picture is poor. On the other hand, since CBR is used to maintain a constant code rate within a period of time, coding loss and coding error in a certain frame are transmitted along with a reference relationship, resulting in poor video effect within a period of time, which is represented by a condition of blurred pictures and obvious blocking effect, and such coding loss and coding error are difficult to be eliminated by increasing the target code rate under the condition that the target code rate is substantially constant.

In ABR coding, the complexity of video frames in a period of time is synthesized, and different code rates are allocated to video frames with different complexity. For more complex video pictures in a period of time, the video code rate is higher than the set target code rate in a short time, the code rate is proportional to the video volume, and the video frame rate in network transmission is reduced due to the excessively high code rate, which is manifested as video picture blocking and large network delay. In video communication scenarios with high real-time requirements, excessive network delay can significantly impact user experience. Meanwhile, compared with a CBR coding mode, the ABR coding method has higher complexity, and higher coding pressure at the user side can cause the increase of CPU (Central Processing Unit ) overhead and power consumption, so that the ABR coding method is generally applied to server coding and has a not wide application range at the user side.

Therefore, a coding method is highly needed, and the display quality of the video can be improved without affecting the video volume.

Exemplary embodiments of the present disclosure first provide an encoding method. Fig. 2 shows an exemplary flow of the encoding method, which may include the following steps S210-S230:

In step S210, the number of target frames in the current encoding period is updated according to the type of the current frame.

In this embodiment, the current frame may be detected to determine the type of the current frame (the type of the current frame includes a target frame and a non-target frame), and then, when the current frame is detected as the target frame, the number of target frames in the current coding period is increased by one, and when the current frame is detected as the non-target frame, the number of target frames in the current coding period is kept unchanged.

For example, taking the example that the number of detected target frames in the current encoding period is 3 as an example, the number of target frames in the current encoding period may be updated to 4 when the current frame is detected as the target frame, and the number of target frames in the current encoding period may be kept to 3 when the current frame is detected as the non-target frame.

The input video may include a plurality of image frames, each frame represents a still image, and then the image being detected or processed at the current moment is the current frame.

The target frame may be an image frame satisfying at least one of the following conditions: the area of the salient region in the current frame is larger than a preset area threshold, the current frame contains a moving target, and the foreground region and the background region of the current frame have relative motion. Accordingly, when the current frame does not satisfy the condition of the target frame, it may be determined that the current frame is a non-target frame.

In step S220, in response to the number of target frames in the current encoding period being greater than the first preset frame number and the number of encoded frames in the current encoding period being greater than the second preset frame number, the current frame is encoded according to the first code rate, and the current encoding period is ended.

In this embodiment, when it is detected that the number of target frames in the current encoding period is greater than the first preset frame number and the number of encoded frames in the current encoding period is greater than the second preset frame number, the current frame may be encoded based on the h.264 or h.265 encoder according to the first code rate. Therefore, the method and the device can encode the current frame according to the first code rate only when the number of encoded frames is larger than the second preset frame number on the premise that the video picture is complex, so that the technical problems of overlarge volume and playing blocking caused by continuous high code rate encoding are avoided, video blocking caused by time delay is reduced, and watching experience of a user is optimized.

After encoding the current frame according to the first code rate, it may jump to the next encoding period (i.e., zero the number of target frames and zero the number of encoded frames). It can be seen that the current coding period in the present disclosure is a period of a non-fixed frame number, and each coding period starts with a video frame encoded according to the second code rate, ends with a video frame encoded according to the first code rate, i.e. once each time encoded at the first code rate, ends the current coding period and enters the next coding period.

The first preset frame number, that is, the number of preset target frames, may be, for example, a value greater than the second preset frame number and less than the third preset frame number, for example, when the second preset frame number is 2 frames and the third preset frame number is 6 frames, the first preset frame number may be set to 5 frames, and the specific value may be set by itself according to the actual situation, which is not particularly limited in the present disclosure. By setting the judgment condition of the first preset frame number, the complexity degree of the video picture in the current coding period can be detected, namely, when the number of target frames in the current coding period is larger than the first preset frame number, the video picture in the current coding period is determined to be sufficiently complex.

The second preset number of frames is a preset threshold of the number of encoded frames in each encoding period, for example: 2 frames, which can be set according to the actual situation, and the present disclosure is not limited in particular. By setting the judging condition of the second preset frame number, two frames of images coded by the first code rate can be separated, the problems of video picture blocking and large network delay caused by continuous high code rate coding are avoided, and the video transmission rate is ensured.

The first code rate is higher than a reference code rate determined based on current network performance. The network performance may be network bandwidth (refer to the amount of data that can be transmitted in a unit time), network delay (refer to the time taken for a packet or a packet to be transmitted from one end of the network to the other end), bandwidth delay product (refer to the product of the capability of a data link and the round-trip communication delay), etc., which may be set by itself according to practical situations, and the disclosure is not limited in particular.

In step S230, in response to the number of target frames in the current encoding period being greater than the first preset frame number and the number of encoded frames in the current encoding period not being greater than the second preset frame number, encoding the current frame according to the second code rate.

In this embodiment, when it is detected that the number of target frames in the current encoding period is greater than the first preset frame number and the number of encoded frames in the current encoding period is not greater than (less than or equal to) the second preset frame number, the encoding process may be performed on the current frame according to the second code rate.

Wherein the second code rate is lower than a reference code rate determined based on the current network performance. Therefore, the method and the device can solve the problems of video picture blocking and larger network delay caused by encoding with high code rate once detecting the video picture in the ABR encoding, and simultaneously can reduce the encoding complexity, reduce the encoding pressure of the user side, and not excessively increase the processing power consumption of the CPU compared with the ABR encoding mode, thereby being applicable to the encoding process of the user side.

According to the method, the complexity of the video picture in each coding period is determined according to the number of target frames in each coding period, on the premise that the video picture is sufficiently complex (the number of the target frames is larger than the first preset frame number), the number of the coded frames is required to be larger than the second preset frame number, the coded frames are coded with a first code rate with a higher value, and if the number of the coded frames is not larger than the second preset frame number, the coded frames are coded with a second code rate with a smaller value. Therefore, on one hand, the technical problems of large picture loss and poor video quality after coding caused by coding according to a constant code rate in CBR coding when video pictures are complex can be solved, and the video quality after coding is improved. On the other hand, the problems of video picture blocking and larger network delay caused by encoding with high code rate once the video picture is detected to be more complex in the ABR encoding can be solved, and the quality of the encoded video is improved on the premise of not affecting the video volume.

In the above step S210, in an alternative embodiment, the type of the current frame may be determined by a saliency detection algorithm, and in particular, referring to fig. 3, fig. 3 shows a flowchart for determining the type of the current frame according to an embodiment of the present disclosure, including steps S301 to S303:

in step S301, saliency detection is performed on the current frame to obtain the area of the saliency region.

In this embodiment, saliency detection may be performed on the current frame to obtain the area of the saliency region in the current frame. Wherein the salient region is the region in the current frame where the most attractive target object is located.

In step S302, in response to the area of the salient region being greater than the preset area threshold, it is determined that the current frame is the target frame.

In this embodiment, if the area of the salient region is greater than a preset area threshold (a preset area value may be changed according to actual conditions, which is not particularly limited in the present disclosure), the current frame may be determined to be the target frame.

In step S303, in response to the area of the salient region being not greater than the preset area threshold, it is determined that the current frame is a non-target frame.

In this embodiment, if the area of the salient region is not greater than (i.e., less than or equal to) the preset area threshold, it may be determined that the current frame is the non-target frame.

In the above step S210, in another alternative embodiment, the type of the current frame may be determined by a motion detection algorithm, and in particular, referring to fig. 4, fig. 4 shows another flowchart for determining the type of the current frame according to an embodiment of the present disclosure, including steps S401 to S403:

in step S401, motion detection is performed on the current frame to acquire whether a moving object is included in the current frame.

In this embodiment, a moving object detection algorithm may be used to perform motion detection on each frame in the input video to determine whether a moving object is included in each frame of image. The moving object is a dynamic object with a change in the position of the moving object in a certain frame of image of the input video. The motion detection algorithm may be an inter-frame difference method (i.e. a picture of a certain frame is selected and subtracted from an image to be identified), a background difference method (background modeling is performed to compare a background image with the picture to be identified), an optical flow method (a motion information of an object between adjacent frames is calculated according to a correspondence between a previous frame and a current frame by using a change of pixels in a time domain and a correlation between adjacent frames in an image sequence), a feature matching method, a KNN (K-Nearest Neighbor) Nearest Neighbor (Nearest Neighbor) classification algorithm, a KNN for short, a variation of these methods (three-frame difference, five-frame difference) and the like, which may be set by themselves according to actual conditions.

In step S402, in response to the moving object being included in the current frame, the current frame is determined as the target frame.

In this step, when the current frame is detected to include the moving object, it may be determined that the image complexity of the current frame is higher, that is, the current frame is the object frame.

In step S403, in response to the moving object not being included in the current frame, the current frame is determined to be a non-target frame.

In this step, when it is detected that the current frame does not include the moving object, it may be determined that the image complexity of the current frame is low, that is, the current frame is the non-object frame.

In the above step S210, in yet another alternative embodiment, the type of the current frame may be determined by a background detection algorithm, and in particular, referring to fig. 5, fig. 5 shows a flowchart for still another determination of the type of the current frame according to an embodiment of the present disclosure, including steps S501-S504:

In step S501, background detection is performed on the current frame to acquire a background area and a foreground area in the current frame.

In this embodiment, background detection may be performed on the current frame to obtain a background area and a foreground area in the current frame. The foreground region is a region where a subject target (e.g., a person) in the picture is located. Background area, i.e. area other than foreground area, for example: external environment, etc., which plays roles of setting off the foreground, coordinating the tone, etc., and is generally located behind the foreground.

In step S502, it is detected whether there is a relative motion between the foreground region and the background region.

In this embodiment, the relative positions of the foreground region and the background region in the current frame may be detected, and further, the relative positions of the foreground region and the background region in the previous frame or the next frame of the current frame may be detected, and the relative positions may be compared to determine whether there is a relative motion between the foreground region and the background region in the current frame.

In step S503, in response to the existence of the relative motion between the foreground region and the background region, it is determined that the current frame is the target frame.

In step S504, in response to there being no relative motion between the foreground region and the background region, it is determined that the current frame is a non-target frame.

In the step S220, if the number of target frames in the current encoding period is not greater than the first preset frame number, reference may be made to fig. 6, and fig. 6 shows a sub-flowchart of the encoding method according to an embodiment of the disclosure, including steps S601 to S602:

In step S601, in response to the number of target frames in the current coding period being not greater than the first preset frame number and the number of coded frames in the current coding period being greater than the third preset frame number, the current frame is coded according to the first code rate, and the current coding period is ended.

In this embodiment, when the number of target frames in the current coding period is detected to be not greater than (less than or equal to) the first preset frame number and the number of coded frames in the current coding period is detected to be greater than the third preset frame number, the current frame may be coded according to the first code rate, so that the coding process can be forced according to the higher code rate under the condition that the complexity of the video picture is lower, so as to avoid the technical problems of blurred video picture and lower quality caused by continuous low code rate coding, and improve the video quality.

Wherein the third preset frame number is a preset threshold value of the number of encoded frames, and the third preset frame number is greater than the second preset frame number. By setting the judging condition of the third preset frame number, the interval between two frames of images coded by the first code rate can be limited in a controllable range, the technical problems of larger video picture loss and poor picture quality caused by continuous low code rate coding are avoided, and the video quality is improved.

In step S602, in response to the number of target frames in the current encoding period being not greater than the first preset frame number and the number of encoded frames in the current encoding period being not greater than the third preset frame number, encoding the current frame according to the second code rate.

In this embodiment, when the number of target frames in the current coding period is detected to be not greater than (less than or equal to) the first preset frame number, and the number of coded frames in the current coding period is detected to be not greater than (less than or equal to) the third preset frame number, the current frame may be coded according to the second code rate, so that the coding can be performed with the second code rate with a smaller value under the condition that the complexity of the video picture is lower, so as to ensure that the code rate in each coding period is relatively balanced, and not to affect the overall volume of the video.

The second code rate may be determined by: the reference code rate is scaled down according to a preset scaling-down ratio to obtain a second code rate, for example, when the reference code rate is 1000 and the preset scaling-down ratio (a number greater than 0 and less than 1) is 0.8, the second code rate may be determined as: 1000 x 0.8=800. After determining the second code rate, considering that the second code rate is lower than the reference code rate, the tangential slope of the QP and λ (rate distortion optimization curve) of the current frame may be optimized by using lagrangian to obtain a rate distortion curve in the rate distortion optimization algorithm, where the linear slope tangent to the rate distortion curve at each code rate point is a λ value, so as to measure the code rate and loss during encoding according to the λ value, and adjust the code rate of the current frame so as to achieve the second code rate.

Specifically, different QPs correspond to different quantization step sizes in encoding and lambda values in rate distortion optimization, and the larger the QP value is, the larger the quantization step size is, the larger lambda is, the smaller the encoded code rate is, namely, the QP and the code rate are in inverse proportion, and the lambda and the code rate are in inverse proportion. Thus, QP and λ may be appropriately adjusted to reduce the coding rate of the current frame so that the coding rate of the current frame reaches the above-mentioned second rate.

The above-described first code rate may be determined by referring to fig. 7, fig. 7 shows a flowchart of determining the first code rate according to an embodiment of the present disclosure, including steps S701 to S703:

in step S701, the first N encoded frames of the current frame are acquired in the current encoding period.

In this embodiment, N is a positive integer greater than the second preset frame number and less than the third preset frame number, and N is exemplified as 3 in the current coding period. Referring to the above explanation of step S220, each time the current encoding period is encoded at the first code rate, the current encoding period is ended and the next encoding period is entered, so that the first N encoded frames of the current frame are all image frames encoded at the second code rate, i.e. the actual code rates of the first N encoded frames of the current frame are all the second code rate.

In step S702, a code rate difference between an actual code rate and a reference code rate of each encoded frame is obtained.

In this embodiment, referring to the explanation of the above steps, the actual code rate of each encoded frame is 800, and the reference code rate is 1000, so that the code rate difference between the actual code rate of each encoded frame and the reference code rate is 1000-800=200.

In step S703, a first code rate is determined according to the N code rate differences and the reference code rate.

In this embodiment, a first accumulated value of N code rate differences, 200+200+200=600, may be obtained, and further, a product of the first accumulated value and a reference code rate is obtained, 1000×600=600000, and further, a second accumulated value of the product and the reference code rate is obtained, 600000+1000= 610000, and the second accumulated value is determined as the first code rate.

After the first code rate is determined, since the first code rate is higher than the reference code rate, and as can be seen from the explanation of the above steps, QP is inversely proportional to the code rate, and λ is inversely proportional to the code rate, so that QP and λ can be appropriately reduced so that the encoding code rate of the current frame reaches the first code rate.

Therefore, the method and the device can save the code rate of the image frames coded according to the second code rate in one coding period and be used for the image frames coded according to the first code rate, so that the overall code rate balance of the video is ensured, the coding performance is improved, the technical problems of overhigh code rate and overlarge video volume of the video are avoided, the transmission bandwidth required by the video is saved, and the quality of the coded video is improved on the premise that the video volume is not influenced.

Fig. 8 shows an overall flowchart of an encoding method according to an embodiment of the present disclosure, including steps S801 to S813:

In step S801, start;

in step S802, initializing (setting zero to the number of detected target frames in the current coding period, setting zero to the number of coded frames in the current coding period, setting zero to the code rate saved by coded frames in the current coding period, and setting the second code rate to be the current reference code rate multiplied by a preset reduction ratio);

In step S803, it is detected whether the current frame is a target frame (the area of the salient region in the current frame is greater than a preset area threshold, and/or the current frame includes a moving target, and/or when there is a relative motion between the foreground region and the background region of the current frame, it is determined that the current frame is a target frame);

if the frame is the target frame, go to step S804, add one to the number of target frames;

If not, directly entering step S805 to detect whether the number of target frames in the current coding period is greater than a first preset frame number (by setting a judgment condition of the first preset frame number, the complexity of the video picture in the current coding period can be detected);

if the number of encoded frames is greater than the first preset frame number, step S806 is performed to determine whether the number of encoded frames is greater than the second preset frame number (by setting the determination condition of the second preset frame number, two frames of images encoded at the first code rate can be separated);

If the number of frames is greater than the second preset number of frames, the step S807 is entered to encode the current frame according to the first code rate (higher than the reference code rate determined according to the current network performance);

in step S808, it is determined whether the current group of pictures is traversed;

If the traversal is completed, the process proceeds to step S809, and ends; otherwise, step S802 is entered.

After the step S805, if the number of target frames is not greater than the first preset frame number, the process proceeds to step S810, where it is determined whether the number of encoded frames is greater than the third preset frame number (by setting a determination condition for the third preset frame number, the interval between two frames of images encoded at the first code rate can be limited within a controllable range); if the number of frames is greater than the third preset number of frames, the process proceeds to step S807; if not, entering step S811, encoding according to the second code rate (lower than the reference code rate determined according to the current network performance);

in step S812, it is determined whether the current frame group is traversed; if not, go to step S813, the number of non-target frames is increased by one;

if yes, the process proceeds to step S809, and ends.

Fig. 9 shows an overall flowchart of another encoding method according to an embodiment of the present disclosure, referring to fig. 9:

after the input video stream is subjected to frame type detection, the number of target frames in the current coding period can be updated;

Responding to the number of target frames in the current coding period being greater than a first preset frame number and the number of coded frames in the current coding period being greater than a second preset frame number, coding the current frames according to a first code rate, and outputting a video code stream;

And responding to the fact that the number of target frames in the current coding period is larger than the first preset frame number and the number of coded frames in the current coding period is not larger than the second preset frame number, carrying out coding processing on the current frames according to the second code rate, and outputting a video code stream.

Fig. 10 is a schematic diagram illustrating an encoding method according to an embodiment of the present disclosure, and referring to fig. 10, a quality frame in the figure is an image frame encoded according to a first code rate, and a non-quality frame is an image frame encoded according to a second code rate.

Exemplary apparatus

Having described the encoding method of the exemplary embodiment of the present disclosure, next, the encoding apparatus of the exemplary embodiment of the present disclosure will be described with reference to fig. 11.

Fig. 11 shows a schematic diagram of an encoding apparatus 1100 according to an embodiment of the present disclosure, the encoding apparatus 1100 including:

A frame type detection module 1110, configured to update the number of target frames in the current encoding period according to the type of the current frame; the type of the current frame comprises a target frame and a non-target frame;

A first encoding module 1120, configured to encode the current frame according to a first code rate and end the current encoding period, in response to the number of target frames in the current encoding period being greater than a first preset frame number and the number of encoded frames in the current encoding period being greater than a second preset frame number;

a second encoding module 1130, configured to perform encoding processing on the current frame according to a second code rate in response to the number of target frames in the current encoding period being greater than the first preset frame number and the number of encoded frames in the current encoding period not being greater than the second preset frame number; wherein the first code rate is higher than a reference code rate determined according to current network performance, and the second code rate is lower than the reference code rate determined according to current network performance.

In an alternative embodiment, first encoding module 1120 is configured to:

responding to the number of target frames in the current coding period not larger than the first preset frame number and the number of coded frames in the current coding period larger than a third preset frame number, coding the current frame according to the first code rate, and ending the current coding period;

wherein the third preset frame number is greater than the second preset frame number.

In an alternative embodiment, the second encoding module 1130 is configured to:

and in response to the number of target frames in the current coding period not being greater than the first preset frame number and the number of coded frames in the current coding period not being greater than the third preset frame number, coding the current frame according to the second code rate.

In an alternative embodiment, the second encoding module 1130 is configured to:

and carrying out reduction processing on the reference code rate according to a preset reduction ratio to obtain the second code rate.

In an alternative embodiment, the first encoding module 1120 is configured to:

Acquiring the first N coded frames of the current frame in the current coding period;

acquiring a code rate difference value between the actual code rate of each coded frame and the reference code rate;

Determining the first code rate according to the N code rate difference values and the reference code rate;

wherein N is a positive integer greater than the second preset number of frames and less than the third preset number of frames.

In an alternative embodiment, the first encoding module 1120 is configured to:

Acquiring N first accumulated values of the code rate difference values;

obtaining the product of the first accumulated value and the reference code rate;

And acquiring a second accumulated value of the product and the reference code rate, and determining the second accumulated value as the first code rate.

In an alternative embodiment, frame type detection module 1110 is configured to:

Performing significance detection on the current frame to obtain the area of a significance region;

determining that the current frame is the target frame in response to the area of the salient region being greater than a preset area threshold;

And determining that the current frame is the non-target frame in response to the area of the salient region being not greater than the preset area threshold.

Performing motion detection on the current frame to obtain whether the current frame contains a moving target or not;

determining the current frame as the target frame in response to the current frame containing the moving target;

And determining the current frame as the non-target frame in response to the current frame not including the moving target.

performing background detection on the current frame to obtain a background area and a foreground area in the current frame;

detecting whether relative motion exists between the foreground region and the background region;

Determining that the current frame is the target frame in response to relative motion of the foreground region and the background region;

And determining that the current frame is the non-target frame in response to the foreground region and the background region not having relative motion.

In addition, other specific details of the embodiments of the present disclosure are described in the foregoing embodiments of the method, and are not described herein.

Exemplary storage Medium

A storage medium according to an exemplary embodiment of the present disclosure is described below.

In the present exemplary embodiment, the above-described method may be implemented by a program product, such as a portable compact disc read only memory (CD-ROM) and including program code, and may be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RE, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (FAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

Exemplary electronic device

An electronic device of an exemplary embodiment of the present disclosure is described with reference to fig. 12.

The electronic device 1200 shown in fig. 12 is merely an example, and should not be construed as limiting the functionality and scope of use of the disclosed embodiments.

As shown in fig. 12, the electronic device 1200 is in the form of a general purpose computing device. Components of electronic device 1200 may include, but are not limited to: at least one processing unit 1210, at least one memory unit 1220, a bus 1230 connecting the different system components (including the memory unit 1220 and the processing unit 1210), and a display unit 1240.

Wherein the storage unit stores program code that is executable by the processing unit 1210 such that the processing unit 1210 performs steps according to various exemplary embodiments of the present disclosure described in the above-described "exemplary methods" section of the present specification. For example, the processing unit 1210 may perform the method steps shown in fig. 2, etc.

The storage units 1220 may include volatile storage units such as random access storage unit (RAM) 1221 and/or cache storage unit 1222, and may further include read only storage unit (ROM) 1223.

Storage unit 1220 may also include a program/utility 1224 having a set (at least one) of program modules 1225, such program modules 1225 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

Bus 1230 may include a data bus, an address bus, and a control bus.

The electronic device 1200 may also communicate with one or more external devices 1300 (e.g., keyboard, pointing device, bluetooth device, etc.) via an input/output (I/O) interface 1250. The electronic device 1200 also includes a display unit 1240 that is connected to an input/output (I/O) interface 1250 for display. Also, the electronic device 1200 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the internet through the network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 over bus 1230. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 1200, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

It should be noted that while several modules or sub-modules of the apparatus are mentioned in the detailed description above, such partitioning is merely exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module in accordance with embodiments of the present disclosure. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.

Furthermore, although the operations of the methods of the present disclosure are depicted in the drawings in a particular order, this is not required or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.

While the spirit and principles of the present disclosure have been described with reference to several particular embodiments, it is to be understood that this disclosure is not limited to the particular embodiments disclosed nor does it imply that features in these aspects are not to be combined to benefit from this division, which is done for convenience of description only. The disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method of encoding, comprising:

updating the number of target frames in the current coding period according to the type of the current frame; the type of the current frame comprises a target frame and a non-target frame; the target frame comprises at least one of the following image frames: the method comprises the steps of image frames with the area of a salient region being larger than a preset area threshold, image frames containing moving targets and image frames with relative movement of a foreground region and a background region;

Responding to the fact that the number of target frames in the current coding period is larger than a first preset frame number and the number of coded frames in the current coding period is larger than a second preset frame number, coding the current frame according to a first code rate, and ending the current coding period;

Responsive to the number of target frames in the current coding period being greater than the first preset frame number and the number of coded frames in the current coding period not being greater than the second preset frame number, performing coding processing on the current frame according to a second code rate;

wherein the first code rate is higher than a reference code rate determined according to current network performance, and the second code rate is lower than the reference code rate determined according to current network performance.

2. The method according to claim 1, wherein the method further comprises:

3. The method according to claim 2, wherein the method further comprises:

4. A method according to any one of claims 1 to 3, wherein the second code rate is determined by:

5. The method of claim 2, wherein the first code rate is determined by:

6. The method of claim 5, wherein said determining said first code rate based on N said code rate differences and said reference code rate comprises:

Acquiring N first accumulated values of the code rate difference values;

7. A method according to any one of claims 1 to 3, characterized in that the type of the current frame is determined by:

8. A method according to any one of claims 1 to 3, characterized in that the type of the current frame is determined by:

9. A method according to any one of claims 1 to 3, characterized in that the type of the current frame is determined by:

10. An encoding device, comprising:

the frame type detection module is used for updating the number of target frames in the current coding period according to the type of the current frame; the type of the current frame comprises a target frame and a non-target frame; the target frame comprises at least one of the following image frames: the method comprises the steps of image frames with the area of a salient region being larger than a preset area threshold, image frames containing moving targets and image frames with relative movement of a foreground region and a background region;

The first coding module is used for responding to the fact that the number of target frames in the current coding period is larger than a first preset frame number and the number of coded frames in the current coding period is larger than a second preset frame number, coding the current frame according to a first code rate, and ending the current coding period;

The second coding module is used for responding to the fact that the number of target frames in the current coding period is larger than the first preset frame number and the number of coded frames in the current coding period is not larger than the second preset frame number, and coding the current frames according to a second code rate;

11. The apparatus of claim 10, wherein the first encoding module is configured to:

12. The apparatus of claim 11, wherein the second encoding module is configured to:

13. The apparatus according to any of claims 10 to 12, wherein the second encoding module is configured to:

14. The apparatus of claim 11, wherein the first encoding module is configured to:

15. The apparatus of claim 14, wherein the first encoding module is configured to:

Acquiring N first accumulated values of the code rate difference values;

16. The apparatus according to any one of claims 10 to 12, wherein the frame type detection module is configured to:

17. The apparatus according to any one of claims 10 to 12, wherein the frame type detection module is configured to:

18. The apparatus according to any one of claims 10 to 12, wherein the frame type detection module is configured to:

19. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method of any one of claims 1-9.

20. An electronic device, comprising:

A processor; and

A memory for storing executable instructions of the processor;

Wherein the processor is configured to perform the method of any one of claims 1-9 via execution of the executable instructions.