CN117676249A

CN117676249A - Live video processing method and device, electronic equipment and storage medium

Info

Publication number: CN117676249A
Application number: CN202311674909.0A
Authority: CN
Inventors: 段晨辉; 陈靖
Original assignee: Shuhang Technology Beijing Co ltd
Current assignee: Shuhang Technology Beijing Co ltd
Priority date: 2023-12-07
Filing date: 2023-12-07
Publication date: 2024-03-08
Anticipated expiration: 2043-12-07

Abstract

The embodiment of the application discloses a live video processing method, a live video processing device, electronic equipment and a storage medium. According to the scheme, the uplink bandwidth of the live video stream pushed to the server at present is acquired, and then the target network state is determined according to the uplink bandwidth, wherein if the target network state does not belong to a preset ideal network state, the coding rate of an encoder corresponding to the live video stream can be adjusted based on the target network state; if the target network state belongs to a preset ideal network state, the encoding code rate of the encoder and/or the image frame quality parameter of the live video stream can be monitored according to a preset time interval, at least one encoding parameter of the encoder is adjusted based on the monitoring result, and finally, the live video stream is processed based on the adjusted encoder. Therefore, the encoder can be adaptively adjusted according to different network states and different coding parameters, so that the coding quality of the encoder is improved.

Description

Live video processing method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method and apparatus for processing live video, an electronic device, and a storage medium.

Background

With the continuous development of computer technology, network live broadcast is more and more exploded, and numerous live broadcast real-time services attract a large number of users due to original novel contents. Live push is a key link in live broadcast, and refers to a process of transmitting content packaged in a collection stage to a server.

In the related art, a hardware encoder is generally used to encode live audio and video, and the encoded audio and video information is pushed to a network. However, in the live broadcast process, the network environment fluctuates, and the existing encoder cannot flexibly adjust the code rate of the encoder, so that the encoding quality of the encoder is affected.

Disclosure of Invention

The embodiment of the application provides a method, a device, electronic equipment and a storage medium for processing live video, which can improve the coding quality of an encoder.

The embodiment of the application provides a processing method of live video, which comprises the following steps:

acquiring an uplink bandwidth of a live video stream currently pushed to a server;

determining a target network state according to the uplink bandwidth;

if the target network state does not belong to a preset ideal network state, adjusting the coding rate of the encoder corresponding to the live video stream based on the target network state;

If the target network state belongs to the preset ideal network state, monitoring the coding rate of the encoder and/or the image frame quality parameter of the live video stream according to a preset time interval, and adjusting at least one coding parameter of the encoder based on a monitoring result;

and processing the live video stream based on the adjusted encoder.

Correspondingly, the embodiment of the application also provides a processing device for live video, which comprises:

the acquisition unit is used for acquiring the uplink bandwidth of the live video stream pushed to the server currently;

a determining unit, configured to determine a target network state according to the uplink bandwidth;

the first adjusting unit is used for adjusting the coding rate of the coder corresponding to the live video stream based on the target network state if the target network state does not belong to a preset ideal network state;

the second adjusting unit is used for monitoring the coding rate of the encoder and/or the image frame quality parameter of the live video stream according to a preset time interval if the target network state belongs to the preset ideal network state, and adjusting at least one coding parameter of the encoder based on a monitoring result;

And the processing unit is used for processing the live video stream based on the adjusted encoder.

Correspondingly, the embodiment of the application also provides electronic equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the processing method of the live video provided by any one of the embodiments of the application.

Accordingly, the embodiment of the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the live video processing method.

Embodiments of the present application also provide a computer program product or computer program comprising computer instructions stored in a storage medium. The computer instructions are read from the storage medium by a processor of the electronic device, which executes the computer instructions, causing the electronic device to perform the method of processing live video provided in various alternative implementations of the above aspects.

According to the embodiment of the application, the uplink bandwidth of the live video stream pushed to the server is acquired, and then the target network state is determined according to the uplink bandwidth, wherein if the target network state does not belong to the preset ideal network state, the encoding code rate of the encoder corresponding to the live video stream can be adjusted based on the target network state; if the target network state belongs to a preset ideal network state, the encoding code rate of the encoder and/or the image frame quality parameter of the live video stream can be monitored according to a preset time interval, at least one encoding parameter of the encoder is adjusted based on the monitoring result, and finally, the live video stream is processed based on the adjusted encoder. Therefore, the encoder can be adaptively adjusted according to different network states and different coding parameters, so that the coding quality of the encoder is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flow chart of a method for processing live video according to an embodiment of the present application.

Fig. 2 is a flow chart of another processing method of live video according to an embodiment of the present application.

Fig. 3 is a flow chart of another processing method of live video according to an embodiment of the present application.

Fig. 4 is a block diagram of a live video processing device according to an embodiment of the present application.

Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

The embodiment of the application provides a live video processing method and device, a storage medium and electronic equipment. Specifically, the method for processing live video in the embodiment of the present application may be performed by an electronic device, where the electronic device may be a device such as a terminal or a server. The terminal can be a terminal device such as a smart phone, a tablet computer, a notebook computer, a touch screen, a personal computer (PC, personal Computer), a personal digital assistant (Personal Digital Assistant, PDA) and the like. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), basic cloud computing services such as big data and artificial intelligent platforms, and the like.

For example, the electronic device may be a terminal, and the terminal may acquire an uplink bandwidth of a live video stream currently pushed to the server; determining a target network state according to the uplink bandwidth; if the target network state does not belong to the preset ideal network state, adjusting the coding rate of the encoder corresponding to the live video stream based on the target network state; if the target network state belongs to a preset ideal network state, monitoring the coding rate of the encoder and/or the image frame quality parameter of the live video stream according to a preset time interval, and adjusting at least one coding parameter of the encoder based on a monitoring result; and processing the live video stream based on the adjusted encoder.

Based on the above problems, the embodiments of the present application provide a method, an apparatus, an electronic device, and a storage medium for processing a live video, which can improve the encoding quality of an encoder.

The following will describe in detail. The following description of the embodiments is not intended to limit the preferred embodiments.

The embodiment of the application provides a method for processing live video, which can be executed by a terminal or a server.

Referring to fig. 1, fig. 1 is a flow chart of a method for processing live video according to an embodiment of the present application. The specific flow of the live video processing method can be as follows:

101. and acquiring the uplink bandwidth of the live video stream currently pushed to the server.

In the embodiment of the application, the live video stream may be video data generated by a hosting client. Upstream bandwidth refers to the data transmission rate at which the anchor client sends information to the server.

102. And determining the state of the target network according to the uplink bandwidth.

The determining the target network state according to the uplink bandwidth may refer to evaluating the network state of the current anchor client pushing the live video stream to the server according to the uplink bandwidth, that is, the target network state.

In some embodiments, in order to accurately detect the target network state, the step of determining the target network state according to the uplink bandwidth may include the following operations:

acquiring a current coding code rate of an encoder and presetting a maximum code rate;

if the uplink bandwidth is smaller than the preset maximum code rate, or the uplink bandwidth is not smaller than the preset code rate and the current coding code rate is smaller than the preset maximum code rate, determining that the target network state does not belong to the preset ideal network state;

if the uplink bandwidth is not smaller than the preset maximum code rate and the current coding code rate is not smaller than the preset maximum code rate, determining that the target network state belongs to the preset ideal network state.

An encoder (encoder) is a device that compiles, converts, or converts, a signal (e.g., a bit stream) or data into a signal form that can be used for communication, transmission, and storage.

In this embodiment of the present application, the encoder may be configured to encode video data of a live video stream, obtain encoded video data, and push the encoded video data to a server. The coding rate of the coder refers to the data size obtained by coding in unit time of the coder. For example, the coding rate may be 100kbps, indicating that the size of data encoded by the encoder per second is 100kb.

The current coding rate may be a maximum coding rate supported by the encoder currently encoding the live video stream. The preset maximum code rate can be the maximum data of data transmission in the unit time which is initially set. The preset maximum code rate can be set according to different conditions, and is not limited herein.

Specifically, the upper bandwidth, the lower bandwidth, the current coding rate and the preset maximum rate can be respectively compared, and the target network state is determined according to the comparison result.

Wherein the preset ideal network state may indicate that the network condition is good.

Firstly, the uplink bandwidth can be compared with a preset maximum code rate, and if the uplink bandwidth is smaller than the preset maximum code rate, the target network state can be determined not to belong to the preset ideal network state; if the uplink bandwidth is greater than or equal to the preset maximum code rate, the current coding code rate can be further compared with the preset maximum code rate, and if the current coding code rate is less than the preset maximum code rate, the target network state can be determined not to belong to the preset ideal network state; if the current coding rate is greater than or equal to the preset maximum coding rate, the target network state can be determined to belong to the preset ideal network state.

For example, the upstream bandwidth may be: the rate, the current coding rate may be maxrate1, and the preset maximum rate may be maxrate0. Respectively comparing the rate, the maxrate1 and the maxrate0, and if the rate is smaller than the maxrate0, determining that the target network state does not belong to a preset ideal network state; if the rate is greater than or equal to the maxrate0 and the maxrate1 is less than the maxrate0, determining that the target network state does not belong to a preset ideal network state; if the rate is not less than maxrate0 and the maxrate1 is not less than maxrate0, determining that the target network state belongs to a preset ideal network state.

In some embodiments, in order to improve the coding quality of the encoder in different network states, the step of determining that the target network state does not belong to the preset ideal network state "may include the following operations:

if the uplink bandwidth is smaller than the preset maximum code rate, determining that the target network state belongs to the first network state;

if the uplink bandwidth is not smaller than the preset code rate and the current coding code rate is smaller than the preset maximum code rate, determining that the target network state belongs to the second network state.

The first network state may be a weak network state, which indicates that the network condition is poor; the second network state may be a transition state from a weak network state to a preset ideal network state, i.e. indicating that the network condition is recovered from poor to good.

For example, the upstream bandwidth may be: the rate, the current coding rate may be maxrate1, and the preset maximum rate may be maxrate0. Respectively comparing the rate, the maxrate1 and the maxrate0, and if the rate is smaller than the maxrate0, determining that the target network state belongs to a weak network state; if the rate is greater than or equal to maxrate0 and maxrate1 is less than maxrate0, determining that the target network state belongs to the network transition state.

103. And if the target network state does not belong to the preset ideal network state, adjusting the coding rate of the encoder corresponding to the live video stream based on the target network state.

The encoder corresponding to the live video stream is an encoder for encoding the live video stream.

In the embodiment of the present application, the network state may include a preset ideal network state, a first network state, a second network state, and the like. The target network state not belonging to the preset ideal network state may include: the target network state belongs to a first network state or the target network state belongs to a second network state.

Wherein different adjustment schemes for the encoder are set for the first network state and the second network state, respectively.

In some embodiments, in order to improve the encoding quality of the encoder in different network states, the step of "if the target network state does not belong to the preset ideal network state, adjusting the encoding rate of the encoder corresponding to the live video stream based on the target network state" may include the following operations:

If the target network state belongs to the first network state, adjusting the coding rate of the coder based on the available coding rate corresponding to the first network state;

and if the target network state belongs to the second network state, adjusting the coding rate of the coder based on the preset maximum code rate.

The available code rate corresponding to the first network state refers to the available code rate detected in the weak network state.

In the embodiment of the application, for the weak network state, a code rate control strategy of the outer layer SDK is preset for adjusting the coding code rate of the encoder, that is, the available code rate detected in the weak network state is used as the coding code rate supported by the encoder.

The detecting of the available code rate in the weak network state can be achieved through a probr detecting module, and the probr detecting module can be used for detecting the current bandwidth rapidly and periodically. The realization principle of the probr detection module is as follows:

(1) The transmitting end transmits data packets at a certain speed, and simultaneously records the transmission time, the serial number (globally unique) and the ID of the detection group of the data packets, wherein the serial number and the ID can be used for distinguishing different transmitted data packets.

(2) The receiving end calculates the code rate (the size of the data packet/(the receiving time-the sending time)) of the receiving end according to the size of the received data packet and the receiving time, and feeds back the code rate to the sending end.

(3) The transmitting end selects the minimum value from the transmitting code rate (the transmitting code rate can be the code rate of the transmitting end transmitting the data packet) and the code rate fed back by the receiving end as the current transmitting code rate value.

In this embodiment of the present application, the anchor may send a data packet to the server at a certain speed in a weak network state, record information such as a sending time of the data packet, and after the server receives the data packet sent by the anchor, calculate a receiving code rate of the server according to a size and a receiving time of the received data packet, and feed back the receiving code rate to the anchor, where the anchor may select a minimum code rate from the sending code rate and the receiving code rate, determine the minimum code rate as a current available code rate, and then set the current available code rate as a currently supported coding code rate of the encoder.

For example, if it is detected that the target network state belongs to the first network state, the available code rate detected in the weak network state is acquired, and the available code rate is set as the maximum coding code rate supported by the encoder.

For another example, if the detected target network state belongs to the second network state, the preset maximum code rate may be set as the maximum coding rate supported by the encoder.

104. If the target network state belongs to a preset ideal network state, monitoring the coding rate of the encoder and/or the image frame quality parameter of the live video stream according to a preset time interval, and adjusting at least one coding parameter of the encoder based on the monitoring result.

The preset time interval may be used to control the adjustment frequency of the encoder, for example, the preset time interval may be t1, and the adjustment frequency of the encoder may be t 1/time, that is, the encoder may be adjusted once every t 1.

In some embodiments, since the adjustment of the faster frequency may cause the encoder to encode disturbance, the preset time interval may be set to be longer, for example, 5 minutes, etc., and may be specifically set in association with different application scenarios.

In this embodiment of the present application, a state machine may be designed, through which the coding rate of the encoder and/or the image frame quality parameter of the live video stream are monitored at preset time intervals, and at least one coding parameter of the encoder is adjusted based on the monitoring result.

The state machine is composed of a state register and a combinational logic circuit, can perform state transition according to a preset state according to a control signal, and is a control center for coordinating related signal actions and completing specific operations.

In some embodiments, in order to improve the coding quality of the encoder in the ideal network state, the step of "monitoring the coding rate of the encoder and/or the image frame quality parameter of the live video stream at preset time intervals, and adjusting at least one coding parameter of the encoder based on the monitoring result" may include the following operations:

Monitoring the coding rate of the coder based on a preset time interval;

comparing the monitored coding rate of the coder with a preset maximum coding rate;

and if the coding code rate of the coder is smaller than the preset maximum code rate, adjusting a first coding parameter of the coder.

Firstly, the coding rate of the coder can be monitored, the monitored coding rate of the coder is compared with a preset maximum coding rate to obtain a first comparison result, and then the coder can be adjusted according to the first comparison result.

Wherein, based on the preset time interval, monitoring the coding rate of the coder comprises: and obtaining the total amount of the coded video data obtained by coding the live video stream at a preset time interval, and calculating the ratio of the total amount of the video data to the preset time interval, thereby obtaining the coding rate of the coder at the preset time interval.

For example, the preset time interval may be 5 minutes, and the total amount of video data encoded by the encoder in a preset time interval may be: 10000kb, the code rate can be calculated to be 33.3kbps, namely, the actual code rate of the encoder is 33.3kb per second in the preset time interval.

On the one hand, if the first comparison result indicates that the coding rate of the encoder is smaller than the preset maximum rate, the coding rate of the current encoder can be considered to be a lower rate for the live video stream under the 720p resolution, and at this time, the first coding parameter of the encoder can be adjusted down.

The first coding parameter refers to an index for evaluating the coding quality of the encoder, and the lower the value of the first coding parameter, the higher the coding quality of the encoder can be represented, whereas the higher the value of the first coding parameter, the lower the coding quality of the encoder can be represented.

In some embodiments, the first encoding adaptation number may be crf, which refers to a fixed rate coefficient, that controls the output quality of the x.264 and x.265 encoders, with a value between 0-51. The lower the CRF value setting, the higher the quality (code rate) of the output video, but the larger the video file, and vice versa.

On the other hand, if the first comparison result indicates that the coding rate of the encoder is greater than or equal to the preset maximum rate, the image frame quality parameter can be monitored, and the encoder is adjusted according to the monitored image frame quality parameter.

In some embodiments, the method may further comprise the steps of:

If the coding code rate of the coder is not less than the preset maximum code rate, monitoring the image frame quality parameter of the live video stream based on a preset time interval;

comparing the monitored image frame quality parameter with a first preset image frame quality parameter;

if the image frame quality parameter is smaller than the first preset image frame quality parameter, the coding rate of the coder is increased.

The image frame quality parameter refers to an index for evaluating whether the image quality of a video frame in a live video stream is good or bad, and the higher the value of the image frame quality parameter, the better the image quality of the video frame can be represented, otherwise, the lower the value of the image frame quality parameter, the worse the image quality of the video frame can be represented.

In some embodiments, the image frame quality parameter may be psnr (Peak Signal to Noise Ratio ), which is an objective criterion for evaluating an image, typically an engineering project for use between maximum signal and background noise. Typically, after image compression, the output image is somewhat different from the original image. To measure the quality of the processed image, the psnr value is referenced to measure whether a certain processing procedure is satisfactory.

Wherein psnr can be defined by mean square error, for example, two m×n images F and G, and the mean square error of the two images can be defined as follows:

and psnr is defined as follows:

the image F may be an image frame before encoding by the encoder, the image G may be an image frame after encoding by the encoder, m and n represent image sizes, and MAX represents a maximum value of a pixel color of the image, and if the color sampling point is represented by 8 bits, MAX may be 255.

In this embodiment of the present application, the anchor side may obtain two video frames of a video frame in a live video stream before and after encoding by an encoder, then obtain image information of the two video frames, including an image size, an image color, and the like, and then calculate according to the above calculation formula, so as to obtain psnr of the video frame before and after encoding, where the greater the psnr, the less distortion of the video frame after encoding, that is, the higher encoding quality of the encoder is reflected.

When the monitored coding rate of the encoder is greater than or equal to the preset maximum coding rate, the image frame quality parameter can be monitored, the monitored image frame quality parameter is compared with the first preset image frame quality parameter, a second comparison result is obtained, and the encoder is adjusted according to the second comparison result.

In one aspect, if the second comparison result indicates that the image frame quality parameter is smaller than the first preset image frame quality parameter, the coding rate of the encoder may be increased, for example, the maximum coding rate of the encoder may be extended.

On the other hand, if the second comparison result indicates that the image frame quality parameter is greater than or equal to the first preset image frame quality parameter, the encoder is not required to be adjusted, and the image frame quality parameter is continuously monitored based on the preset time interval.

In some embodiments, in order for the monitored image frame quality parameter to reflect the image frame transient, the step of "listening to the image frame quality parameter of the live video stream based on a preset time interval" may include the following parameters:

acquiring a picture group currently coded in a live video stream;

and obtaining the image frame quality parameters of the live video stream according to the average value of the image frame quality parameters of the video frames in the picture group.

Wherein the group of pictures, gop (Group of Pictures), comprises a set of consecutive pictures, i.e. a plurality of video frames. The gp is a picture set in the sequence used to assist random access. The first picture of the GOP must be an I-frame to ensure that the GOP can be decoded independently without reference to other pictures.

The group of pictures currently being encoded may refer to a group of pictures currently being encoded by the encoder within a preset time interval.

Specifically, calculating the average value of the image frame quality parameters of all the video frames in the group of pictures according to the average value of the image frame quality parameters of the video frames in the group of pictures may include calculating the average value of the image frame quality parameters of all the video frames in the group of pictures.

For example, the frame group may include a first video frame, a second video frame, a third video frame, and a fourth video frame, where the image frame quality parameter of the first video frame may be p1, the image frame quality parameter of the second video frame may be p2, the image frame quality parameter of the third video frame may be p3, the image frame quality parameter of the fourth video frame may be p4, and an average value of p1, p2, p4, and p4 is calculated as the image frame quality parameter of the live video stream in real time.

In some embodiments, to improve the image quality of the video frame, after the step of "raising the coding rate of the encoder", the following steps may be further included:

if the monitored image frame quality parameter is larger than the second preset image frame quality parameter, the coding rate of the coder is reduced.

Wherein the second preset image frame quality parameter is greater than the first preset image frame quality parameter.

Monitoring an image frame quality parameter after expanding the maximum coding rate of the encoder, and if the monitored image frame quality parameter is larger than a second preset image frame quality parameter, reducing the maximum coding rate of the encoder; if the monitored image frame quality parameter is in the range between the first preset image frame quality parameter and the second preset image frame quality parameter, the encoder is not required to be adjusted, and the image frame quality parameter is continuously monitored based on the preset time interval.

and if the monitored image frame quality parameter is smaller than the first preset image frame quality parameter, adjusting a second coding parameter of the coder.

The second coding parameter refers to an index for evaluating the image quality of the encoded video frame, if the second coding parameter is higher, the image quality of the encoded video frame is higher, otherwise, if the second coding parameter is lower, the image quality of the encoded video frame is lower.

In some embodiments, the second encoding parameter may be a noise reduction parameter of the encoder.

In this embodiment of the present application, the encoder may be a software encoder (that is, a manner of encoding video by using software itself or using a CPU), for example, may be an x264 software encoder, where an x264 software encoder is configured with a filtering module, and the filtering module may be used to perform noise reduction processing on a video frame image in a live video stream before encoding the live video stream.

The process of noise reduction processing on the video frame image through the filtering module may include: dividing a video frame image into a plurality of pixel blocks, wherein each pixel block comprises a plurality of pixel points, processing each pixel block by a preset filtering processing mode such as a mean value filtering mode, and obtaining the video frame image after all the pixel blocks are processed.

The noise reduction parameter may be an NR parameter in the filtering module, where the NR parameter is used to control the number of pixel blocks obtained by dividing the video frame image, and the larger the NR parameter is, the more the number of pixel blocks obtained by dividing the video frame image is, the higher the fineness of processing the pixel blocks in a preset filtering manner is, so that the better the noise reduction effect of the processed video frame image is obtained after the video frame image is filtered.

In the embodiment of the application, the noise reduction parameters of the filtering module in the encoder are dynamically adjusted through the detected image frame quality representation, so that the encoded live video stream can be ensured to keep a good image effect.

After the coding rate of the encoder is increased, if the image frame quality parameter is still smaller than the first preset image frame quality parameter, the simple increase of the coding rate does not improve the image quality of the video frame, which is caused by too much noise in the image, so that the noise reduction parameter of the encoder can be reduced.

In some embodiments, to improve the image quality of the video frame, after the step of "adjusting the second encoding parameter of the encoder", the steps of:

if the monitored image frame quality parameter is larger than the second preset image frame quality parameter, the coding rate of the coder is reduced, and the second coding parameter of the coder is reduced.

After adjusting the second coding parameter of the encoder, monitoring the number of image frame quality adoption continuously, and if the monitored image frame quality parameter is larger than the second preset image frame quality parameter, adjusting the coding code rate of the encoder and adjusting the second coding parameter of the encoder; if the monitored image frame quality parameter is in the range between the first preset image frame quality parameter and the second preset image frame quality parameter, the encoder is not required to be adjusted, and the image frame quality parameter is continuously monitored based on the preset time interval.

105. And processing the live video stream based on the adjusted encoder.

In the embodiment of the application, when the adjusted encoder meets the requirements that the encoding quality and the image frame quality parameters are in good states, the video data of the live video stream can be encoded, and the encoded video data is pushed to the server, so that the server sends the live video data to the client of the audience, and the audience users watch live broadcast.

The embodiment of the application discloses a processing method of live video, which comprises the following steps: acquiring an uplink bandwidth of a live video stream currently pushed to a server; determining a target network state according to the uplink bandwidth; if the target network state does not belong to the preset ideal network state, adjusting the coding rate of the encoder corresponding to the live video stream based on the target network state; if the target network state belongs to a preset ideal network state, monitoring the coding rate of the encoder and/or the image frame quality parameter of the live video stream according to a preset time interval, and adjusting at least one coding parameter of the encoder based on a monitoring result; and processing the live video stream based on the adjusted encoder. Thus, the coding quality of the encoder can be improved.

In light of the foregoing, the processing method of live video of the present application will be further described below by way of example. Referring to fig. 2, fig. 2 is a flow chart of another method for adjusting live content according to an embodiment of the present application, and a specific flow may be as follows:

firstly, acquiring a current uplink bandwidth, judging whether the current uplink bandwidth is in a weak network, and specifically, judging whether a target code rate corresponding to the current uplink bandwidth is smaller than an initialization maximum code rate maxrate, wherein the initialization maximum code rate can be: m1.

If the current target code rate is smaller than the initialized maximum code rate maxrate, the current target code rate can be considered to be in a weak network state, and if the current target code rate is in a weak network state, the code rate control strategy of the whole software encoder can be according to the code rate control strategy of the outer layer sdk, namely the available code rate detected by the current weak network is used as the maximum code rate maxrate supported by the encoder;

if the encoder is not currently in the weak network, but the maxrate of the current encoder is smaller than the initialized maxrate: m1, the current network state can be considered to be just recovered from the weak network, and the initialized maxrate, m1 can be set to the encoder;

if the network is currently in an ideal network, a state machine can be designed, and the state machine can be monitored and adjusted 1 time at intervals of t1 time, and the state machine can comprise the following procedures:

Firstly, monitoring the coding rate of an encoder, if the coding rate of the encoder is monitored to be lower than m1, considering that the coding rate is a lower rate for the resolution of live 720p, and at the moment, adjusting down the crf to ensure that the encoder spends more bits to code better quality under a given maxrate so as to achieve the quality improvement effect;

if the coding rate of the encoder is monitored to be greater than or equal to m1, psnr can be monitored and the encoder adjusted 1 time per interval t1 to adjust the state transition, wherein:

initialization State State0: if psnr > =p1, continuing to monitor psnr, and doing nothing; if psnr < p1, the encoder maxrate can be extended to m2, the code rate is increased to cope with such complex scenes, and the State is switched to State1, wherein m2> m1;

adjusting code rate State1: if the psnr is monitored to be in the range of [ p1, p2] after the maxrate of the encoder is adjusted, monitoring can be continued, and no operation exists; if psnr > p2, the maxrate of the encoder will be reduced to m1 and State0 will be returned; if psnr is still < p1, that is, the simple increase of the code rate does not improve the video image quality, the noise in the image is too much, so that the noise reduction threshold of the encoder can be adjusted to n2, and the State2 is shifted;

Adjusting the noise State State2: if the noise is adjusted, psnr is in the range of [ p1, p2], monitoring can be continued, and no operation exists; if psnr > p2, the maxrate of the encoder is reduced to m1 and the noise reduction threshold is reduced to n1 and converted to State0. By using a state machine to carry out self-adaptive code rate control on different coding parameters, the coding quality of the coder can be improved.

In the embodiment of the application, a method for processing live video suitable for a live broadcast system is also provided, wherein the live broadcast system comprises a viewer terminal, a host terminal and a server (including but not limited to a live broadcast server) of an interaction platform.

Referring to fig. 3, fig. 3 is a flowchart of another method for adjusting live content according to an embodiment of the present application, where the method for processing live video includes:

201. the method comprises the steps that a main broadcasting end obtains an uplink bandwidth of a live video stream pushed to a server currently;

202. the anchor end determines a target network state according to the uplink bandwidth;

203. if the target network state does not belong to the preset ideal network state, the anchor terminal adjusts the coding rate of the encoder corresponding to the live video stream based on the target network state;

204. if the target network state belongs to a preset ideal network state, the anchor end monitors the coding rate of the encoder and/or the image frame quality parameter of the live video stream according to a preset time interval, and adjusts at least one coding parameter of the encoder based on a monitoring result;

205. The main broadcasting terminal processes the live video stream based on the adjusted encoder;

206. the anchor terminal sends the processed live video stream to a server;

207. the server starts a target live broadcasting room according to the live video stream and the user information of the host broadcasting, and generates live broadcasting interface data required by the interface display of the live broadcasting room;

208. the server pushes the live interface data to the anchor terminal;

209. the main broadcasting end renders and displays a live broadcasting room interface of the target live broadcasting room according to the live broadcasting interface data;

210. the audience terminal initiates a watching request to a server of a target living broadcast room;

it is understood that step 210 may be performed after the target live room is opened.

211. The server sends live interface data to the audience terminal;

212. and the audience terminal receives the live broadcast interface data, and renders and displays a live broadcast room interface of the target live broadcast room.

By adopting the embodiment, the method and the device for the live broadcast provide a method and device for adaptively adjusting the encoder according to different network states and different encoding parameters, so that the quality of encoding the live video stream by the encoder is improved, the quality effect of the live video stream is ensured, and the live broadcast watching experience of a user is facilitated.

In order to facilitate better implementation of the live video processing method provided by the embodiment of the application, the embodiment of the application also provides a live video processing device based on the live video processing method. The meaning of the nouns is the same as that in the live video processing method, and specific implementation details can be referred to the description in the method embodiment.

Referring to fig. 4, fig. 4 is a block diagram of a live video processing apparatus according to an embodiment of the present application, where the apparatus includes:

an obtaining unit 401, configured to obtain an uplink bandwidth of a live video stream currently pushed to a server;

a determining unit 402, configured to determine a target network state according to the uplink bandwidth;

a first adjusting unit 403, configured to adjust, based on the target network state, an encoding rate of an encoder corresponding to the live video stream if the target network state does not belong to a preset ideal network state;

a second adjusting unit 404, configured to monitor, if the target network state belongs to the preset ideal network state, a coding rate of the encoder and/or an image frame quality parameter of the live video stream according to a preset time interval, and adjust at least one coding parameter of the encoder based on a monitoring result;

and a processing unit 405, configured to process the live video stream based on the adjusted encoder.

In some embodiments, the second adjustment unit 404 may include:

a monitoring subunit, configured to monitor a coding rate of the encoder based on the preset time interval;

the first comparison subunit is used for comparing the monitored coding rate of the coder with a preset maximum coding rate;

And the first adjusting subunit is used for adjusting the first coding parameter of the coder if the coding code rate of the coder is smaller than the preset maximum code rate.

In some embodiments, the second adjustment unit 404 may further include:

the second monitoring subunit is used for monitoring the image frame quality parameters of the live video stream based on the preset time interval if the coding code rate of the coder is not less than the preset maximum code rate;

a second comparing subunit, configured to compare the monitored image frame quality parameter with a first preset image frame quality parameter;

and the second adjusting subunit is used for adjusting the coding rate of the coder if the image frame quality parameter is smaller than the first preset image frame quality parameter.

In some embodiments, the second listening subunit may be specifically configured to:

acquiring a picture group which is currently coded in the live video stream, wherein the picture group comprises at least one video frame;

In some embodiments, the second adjustment unit 404 may further include:

And the third adjusting subunit is used for adjusting the coding rate of the coder if the monitored image frame quality parameter is larger than a second preset image frame quality parameter, wherein the second preset image frame quality parameter is larger than the first preset image frame quality parameter.

In some embodiments, the second adjustment unit 404 may further include:

and the fourth adjusting subunit is used for adjusting the second coding parameter of the coder if the monitored image frame quality parameter is smaller than the first preset image frame quality parameter.

In some embodiments, the second adjustment unit 404 may further include:

and a fifth adjusting subunit, configured to adjust the coding rate of the encoder and adjust the second coding parameter of the encoder if the monitored image frame quality parameter is greater than a second preset image frame quality parameter.

In some embodiments, the determining unit 402 may include:

the acquisition subunit is used for acquiring the current coding rate of the coder and presetting the maximum coding rate;

a first determining subunit, configured to determine that the target network state does not belong to the preset ideal network state if the uplink bandwidth is smaller than the preset maximum code rate, or if the uplink bandwidth is not smaller than the preset code rate and the current coding code rate is smaller than the preset maximum code rate;

And the second determining subunit is configured to determine that the target network state belongs to the preset ideal network state if the uplink bandwidth is not less than a preset maximum code rate and the current coding code rate is not less than the preset maximum code rate.

In some embodiments, the first determining subunit may be specifically configured to:

if the uplink bandwidth is smaller than the preset maximum code rate, determining that the target network state belongs to a first network state;

and if the uplink bandwidth is not smaller than the preset code rate and the current coding code rate is smaller than the preset maximum code rate, determining that the target network state belongs to a second network state.

In some embodiments, the first adjustment unit 403 may include:

a sixth adjustment subunit, configured to adjust, if the target network state belongs to the first network state, an encoding code rate of the encoder based on an available code rate corresponding to the first network state;

and a seventh adjustment subunit, configured to adjust, if the target network state belongs to the second network state, the coding rate of the encoder based on the preset maximum code rate.

The embodiment of the application discloses a processing device for live video, which acquires an uplink bandwidth of a live video stream currently pushed to a server through an acquisition unit 401; the determining unit 402 determines a target network state according to the uplink bandwidth; the first adjusting unit 403 adjusts the encoding rate of the encoder corresponding to the live video stream based on the target network state if the target network state does not belong to a preset ideal network state; if the target network state belongs to the preset ideal network state, the second adjusting unit 404 monitors the encoding code rate of the encoder and/or the image frame quality parameter of the live video stream according to a preset time interval, and adjusts at least one encoding parameter of the encoder based on a monitoring result; the processing unit 405 processes the live video stream based on the adjusted encoder. Thus, the coding quality of the encoder can be improved.

Correspondingly, the embodiment of the application also provides electronic equipment which can be a server. As shown in fig. 5, fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device 500 includes a processor 501 having one or more processing cores, a memory 502 having one or more computer readable storage media, and a computer program stored on the memory 502 and executable on the processor. The processor 501 is electrically connected to the memory 502. It will be appreciated by those skilled in the art that the electronic device structure shown in the figures is not limiting of the electronic device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.

The processor 501 is a control center of the electronic device 500, connects various parts of the entire electronic device 500 using various interfaces and lines, and performs various functions of the electronic device 500 and processes data by running or loading software programs and/or modules stored in the memory 502, and invoking data stored in the memory 502, thereby performing overall listening to the electronic device 500.

In the embodiment of the present application, the processor 501 in the electronic device 500 loads the instructions corresponding to the processes of one or more application programs into the memory 502 according to the following steps, and the processor 501 executes the application programs stored in the memory 502, so as to implement various functions:

determining a target network state according to the uplink bandwidth;

if the target network state does not belong to the preset ideal network state, adjusting the coding rate of the encoder corresponding to the live video stream based on the target network state;

if the target network state belongs to a preset ideal network state, monitoring the coding rate of the encoder and/or the image frame quality parameter of the live video stream according to a preset time interval, and adjusting at least one coding parameter of the encoder based on a monitoring result;

and processing the live video stream based on the adjusted encoder.

In some embodiments, monitoring the encoding rate of the encoder and/or the image frame quality parameter of the live video stream at preset time intervals, and adjusting at least one encoding parameter of the encoder based on the monitoring result includes:

monitoring the coding rate of the coder based on a preset time interval;

In some embodiments, further comprising:

In some embodiments, listening to an image frame quality parameter of a live video stream based on a preset time interval comprises:

acquiring a picture group which is currently coded in a live video stream, wherein the picture group comprises at least one video frame;

In some embodiments, after the code rate of the encoder is increased, further comprising:

if the monitored image frame quality parameter is larger than the second preset image frame quality parameter, the coding rate of the coder is reduced, wherein the second preset image frame quality parameter is larger than the first preset image frame quality parameter.

In some embodiments, after the code rate of the encoder is reduced, further comprising:

In some embodiments, after adjusting the second encoding parameter of the encoder, further comprising:

In some embodiments, determining the target network state from the upstream bandwidth includes:

In some embodiments, if the uplink bandwidth is smaller than the preset maximum code rate, or if the uplink bandwidth is not smaller than the preset code rate and the current coding code rate is smaller than the preset maximum code rate, determining that the target network state does not belong to the preset ideal network state includes:

In some embodiments, if the target network state does not belong to the preset ideal network state, adjusting the encoding rate of the encoder corresponding to the live video stream based on the target network state includes:

The specific implementation of each operation above may be referred to the previous embodiments, and will not be described herein.

Optionally, as shown in fig. 5, the electronic device 500 further includes: a touch display screen 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. The processor 501 is electrically connected to the touch display 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507, respectively. It will be appreciated by those skilled in the art that the electronic device structure shown in fig. 5 is not limiting of the electronic device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.

The touch display screen 503 may be used to display a graphical user interface and receive operation instructions generated by a user acting on the graphical user interface. The touch display screen 503 may include a display panel and a touch panel. Wherein the display panel may be used to display information entered by a user or provided to a user as well as various graphical user interfaces of the electronic device, which may be composed of graphics, guidance information, icons, video, and any combination thereof. Alternatively, the Display panel may be configured in the form of a liquid crystal Display (LCD, liquiDCrystal Display), an Organic Light-Emitting Diode (OLED), or the like. The touch panel may be used to collect touch operations on or near the user (such as operations on or near the touch panel by the user using any suitable object or accessory such as a finger, stylus, etc.), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Alternatively, the touch panel may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch azimuth of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 501, and can receive commands from the processor 501 and execute them. The touch panel may overlay the display panel, and upon detection of a touch operation thereon or thereabout, the touch panel is passed to the processor 501 to determine the type of touch event, and the processor 501 then provides a corresponding visual output on the display panel based on the type of touch event. In the embodiment of the present application, the touch panel and the display panel may be integrated into the touch display screen 503 to implement the input and output functions. In some embodiments, however, the touch panel and the display panel may be implemented as two separate components to implement the input and output functions. I.e. the touch sensitive display 503 may also implement an input function as part of the input unit 506.

The radio frequency circuitry 504 may be used to transceive radio frequency signals to establish wireless communication with a network device or other electronic device via wireless communication.

The audio circuitry 505 may be used to provide an audio interface between a user and the electronic device through a speaker, microphone. The audio circuit 505 may transmit the received electrical signal after audio data conversion to a speaker, and convert the electrical signal into a sound signal for output by the speaker; on the other hand, the microphone converts the collected sound signals into electrical signals, which are received by the audio circuit 505 and converted into audio data, which are processed by the audio data output processor 501 for transmission to, for example, another electronic device via the radio frequency circuit 504, or which are output to the memory 502 for further processing. The audio circuit 505 may also include an ear bud jack to provide communication of the peripheral ear bud with the electronic device.

The input unit 506 may be used to receive input numbers, character information, or user characteristics (e.g., fingerprint, iris, facial information, etc.), as well as to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

The power supply 507 is used to power the various components of the electronic device 500. Alternatively, the power supply 507 may be logically connected to the processor 501 through a power management system, so as to implement functions of managing charging, discharging, and power consumption management through the power management system. The power supply 507 may also include one or more of any components, such as a direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

Although not shown in fig. 5, the electronic device 500 may further include a camera, a sensor, a wireless fidelity module, a bluetooth module, etc., which are not described herein.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.

As can be seen from the above, the electronic device provided in this embodiment may obtain the uplink bandwidth of the live video stream currently pushed to the server; determining a target network state according to the uplink bandwidth; if the target network state does not belong to the preset ideal network state, adjusting the coding rate of the encoder corresponding to the live video stream based on the target network state; if the target network state belongs to a preset ideal network state, monitoring the coding rate of the encoder and/or the image frame quality parameter of the live video stream according to a preset time interval, and adjusting at least one coding parameter of the encoder based on a monitoring result; and processing the live video stream based on the adjusted encoder.

Those of ordinary skill in the art will appreciate that all or a portion of the steps of the various methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.

To this end, embodiments of the present application provide a computer readable storage medium having stored therein a plurality of computer programs that can be loaded by a processor to perform steps in any of the live video processing methods provided by embodiments of the present application. For example, the computer program may perform the steps of:

determining a target network state according to the uplink bandwidth;

And processing the live video stream based on the adjusted encoder.

monitoring the coding rate of the coder based on a preset time interval;

In some embodiments, further comprising:

Wherein the computer-readable storage medium may comprise: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.

Because the computer program stored in the computer readable storage medium may execute the steps in any one of the live video processing methods provided in the embodiments of the present application, the beneficial effects that any one of the live video processing methods provided in the embodiments of the present application may be achieved, which is detailed in the previous embodiments and will not be described herein.

According to one aspect of the present application, there is also provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to cause the electronic device to perform the methods provided in the various alternative implementations of the embodiments described above.

The foregoing describes in detail a method, an apparatus, a storage medium, and an electronic device for processing live video provided in the embodiments of the present application, where specific examples are applied to illustrate principles and implementations of the present application, and the descriptions of the foregoing embodiments are only used to help understand the method and core ideas of the present application; meanwhile, those skilled in the art will have variations in the specific embodiments and application scope in light of the ideas of the present application, and the present description should not be construed as limiting the present application in view of the above.

Claims

1. A method for processing live video, which is applied to a host, the method comprising:

determining a target network state according to the uplink bandwidth;

and processing the live video stream based on the adjusted encoder.

2. The method according to claim 1, wherein the listening to the coding rate of the encoder and/or the image frame quality parameter of the live video stream at preset time intervals and adjusting at least one coding parameter of the encoder based on the listening result comprises:

monitoring the coding rate of the coder based on the preset time interval;

comparing the monitored coding code rate of the coder with a preset maximum code rate;

3. The method as recited in claim 2, further comprising:

if the coding code rate of the coder is not smaller than the preset maximum code rate, monitoring the image frame quality parameter of the live video stream based on the preset time interval;

and if the image frame quality parameter is smaller than the first preset image frame quality parameter, the coding rate of the coder is increased.

4. A method according to claim 3, wherein said listening to an image frame quality parameter of said live video stream based on said preset time interval comprises:

5. The method of claim 3, further comprising, after said increasing the coding rate of the encoder:

and if the monitored image frame quality parameter is larger than a second preset image frame quality parameter, the coding rate of the coder is reduced, wherein the second preset image frame quality parameter is larger than the first preset image frame quality parameter.

6. The method of claim 3, further comprising, after said increasing the coding rate of the encoder:

7. The method of claim 6, further comprising, after said adjusting the second encoding parameters of the encoder:

and if the monitored image frame quality parameter is larger than a second preset image frame quality parameter, the coding rate of the coder is regulated down, and the second coding parameter of the coder is regulated down.

8. The method of claim 1, wherein said determining a target network state from said upstream bandwidth comprises:

acquiring a current coding code rate of the coder and presetting a maximum code rate;

and if the uplink bandwidth is not smaller than a preset maximum code rate and the current coding code rate is not smaller than the preset maximum code rate, determining that the target network state belongs to the preset ideal network state.

9. The method of claim 8, wherein the determining that the target network state does not belong to the preset ideal network state if the upstream bandwidth is less than the preset maximum code rate or the upstream bandwidth is not less than the preset code rate and the current coding code rate is less than the preset maximum code rate comprises:

10. The method of claim 9, wherein if the target network state does not belong to a preset ideal network state, adjusting the encoding rate of the encoder corresponding to the live video stream based on the target network state comprises:

if the target network state belongs to the first network state, adjusting the coding rate of the coder based on the available code rate corresponding to the first network state;

11. A device for processing live video, the device comprising:

12. An electronic device comprising a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method of any of claims 1-10.

13. A computer readable storage medium, comprising a computer program for causing an electronic device to perform the method of any one of claims 1-10 when the computer program is run on the electronic device.

14. A computer program product comprising a computer program, the computer program being stored on a computer readable storage medium; when the computer program is read from the computer readable storage medium by a processor of an electronic device, the processor executes the computer program, causing the electronic device to perform the steps of the method of any one of claims 1-10.