CN113194324B - Video frame image quality enhancement method, live broadcast server and electronic equipment - Google Patents
Video frame image quality enhancement method, live broadcast server and electronic equipment Download PDFInfo
- Publication number
- CN113194324B CN113194324B CN202110462217.4A CN202110462217A CN113194324B CN 113194324 B CN113194324 B CN 113194324B CN 202110462217 A CN202110462217 A CN 202110462217A CN 113194324 B CN113194324 B CN 113194324B
- Authority
- CN
- China
- Prior art keywords
- frame
- video
- video frame
- motion
- intensity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 230000033001 locomotion Effects 0.000 claims description 47
- 230000002708 enhancing effect Effects 0.000 claims description 6
- 230000002596 correlated effect Effects 0.000 claims 1
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 239000013598 vector Substances 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000006835 compression Effects 0.000 description 3
- 238000007906 compression Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
- H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
- H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The application provides a video frame image quality enhancement method, a live broadcast server and electronic equipment, wherein the method comprises the following steps: determining the importance of a video frame at least according to the frame type of the video frame; determining at least one target frame from a plurality of video frames based on the importance of the video frames; and performing image quality enhancement processing on the target frame. By the method, only the relatively important video frames in the video can be selected for image quality enhancement processing, and all the video frames do not need to be processed, so that the operation load of the equipment can be effectively reduced, the processing time is shortened, and the user experience is improved.
Description
Technical Field
The application relates to the technical field of video processing, in particular to a video frame image quality enhancement method, a live broadcast server and electronic equipment.
Background
In order to reduce the amount of data transmission, video often needs to be compressed before transmission. The picture of the compressed video is distorted, and the decoded video usually needs to be subjected to image quality enhancement processing to improve the picture quality. The image quality enhancement processing has high requirements on hardware configuration of equipment, and for equipment with low computing capability, the image quality enhancement processing needs to take a long time, so that the user experience is influenced.
Disclosure of Invention
The application provides a video frame image quality enhancement method, a live broadcast server and an electronic device, which can effectively reduce the operation load when the device performs image quality enhancement processing.
According to a first aspect of an embodiment of the present application, a method for enhancing image quality of a video frame is provided, the method comprising:
determining the importance of a video frame according to at least the frame type of the video frame;
determining at least one target frame from a plurality of video frames based on the importance of the video frames;
and performing image quality enhancement processing on the target frame.
According to a second aspect of the embodiments of the present application, a live broadcast server is provided, where the live broadcast server includes:
the importance determining module is used for determining the importance of the video frame at least according to the frame type of the video frame;
the target frame determining module is used for determining at least one target frame from a plurality of video frames based on the importance of the video frames;
and the image quality enhancement module is used for carrying out image quality enhancement processing on the target frame.
According to a third aspect of embodiments of the present application, there is provided an electronic apparatus, including:
a processor;
a memory for storing processor-executable instructions;
Wherein the processor is configured to:
determining the importance of a video frame according to at least the frame type of the video frame;
determining at least one target frame from a plurality of video frames based on the importance of the video frames;
and performing image quality enhancement processing on the target frame.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
the application provides a video frame image quality enhancement method, a live broadcast server and electronic equipment, wherein a target frame is selected from a plurality of video frames according to the importance of the video frames for image quality enhancement processing, wherein the importance of the video frames is determined at least according to the frame types of the video frames.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
Fig. 1 is a flowchart illustrating a method for enhancing image quality of a video frame according to an exemplary embodiment of the present disclosure.
Fig. 2 is a schematic diagram of an L1 distance calculation method shown in the present application according to an exemplary embodiment.
FIG. 3 is a schematic diagram of a rank position calculation method shown herein according to an example embodiment.
Fig. 4 is a flowchart illustrating a method for enhancing image quality of a video frame according to another exemplary embodiment of the present application.
Fig. 5 is an application scenario of a video frame quality enhancement method according to an exemplary embodiment of the present application.
Fig. 6 is a flowchart illustrating a method for enhancing image quality of a video frame according to another exemplary embodiment of the present application.
Fig. 7 is a block diagram of a live service shown in the present application according to an example embodiment.
FIG. 8 is a block diagram of hardware of an electronic device according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
Next, examples of the present application will be described in detail.
With the popularization of large-size and high-resolution display terminals, the image quality requirement of video images is higher and higher. However, the video is limited in transmission bandwidth or storage space, and often needs to be compressed when being transmitted or stored. The lossy compressed video pictures are distorted, and image quality enhancement processing is required before playing and using, so as to improve the picture quality. The image quality enhancement processing generally includes a series of processing such as noise reduction, super-resolution, contrast enhancement, histogram equalization, and sharpening. In the related art, it is common to perform an undifferentiated image quality enhancement process for each frame of a video. However, the requirement of the image quality enhancement processing on the hardware configuration of the device is high, and in addition, the image quality enhancement processing needs to be performed on each frame of the video without difference, which needs to take a long time to process and operate for the device with weak operation capability, thereby greatly influencing the user experience. In order to reduce the dependence of image quality enhancement on hardware configuration, the application provides a video frame image quality enhancement method, which can be applied to a scene in which image quality enhancement of a video frame is performed by a device before a compressed and stored video is used or played, and can also be applied to a scene in which an image quality enhancement processing is performed by extracting the video frame from a video stream after a local terminal receives the video stream which is compressed and transmitted by an opposite terminal, and then the video scene is used or played. The method comprises the steps as shown in fig. 1:
Step 110: determining the importance of a video frame according to at least the frame type of the video frame;
step 120: determining at least one target frame from a plurality of video frames based on the importance of the video frames;
step 130: and performing image quality enhancement processing on the target frame.
The method comprises the steps of determining the importance of video frames according to the frame types of the video frames to represent the relative importance of the video frames, then determining target frames from a plurality of video frames according to the importance of the video frames, and carrying out image quality enhancement processing on the target frames. Therefore, compared with the technical scheme of performing indiscriminate image quality enhancement processing on each video frame in the related technology, the video frame image quality enhancement method provided by the application can only select relatively important video frames in the video to perform image quality enhancement processing, and does not need to process all the video frames, so that on one hand, the operation load of equipment can be effectively reduced, the processing time is shortened, and on the other hand, due to the visual persistence, only the relatively important video frames are subjected to image quality enhancement, and the visual experience of a user is not influenced. Further, in some embodiments, a specific enhancement frame rate may be set according to actual needs, and is used to specify the number of target frames selected from video frames played per second, that is, the number of frames with enhanced image quality. For example, in a video with a playback frame rate or a display frame rate of 60fps, a specified enhancement frame rate of 20fps may be set to indicate that the video plays 60 video frames per second, and 20 frames of the video are subjected to image quality enhancement processing.
Video frames can be classified into various types and there are various classification methods, for example, video frames are classified according to video coding sequences, and video frames can be classified into I-frames (Intra-coded Picture), P-frames (Predictive-coded Picture), and B-frames (bidirectional Predictive coded Picture). Classified from a video codec perspective, video frames can be classified into forward reference frames, backward reference frames, and non-reference frames. From an animation perspective, video frames can also be classified as key frames, normal frames, and blank key frames.
There are many different classification methods for video frames, and the relative importance of different types of video frames varies among different classification methods. For example, from the viewpoint of video encoding and decoding, a complete video can be restored only after an I frame is obtained, and a non-reference frame needs to be superimposed with a reference frame to decode a final picture. From the viewpoint of animation production, the key frame is a basic unit constituting the animation, and the animation cannot be produced without the key frame. The importance of a video frame can be determined according to the frame type. In some embodiments, the frame type may include an I-frame, which may have a relatively greater importance than other types of video frames because I-frames have a low compression efficiency, retain relatively complete picture content, and may reconstruct a complete image when decoded, and therefore may determine the importance of the video frame according to the frame type. Determining the target frame from the plurality of video frames according to the importance degree may include determining a video frame with a frame type of I frame as the target frame. And then carrying out image quality enhancement processing on the I frame.
Further, in some embodiments, the frame types may also include reference frames or non-reference frames. The reference frame refers to a frame which needs to be referred to during IPB coding, the non-reference frame is a frame which does not need to be referred to, and whether the video frame is a reference frame or a non-reference frame can be judged according to a specific identification bit in a video frame code stream. And both reference and non-reference frames may include B frames and P frames. When the frame type is a reference frame or a non-reference frame, the importance of the video frame can be determined according to the intensity of the motion of the video frame and the instantaneous frame rate; wherein, the intensity of motion is the intensity of motion of the video frame compared with the previous target frame.
The intensity of motion of a video frame refers to the degree of change in picture content compared to the previous frame. The lower the intensity of motion, the more similar the picture content between two video frames, and vice versa. The intensity of motion of the video frame compared with the previous target frame is the difference between the picture content of the current frame and the picture content of the previous target frame. The previous target frame may be separated from the current frame by several frames.
As described above, the frame type may reflect the relative importance between video frames. The intensity of the motion may reflect the similarity of the picture content between the video frames. In some embodiments, the greater the difference in picture content between a video frame and a previous target frame, i.e., the greater the intensity of motion of the video frame compared to the previous target frame, the greater the importance of the video frame. In addition, the instantaneous frame rate reflects how many target frames are currently selected for image quality enhancement, which can reflect the speed of selection, so that the importance degree of the video frames can be determined by considering the frame type, the intensity degree of motion and the instantaneous frame rate.
In some embodiments, the weight value of the video frame may be determined according to the intensity of the motion and the frame type; the motion intensity and the weight value are in a positive correlation relationship, that is, the higher the difference between the picture contents of the video frame and the previous target frame is, the more dissimilar the picture contents are, and the higher the weight value of the video frame is.
In some embodiments, different parameters may be introduced to characterize the frame type, the severity of the motion, and the weight value of the video frame. For example, the different types of video frames are characterized by a parameter Level, for example, when a video frame is an I frame, the Level is assigned to be 0; when the video frame is a reference frame, the Level value is 1; when the video frame is a non-reference frame, the Level value is 2.
In addition, the L1 distance may be used to characterize the severity of motion of a video frame compared to a previous target frame. The L1 distance, manhattan distance, is the sum of the distances in each dimension of two n-dimensional vectors. As shown in fig. 2, if a frame of video frame is taken as an n-dimensional vector and a pixel is taken as a dimension of the vector, the L1 distance L1dist of the video frame from the previous target frame is the sum of absolute values of differences between pixel values of the pixel in the video frame and the previous target frame. It can be seen that the larger the L1 distance is, the larger the difference between the picture contents of the video frame and the previous target frame is, i.e. the larger the motion intensity is. In addition to the L1 distance, the intensity of motion can also be characterized by the L2 distance, the euclidean distance. Similarly, the L2 distance of a video frame compared to a previous target frame is the sum of the squares and the re-root of the difference between the pixel values of each pixel point in the video frame and the previous target frame. Likewise, the greater the L2 distance, the greater the intensity of the motion. Those skilled in the art can select other parameters to characterize the intensity of the motion according to actual needs, including motion vectors of video frames, the number of macroblocks in the frames, and the like, and this application does not describe in detail here, nor does it limit the parameters that characterize the intensity of the motion.
When the frame type is represented by a Level with different assignments and the motion intensity compared with the previous target frame is represented by a distance L1dist of L1, the weight value weight of the video frame can be represented as:
weight＝(10-Level)*100+l1dist*100
in addition, a person skilled in the art may set the values of 10 and 100 in the above expression to other values according to actual needs, and the application is not limited herein. According to the above expression, the Level and the weight value of the video frame are in a negative correlation, and the L1 distance L1dist and the weight value of the video frame are in a positive correlation.
Further, in some embodiments, the frame type may be weighted more heavily than the motion intensity. That is, the influence of the change of the frame type Level on the weight value weight is greater than the influence of the motion intensity l1dist on the weight value weight. For example, in the above expression of weight value weight, the weight of Level to weight may be set to be greater than the weight of l1dist to weight. Specifically, the L1 distance may be normalized, that is, L1dist is the result of the normalized L1 distance, so that the value range of L1dist is between 0 and 1.
After the weight value of the video frame is determined, the importance of the video frame can be determined according to the intensity of the movement, the ranking position of the weight value and the instantaneous frame rate; the ranking position is the ranking position of the video frame in all target frames in a preset time period according to the weight value. As shown in fig. 3, in determining the importance of the video frame D, the preset time period may be the first 1 second of the video frame D, where the first 1 second includes several video frames, and the target frame A, B, C is shared, and their weight values are a, b, and c, respectively, and the weight value of the video frame D is D. By comparing the sizes of a, b, c and D, the ranking positions of the weighted values in all target frames of the video frame D in the previous second can be obtained. The instantaneous frame rate is an enhanced frame rate, and may be a fixed value or may vary.
In some embodiments, the determining the importance of the video frame according to the intensity of the motion, the ranking position of the weight value, and the instantaneous frame rate may include determining that the video frame is the target frame when a parameter representing the intensity of the motion is greater than a preset threshold, the ranking position is after a specified position, and the instantaneous frame rate is less than a preset first frame rate threshold. Wherein the preset threshold may be 5 × 10 -5 When the normalized L1 distance is greater than the preset threshold, it indicates that the difference between the picture contents of the video frame and the previous target frame is larger. When the enhancement frame rate is a fixed value, namely the video frames with the same number are selected per second for image quality enhancement processing, a ranking position can be designated; when the enhancement frame rate is not fixed, that is, different numbers of video frames are selected per second for image quality enhancement processing, the ranking position after the designated position may mean that the number percentage of the target frames with the weight values smaller than that of the video frames is smaller than the preset number percentage. For example, the preset number percentage may be 20%, and the ranking position after the designated position indicates that the weight value of less than 20% of the target frames is less than the weight value of the video frames, which are ranked later. When the parameter value representing the intensity of the motion is larger than the preset threshold, the ranking position of the weight value is behind the designated position, and the instantaneous frame rate is smaller than the preset first frame rate threshold, it is indicated that the difference of the picture content of the video frame is larger than that of the previous target frame, but the weight value is ranked behind, but the video frame can be determined as the target frame because the instantaneous frame rate does not reach the preset first frame rate and the video frame selection speed is slower.
In addition to the above, in some embodiments, the determining the importance of the video frame according to the intensity of the motion, the ranking position of the weight value, and the instantaneous frame rate may further include determining that the video frame is a target frame when a parameter representing the intensity of the motion is greater than a preset threshold, the ranking position is before a specified position, and the instantaneous frame rate is less than a second frame rate threshold; wherein the second frame rate threshold is determined according to the ranking position. Similarly, when the enhanced frame rate is a fixed value, a ranking position can be designated; when the enhancement frame rate is not fixed, the ranking position before the designated position may mean that the number percentage of the target frames having the weight value smaller than that of the video frames is greater than a preset number percentage. For example, the preset number percentage may be 20%, and the ranking position before the designated position indicates that the weight value of more than 20% of the target frames is less than the weight value of the video frames, which are ranked higher. When the parameter value representing the intensity of the motion is larger than the preset threshold value and the ranking position of the weight value is before the designated position, the difference of the picture content of the video frame is larger than that of the previous target frame, and the weight value is ranked to be ahead. At this time, the second preset frame rate threshold may be determined according to the ranking position. For example, the higher the ranking position is, the larger the second preset frame rate threshold is, which is equivalent to increasing the probability that the video frame with the higher ranking position is selected as the target frame.
And after the target frame is determined, performing image quality enhancement processing on the target frame. In some embodiments, the image quality enhancement processing can be performed on other video frames which are not determined as the target frame, but the enhancement precision and intensity of the other video frames are lower than those of the target frame.
According to the video frame image quality enhancement method, the target frame is selected from the video frames according to the importance of the video frames for image quality enhancement processing, wherein the importance of the video frames is determined at least according to the frame types of the video frames.
In addition, the present application also provides a method for enhancing image quality of a video frame, comprising the steps shown in fig. 4:
step 410: judging the type of the current video frame;
if the current video frame type is i frame, the Level value is assigned to 0, and step 470 is directly executed; if the current video frame type is a reference frame, the Level is assigned to be 1; for non-reference frames, the Level value is 0, and step 420 is performed.
Step 420: calculating a normalized L1 distance L1dist of the video frame compared to a previous target frame;
step 430: calculating the weight value weight of the video frame as (10-Level) 100+ l1dist 100;
step 440: obtaining ranking rp of the video frames in all target frames within the first 1 second according to the weight values;
step 450: acquiring an instantaneous frame rate fp;
wherein, the steps 450 and the steps 410-440 are not executed in an absolute time sequence, the steps 450 may be executed first, and then the steps 410-440 are executed, or vice versa, or the steps 450 may be executed during the steps 410-440, which is not limited herein.
Condition 1: l1dist>5×10 -5 ；rp≤0.2；fp≤maxFps
Where maxFps may be the specified enhancement frame rate described above. rp < 0.2 indicates that less than 20% of all object frames in the previous 1 second have a weight value weight less than that of the video frames. Condition 1 indicates that L1 distance L1dist is greater than 5 × 10 -5 The weight value weight of less than 20% of the target frames is less than that of the video frames, and the instantaneous frame rate fp is less than the specified enhancement frame rate, that is, the difference between the picture content of the video frame and the picture content of the previous target frame is larger, but the weight value is ranked later, and the instantaneous frame rate does not reach the specified enhancement frame rate.
Condition 2: l1dist > 5X 10-5; rp > 0.2; fp is not more than maxFps x [ (rp-0.2)/1.2+1]
Where rp >0.2 indicates that more than 20% of all target frames in the previous 1 second have a weight value weight less than that of the video frames. And "maxFps × [ (rp-0.2)/1.2+1 ]" is the second preset frame rate threshold mentioned above, and it can be seen that the second preset frame rate threshold is determined according to the ranking rp, and when rp >0.2, rp is larger, that is, the more target frames have weight values smaller than that of the video frames, the larger the second preset frame rate threshold is. Furthermore, since [ (rp-0.2)/1.2+1] >1, the second preset frame rate threshold is larger than maxFps, i.e. the enhancement frame rate is specified. In other words, when the picture content of the video frame is greatly different from that of the previous target frame and the weight value of the video frame is ranked higher, the frame rate threshold may be appropriately increased, and the degree of the increase is determined according to the ranked higher degree. The reason why two different frame rate thresholds are set for the condition 1 and the condition 2 is that if the same frame rate threshold is set, in some cases, in a continuous video frame, if the first several video frames all satisfy the condition 1 and are determined as the target frame, the instantaneous frame rate fp is maxFps, and if there are video frames satisfying the condition 2 after the several video frames, the video frame satisfying the condition 2 cannot be selected. Obviously, the video frames satisfying the condition 2 are more suitable for image quality enhancement processing than the video frames satisfying the condition 1 because the weighted value is ranked more ahead than the weighted value of the video frames satisfying the condition 1. In order to solve the above problem, a first frame rate threshold and a second frame rate threshold may be set for condition 1 and condition 2, respectively, and the second frame rate threshold is greater than the first frame rate threshold, and the second frame rate threshold is determined according to the ranking rp of the weight value. In addition, the maximum value of the second frame rate threshold may also be appropriately adjusted according to the computing capability of the device. As in the above expression maxFps × [ (rp-0.2)/1.2+1], it is apparent that the second frame rate maximum value is a calculated value when rp is 1, which is about 1.67 maxFps. When the computing power of the device is poor, a factor greater than 1.2 may be selected such that the maximum value of the second frame rate threshold is reduced, and vice versa.
Step 460: judging whether the condition 1 or the condition 2 is met;
if so, go to step 470, otherwise go to step 480.
Step 470: determining the video frame as a target frame, and performing image quality enhancement processing; step 480 is then performed.
Step 480: switch to the next video frame and return to step 410.
In some embodiments, after determining that the video frame satisfies condition 1 or condition 2 in step 460, step 470 may not be performed, but the video frame satisfying condition 1 or condition 2 is buffered in a storage space or marked, and then step 480 is performed to switch to the next video frame to determine whether the next video frame is the target frame. And sequencing the video frames cached in the storage space or the marked video frames until all the video frames or all the video frames within a preset period of time are subjected to the steps, preferentially determining the video frames meeting the condition 2 as target frames, and selecting at least part of the video frames from the video frames meeting the condition 1 as the target frames according to the ranking position when the frame rate does not reach the maximum frame rate until the frame rate reaches the maximum frame rate or the video frames meeting the condition 1 are determined as the target frames.
By the method, only the relatively important video frames in the video can be selected for image quality enhancement processing, and all the video frames do not need to be processed, so that the operation load of the equipment can be effectively reduced, the processing time is shortened, and the user experience is improved.
Referring to fig. 5, an application scenario of the video frame image quality enhancement method provided in the present application is shown. As shown in fig. 5, when the anchor is playing live, the anchor terminal 510 uploads the live video stream to the live server 520 via the network, and then the live server 520 distributes the live video stream to the viewer terminals 530. When the viewer wants to interact with the anchor, for example, send some interesting videos to the anchor, the viewer side 530 can also upload the video stream to the live service side 520, and the live service side 520 forwards the video stream to the anchor side 510. In some scenarios, when the anchor end 510 is playing a game live, the anchor end 510 needs to upload an anchor game interface in addition to a video stream containing an anchor portrait. On one hand, due to the limitation of the uploading bandwidth, the video stream uploaded by the anchor terminal 510 may need to undergo lossy compression, thereby damaging the video picture; on the other hand, if the hardware configuration of the device where the anchor terminal 510 is located is low, and the game picture quality cannot be configured to a higher level, the image quality of the game interface may be damaged more seriously after lossy compression. Therefore, the video frame image quality enhancement method provided by the application can be applied to the live broadcast server 520, and the video frames are extracted from the video stream sent by the client. Which may include a broadcaster end 510 and a viewer end 530. The live broadcast server 520 performs image quality enhancement processing on the uploaded video stream, and then distributes the video stream to each viewer 530. Of course, the video frame image quality enhancement method provided by the present application may also be applied to the client 530, and different first and second frame rate thresholds are set by each client according to the computing capability of the device where the client is located.
When the video frame image quality enhancement method provided by the application is applied to a live scene, the problem of real-time performance of the video frame image quality enhancement method also needs to be considered. As described above, the image quality enhancement processing often takes a long time, and this is not suitable for a live scene requiring high real-time performance. According to the video frame image quality enhancement method, on one hand, only the relatively important video frames in the video are selected to be subjected to image quality enhancement processing, and all the video frames do not need to be processed, so that the processing time can be effectively shortened. On the other hand, in a conventional thinking manner, when a relatively important video frame needs to be selected from a plurality of video frames, the relative importance of the video frames in all the video frames needs to be distinguished one by one, that is, the current video frame needs to refer to the relative importance of the preceding and following video frames to determine whether the current video frame is a target frame. When the method and the device determine whether the video frame is the target frame, only the target frame in a time period needs to be referred to forwards, and the subsequent video frame does not need to be referred to, so that the conventional thinking mode is broken through. Because the current video frame does not need to refer to the later video frame, the current video frame can be processed in real time, and delay is not introduced, so that the method is well suitable for scenes with high real-time requirements such as live broadcast and the like.
When the video frame image quality enhancement method provided by the application is applied to a live scene, the method comprises the following steps as shown in fig. 6:
step 610: the anchor terminal 10 uploads a video stream to the live broadcast server 20;
step 620: the live broadcast server 20 extracts video frames from the video stream, and determines the importance of the video frames at least according to the frame types;
step 630: the live broadcast server 20 determines at least one target frame from the video frames based on the importance of the video frames;
step 640: the live broadcast server 20 performs image quality enhancement processing on the target frame;
step 650: the live broadcast server 20 transmits the video stream after the image quality enhancement processing to the viewer 30.
For the specific implementation of the above steps, refer to the above embodiments, which are not described herein again.
Based on the video frame image quality enhancement method according to any of the embodiments, the present application further provides a live broadcast server 700 shown in fig. 7, including:
an importance determination module 710 for determining importance of a video frame according to at least a frame type of the video frame;
a target frame determining module 720, configured to determine at least one target frame from the video frames based on the importance of the video frames;
the image quality enhancement module 730 is configured to perform image quality enhancement processing on the target frame.
The implementation process of the functions and actions of each module in the above device is detailed in the implementation process of the corresponding steps in the above method, and is not described herein again.
Based on the video frame image quality enhancement method according to any of the embodiments, the present application further provides a schematic structural diagram of an electronic device as shown in fig. 8. As shown in fig. 8, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, but may also include hardware required for other services. The processor reads a corresponding computer program from the nonvolatile memory into the memory and runs the computer program to implement the video frame image quality enhancement method according to any of the embodiments.
The foregoing description has been directed to specific embodiments of this application. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
Claims (9)
1. A method for enhancing image quality of video frames, the method comprising:
if the frame type of the current video frame is a reference frame or a non-reference frame, determining the importance of the video frame according to the frame type, the motion intensity and the instantaneous frame rate of the current video frame; the intensity of the motion is the intensity of the motion of the video frame compared with the previous target frame; the instantaneous frame rate is the number of the selected target frames in the video frames played per second;
determining whether the video frame is a target frame based on the importance of the video frame;
and if so, performing image quality enhancement processing on the video frame.
2. The method of claim 1, further comprising:
and if the frame type of the current video frame is an I frame, determining that the video frame is a target frame, and performing image quality enhancement processing on the video frame.
3. The method according to claim 1, wherein said determining the importance of the video frame according to the frame type, the intensity of motion and the instantaneous frame rate of the current video frame comprises the steps of:
determining a weight value of the video frame according to the motion intensity and the frame type; wherein the intensity of the motion is positively correlated with the weight;
determining the importance of the video frame according to the intensity of the motion, the ranking position of the weight value and the instantaneous frame rate; and the ranking position is the ranking position of the video frame in all target frames within a preset time period according to the weight value.
4. The method of claim 3, wherein determining the importance of the video frame according to the intensity of the motion, the ranking position of the weight value, and the instantaneous frame rate comprises:
and when the parameter representing the intensity degree of the motion is greater than a preset threshold value, the ranking position is behind the designated position, and the instantaneous frame rate is less than a preset first frame rate threshold value, determining that the video frame is a target frame.
5. The method of claim 3, wherein determining the importance of the video frame according to the intensity of the motion, the ranking position of the weight value, and the instantaneous frame rate comprises:
when the parameter representing the intensity degree of the motion is larger than a preset threshold value, the ranking position is in front of the designated position, and the instantaneous frame rate is smaller than a second frame rate threshold value, determining that the video frame is a target frame; wherein the second frame rate threshold is determined according to the ranking position.
6. The method of claim 3, wherein the weighting value for the frame type is greater than the weighting value for the intensity of the motion.
7. The method according to claim 1, wherein the method is applied to a live service, and the video frames are extracted from a video stream sent by a client.
8. The utility model provides a live broadcast server, its characterized in that, live broadcast server includes:
the importance degree determining module is used for determining the importance degree of the video frame according to the frame type, the motion intensity and the instantaneous frame rate of the current video frame if the frame type of the current video frame is a reference frame or a non-reference frame; the intensity of the motion is the intensity of the motion of the video frame compared with the previous target frame; the instantaneous frame rate is the number of the selected target frames in the video frames played per second;
A target frame determination module, configured to determine whether the video frame is a target frame based on the importance of the video frame;
and the image quality enhancement module is used for carrying out image quality enhancement processing on the video frame if the video frame is the video frame.
9. An electronic device, characterized in that the electronic device comprises:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
if the frame type of the current video frame is a reference frame or a non-reference frame, determining the importance of the video frame according to the frame type, the motion intensity and the instantaneous frame rate of the current video frame; the intensity of the motion is the intensity of the motion of the video frame compared with the previous target frame; the instantaneous frame rate is the number of the selected target frames in the video frames played per second;
determining whether the video frame is a target frame based on the importance of the video frame;
and if so, performing image quality enhancement processing on the video frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110462217.4A CN113194324B (en) | 2021-04-27 | 2021-04-27 | Video frame image quality enhancement method, live broadcast server and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110462217.4A CN113194324B (en) | 2021-04-27 | 2021-04-27 | Video frame image quality enhancement method, live broadcast server and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113194324A CN113194324A (en) | 2021-07-30 |
| CN113194324B true CN113194324B (en) | 2022-07-29 |
Family
ID=76979704
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110462217.4A Active CN113194324B (en) | 2021-04-27 | 2021-04-27 | Video frame image quality enhancement method, live broadcast server and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113194324B (en) |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110830788A (en) * | 2018-08-07 | 2020-02-21 | 北京优酷科技有限公司 | Method and device for detecting black screen image |
| CN109729405B (en) * | 2018-11-27 | 2021-11-16 | Oppo广东移动通信有限公司 | Video processing method and device, electronic equipment and storage medium |
| CN109379624B (en) * | 2018-11-27 | 2021-03-02 | Oppo广东移动通信有限公司 | Video processing method and device, electronic equipment and storage medium |
| CN110738611B (en) * | 2019-09-20 | 2022-08-16 | 网宿科技股份有限公司 | Video image quality enhancement method, system and equipment |
| CN110781740B (en) * | 2019-09-20 | 2023-04-07 | 网宿科技股份有限公司 | Video image quality identification method, system and equipment |
| CN111031346B (en) * | 2019-10-28 | 2021-11-02 | 网宿科技股份有限公司 | Method and device for enhancing video image quality |
| CN111629262B (en) * | 2020-05-08 | 2022-04-12 | Oppo广东移动通信有限公司 | Video image processing method and device, electronic equipment and storage medium |
-
2021
- 2021-04-27 CN CN202110462217.4A patent/CN113194324B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN113194324A (en) | 2021-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8036263B2 (en) | Selecting key frames from video frames | |
| US7949053B2 (en) | Method and assembly for video encoding, the video encoding including texture analysis and texture synthesis, and corresponding computer program and corresponding computer-readable storage medium | |
| US6618507B1 (en) | Methods of feature extraction of video sequences | |
| JP5555221B2 (en) | Method and apparatus for adaptive color model parameter estimation of object of interest | |
| US9729870B2 (en) | Video coding efficiency with camera metadata | |
| EP2727344B1 (en) | Frame encoding selection based on frame similarities and visual quality and interests | |
| KR101409826B1 (en) | Method for motion prediction using adaptive search range | |
| CN112351278B (en) | Video encoding method and device and video decoding method and device | |
| CN113194324B (en) | Video frame image quality enhancement method, live broadcast server and electronic equipment | |
| JP4620126B2 (en) | Video identification device | |
| CN116489385A (en) | Video encoding method, decoding method, device, electronic equipment and storage medium | |
| CN112714336B (en) | Video segmentation method and device, electronic equipment and computer readable storage medium | |
| CN113115075B (en) | Method, device, equipment and storage medium for enhancing video image quality | |
| CN114422792B (en) | Video image compression method, device, equipment and storage medium | |
| US10856002B2 (en) | Method, device and system for method of encoding a sequence of frames in a video stream | |
| CN112533024A (en) | Face video processing method and device and storage medium | |
| CN112954318A (en) | Data coding method and device | |
| US7706440B2 (en) | Method for reducing bit rate requirements for encoding multimedia data | |
| CN111050175A (en) | Method and apparatus for video encoding | |
| CN116708933B (en) | Video coding method and device | |
| Komamaki et al. | Efficient data reduction based on predicted regions of interest for game videos | |
| CN117896552A (en) | Video conference processing method, video conference system and related device | |
| CN113766232A (en) | Encoding and decoding methods and devices, electronic equipment and storage medium | |
| CN116233447A (en) | Image group size determining method, electronic device and storage medium | |
| CN113473137A (en) | Encoding method, terminal, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |