WO2024152893A1 - 视频显示控制方法及其装置、设备、介质、产品 - Google Patents

视频显示控制方法及其装置、设备、介质、产品 Download PDF

Info

Publication number
WO2024152893A1
WO2024152893A1 PCT/CN2024/070219 CN2024070219W WO2024152893A1 WO 2024152893 A1 WO2024152893 A1 WO 2024152893A1 CN 2024070219 W CN2024070219 W CN 2024070219W WO 2024152893 A1 WO2024152893 A1 WO 2024152893A1
Authority
WO
WIPO (PCT)
Prior art keywords
image frame
resolution
interface
scaling
interface type
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/070219
Other languages
English (en)
French (fr)
Inventor
包佳晶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bigo Technology Pte Ltd
Original Assignee
Bigo Technology Pte Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bigo Technology Pte Ltd filed Critical Bigo Technology Pte Ltd
Publication of WO2024152893A1 publication Critical patent/WO2024152893A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the present application relates to the field of live video processing technology, and in particular to a video display control method and its device, equipment, medium, and product.
  • the live video stream transmitted through the Internet will be encoded and transmitted.
  • the display quality of the terminal device at the receiving end is taken into consideration.
  • a set of encoding control parameters are determined by the associated changes in multiple dimensions such as bandwidth, quality indicators, bit rate, and resolution. These encoding control parameters are used to ensure the quality of the live video stream obtained by the terminal device, so that high-quality display effects can be obtained when the live video stream is played.
  • the corresponding scaling algorithms are mainly divided into two categories: software scaling and hardware scaling.
  • hardware scaling mainly depends on the hardware conditions of the terminal device, which can save device power consumption and CPU computing power overhead, but the image effect after scaling is relatively general; there are many types of software scaling, and different software scaling algorithms have different requirements for computing power and different effects.
  • software algorithms with high computing power requirements have better image effects after scaling.
  • the matching of the scaling algorithm can also determine the display effect of the live video stream on the terminal, but traditional technologies often ignore the key response plan for this technology, resulting in that although the encoding control of the live video stream is implemented on the server side, the image display effect of the live video stream when displayed in different terminal devices varies greatly.
  • the purpose of the present application is to solve the above-mentioned problems and to provide a video display control method and its corresponding device, equipment, non-volatile readable storage medium, and computer program product.
  • a video display control method comprising the following steps:
  • the live video stream includes a plurality of image frames and target quality indicators thereof;
  • an interface type corresponding to a scaling operation of the image frame according to an encoding resolution of the image frame and a corresponding target quality index thereof, wherein the interface type is a hardware interface type or a software interface type;
  • the zoom interface corresponding to the interface type is called to perform a zoom operation on the image frame and then rendered and displayed in the video playback window of the live broadcast room interface.
  • a video display control device comprising:
  • a data acquisition module configured to acquire a live video stream, wherein the live video stream includes a plurality of image frames and target quality indicators thereof;
  • An interface matching module configured to determine an interface type corresponding to the scaling operation of the image frame according to the encoding resolution of the image frame and its corresponding target quality index, wherein the interface type is a hardware interface type or a software interface type;
  • the zoom display module is configured to call the zoom interface corresponding to the interface type to perform a zoom operation on the image frame and then render and display it in the video playback window of the live broadcast room interface.
  • a video display control device comprising a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the video display control method described in the present application.
  • a non-volatile readable storage medium which stores a computer program implemented according to the video display control method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.
  • a computer program product including a computer program/instruction, wherein when the computer program/instruction is executed by a processor, the steps of the method described in any one of the embodiments of the present application are implemented.
  • the present application synchronously obtains the target quality index corresponding to each image frame in the live video stream when acquiring the live video stream, and then selects the interface type corresponding to the scaling operation according to the encoding resolution and target quality index of the image frame, calls the scaling interface corresponding to the interface type to implement the scaling operation on the image frame, and then renders and displays it in the video playback window of the live broadcast room interface, taking the target quality index of the image frame given when pushing the live video stream as the standard, matches the interface type based on the target quality index, and implements the scaling operation on the image frame by the scaling interface corresponding to the interface type, ensuring that the image effect after the image frame is scaled can meet the standard established by the target quality index to the greatest extent, so that when the live video stream is played in the live broadcast room interface, a relatively high-quality and stable picture effect can be obtained, which can improve the user experience of online live broadcast users.
  • FIG1 is a schematic diagram of a network architecture corresponding to the network live broadcast environment used in the present application.
  • FIG2 is a schematic diagram of the principle of the technical framework for encoding a live video stream in the present application
  • FIG3 is a flow chart of an embodiment of a video display control method of the present application.
  • FIG4 is a schematic diagram of a flow chart of determining an interface type corresponding to an image frame scaling operation by means of a resolution prediction model in one embodiment of the present application;
  • FIG5 is a schematic diagram of a data curve constructed to reveal the mapping relationship between a given target quality index and a predicted ideal resolution
  • FIG6 is a schematic diagram of a flow chart of determining an interface type corresponding to an image frame scaling operation by querying an interface mapping table in another embodiment of the present application;
  • FIG7 is a schematic diagram of a flow chart of distinguishing different interface types for processing image frames in an embodiment of the present application
  • FIG8 is a flow chart of calling different software zooming interfaces to perform zooming operations on image frames in different situations according to an embodiment of the present application
  • FIG9 is a schematic diagram of a process of adjusting the image frame after scaling in an embodiment of the present application.
  • FIG10 is a schematic diagram of a process of encoding a live video stream on a media server side in an embodiment of the present application
  • FIG11 is a functional block diagram of the video display control device of the present application.
  • FIG. 12 is a schematic diagram of the structure of a video display control device used in the present application.
  • FIG1 shows the network architecture used in the exemplary application scenario of the present application, which can be used to deploy a live broadcast service, where the live video stream of the live broadcast service is transmitted to the terminal device after being encoded.
  • the application server 81 shown in FIG1 can be used to support the operation of the live broadcast room of the live broadcast, and the media server 82 can be used to The relay is realized by processing the decoding process of the live video stream pushed by each live user, wherein the terminal devices such as the computer 83 and the mobile phone 84 are generally provided to the terminal users as the client.
  • the computer program product programmed and implemented according to the video display control method of the present application can be applied to realize the zooming operation of the image frame in the live video stream and display it in the video playback window of the live room interface.
  • a computer program product programmed according to the video display control method of the present application can be implemented as a plug-in of a live broadcast room application, which runs along with the loading of the live broadcast room to implement zoom operations for the live video stream of the live broadcast room.
  • the principle block diagram shown in Figure 2 reveals the implementation principle of the media server of the network live broadcast service for pushing live video streams. It can be seen from the figure that in order to provide the best resolution parameters to the encoder, the encoder is adaptively controlled to encode the image frames of the video stream.
  • a parameter optimization model simulates an approximate function that characterizes the mapping relationship between the encoding resolution and the target quality index according to the encoder's export bandwidth.
  • An image detection module determines the subjective evaluation parameters determined by the subjective quality evaluation method of the reference image of the current image frame according to the live video stream.
  • the subjective evaluation parameters include reference resolution and subjective quality indicators.
  • both the approximate function and the subjective evaluation parameters contain the corresponding relationship between resolution and quality indicators.
  • the approximate function is inferred by the parameter optimization model and optimized by the objective quality indicator of the encoded image frame. Therefore, the quality indicator inferred by the approximate function can be quantified in the form of objective quality indicators, and the quality indicator in the subjective evaluation parameters is determined by referring to the data obtained by quantifying the user's subjective feelings according to the subjective image quality evaluation method. Therefore, its quality indicator can be quantified in the form of subjective quality indicators.
  • a resolution determination module obtains the numerical approximation point between the approximate function and the subjective evaluation parameter, obtains the encoding resolution corresponding to the numerical approximation point as the optimal resolution, and determines the target quality indicator corresponding to the encoding resolution, and controls the encoder to encode the image frame with the optimal resolution.
  • the encoded live video stream can be remotely pushed to the terminal devices participating in the live broadcast room.
  • the original current image frame is referred to to calculate the corresponding current image frame in the live video stream decoded by the decoder to determine the objective quality index actually obtained by the latter, and then the weight parameters of the parameter optimization model are corrected by backpropagation based on the objective quality index and the optimal resolution during encoding, so that the parameter optimization model iteratively improves its simulation approximation function to predict the optimal resolution, thereby continuously optimizing the encoding efficiency during the video stream encoding process and obtaining high-quality encoding effects.
  • the target quality index corresponding to the optimal resolution is also determined, wherein the optimal resolution is used to encode the corresponding image frame, and the target quality index can be used as supplementary enhancement information (SEI, Supplemental enhancement information) embedded in the live video stream along with the image frame, and pushed to the corresponding terminal device along with the live video stream for its use in scaling operations.
  • SEI Supplemental enhancement information
  • the supplementary enhancement information is added in the protocol version after H.264, which is convenient for loading additional data in the video stream and can therefore be used.
  • a video display control method provided according to one aspect of the present application, in one embodiment thereof, comprises the following steps:
  • Step S1100 obtaining a live video stream, wherein the live video stream includes a plurality of image frames and target quality indicators thereof;
  • the media server After the media server encodes the live video stream, it will be sent to the terminal devices of each user in the corresponding live broadcast room.
  • the user's terminal device can receive the live video stream and then decode it to obtain each image frame and corresponding additional enhancement information.
  • the live video stream is composed of a series of image frames, each image frame has a certain encoding resolution, and the encoding resolution can be directly determined for later use.
  • the target quality index corresponding to the image frame can also be determined.
  • the target quality index is embedded in the live video stream by the media server when encoding the image frame.
  • the target quality indicator is a quality indicator corresponding to better image quality determined by the media server for the image frame during the process of encoding the image frame.
  • the specific type of the quality indicator can be any one of Peak Signal to Noise Ratio (PSNR), mean square error, and structural similarity, which is determined by the specific quality indicator adopted by the media server during the encoding process.
  • PSNR Peak Signal to Noise Ratio
  • mean square error mean square error
  • structural similarity which is determined by the specific quality indicator adopted by the media server during the encoding process.
  • the peak signal-to-noise ratio algorithm formula is as follows:
  • bits refers to the number of bits occupied by each pixel.
  • Peak signal-to-noise ratio that is, the ratio of the peak signal energy to the average energy of the noise
  • dB decibels
  • MSE is the mean square error indicator, and the corresponding formula of its algorithm is as follows:
  • fij , f'ij represent the real image and the noisy image respectively
  • M, N represent the height and width of the image frame respectively. Since the real image and the noisy image are of the same scale, their height and width are the same.
  • Step S1200 determining an interface type corresponding to a scaling operation of the image frame according to a coding resolution of the image frame and a corresponding target quality index thereof, wherein the interface type is a hardware interface type or a software interface type;
  • the media server When the media server performs corresponding encoding processing on any image frame, it uses the corresponding encoding resolution to perform corresponding encoding operations on the image frame according to the target quality index it pursues.
  • different terminal devices have their own inherent resolutions. For example, 2K display screens, 4K display screens, etc. have different display resolutions, but the encoding resolutions of image frames may be different resolutions such as 360P, 720P, 1080P, etc., so it is necessary to perform corresponding scaling operations on the image frames in the live video stream to make them match the actual resolution of the terminal device as much as possible, so as to obtain better display effects.
  • the scaling operation on the image frame is implemented by calling a scaling interface
  • the scaling interface can be a hardware scaling interface, which belongs to the hardware interface type, or one or more software scaling interfaces. It belongs to the software interface type.
  • the hardware scaling interface is usually an interface corresponding to the scaling operation performed by the hardware of the terminal device.
  • the software scaling interface uses software algorithms to implement various forms of processing such as super-resolution enhancement and interpolation of image frames, thereby achieving enlargement or reduction of the scale of the image frame while ensuring that better image quality can be obtained after scaling.
  • Various software algorithms can be integrated in the operating system of the terminal device, or integrated into the live broadcast room application for calling.
  • the corresponding interface type can be determined based on whether both the resolution and target quality indicators are higher than corresponding preset thresholds. When both are higher than the corresponding preset thresholds, the interface type corresponding to the scaling operation of the image frame is determined to be a software interface type, otherwise it is a hardware interface type.
  • the coding resolution, target quality index and interface type can be constructed into a mapping table, in which the interface type to be adopted in the case of a specific coding resolution and a specific target quality index combination has been pre-stored, which is a hardware interface type or a software interface type. Therefore, by querying the mapping table, a group of data records closest to the coding resolution and target quality index of the image frame is determined, and the corresponding interface type can be determined therefrom.
  • the mapping relationship between the target quality index and its ideal resolution can be fitted into a function, and a corresponding resolution prediction model is generated according to the function.
  • the target quality index of the image frame is input into the resolution prediction model to predict its ideal resolution, and then the corresponding interface type is determined by comparing the target resolution corresponding to the scaling target with the ideal resolution.
  • the hardware interface type is used to improve the computing efficiency, otherwise the software interface type is used to improve the image quality.
  • the target resolution can be the inherent resolution of the display screen of the terminal device, or it can be a resolution designed to match different display screens.
  • Step S1300 Call the zoom interface corresponding to the interface type to perform a zoom operation on the image frame and then render and display it in the video playback window of the live broadcast room interface.
  • the corresponding scaling interface can be called according to the interface type to implement the scaling operation on the image frame.
  • the image frame is scaled to a target resolution that is adapted to the display screen of the terminal device and rendered to the video playback window of the live broadcast room interface for display.
  • the scaling operation is implemented by the hardware of the terminal device responsible for processing the image, such as a graphics card chip;
  • the interface type is a software interface type
  • the CPU of the terminal device calls the code instructions for executing the corresponding software algorithm, and executes the corresponding software algorithm, and implements the scaling operation of the image frame through the execution of the software algorithm.
  • the display size of the image frame is restricted by the video playback window.
  • the video playback window can be either a full-screen window or a partial window, but its actual resolution is always subject to the inherent resolution of the display screen.
  • the image frame scaling operation is matched with the corresponding interface type according to the encoding resolution and target quality index of the image frame, so that image frames under different encoding resolutions and different target quality index conditions can achieve image scaling with a more suitable scaling interface. Therefore, when the image frames of the live video stream are successively displayed in the video playback window, a generally stable display quality can be obtained.
  • the present application synchronously obtains the target quality index corresponding to each image frame in the live video stream when acquiring the live video stream, and then selects the interface type corresponding to the scaling operation according to the encoding resolution and target quality index of the image frame, calls the scaling interface corresponding to the interface type to implement the scaling operation on the image frame, and then renders and displays it in the video playback window of the live broadcast room interface, taking the target quality index of the image frame given when pushing the live video stream as the standard, matches the interface type based on the target quality index, and implements the scaling operation on the image frame by the scaling interface corresponding to the interface type, ensuring that the image effect after scaling the image frame can meet the standard established by the target quality index to the greatest extent, so that when the live video stream is played in the live broadcast room interface, a relatively high-quality and stable picture effect can be obtained, which can improve the user experience of online live broadcast users.
  • determining the interface type corresponding to the scaling operation of the image frame according to the encoding resolution of the image frame and its corresponding target quality index includes:
  • Step S1211 inputting the target quality index of the image frame into a resolution prediction model to determine an ideal resolution
  • the target quality index of the image frame in the live video stream can be used to predict the resolution that the image frame can achieve while maintaining the corresponding target quality index, which is referred to as the ideal resolution in this application.
  • the present application pre-models a mathematical model as a resolution prediction model for implementation.
  • x represents the empirical conversion value of the quality indicator, and its conversion relationship can be determined in advance
  • y ideal represents the resolution
  • ⁇ is the learnable weight
  • b is the learnable bias parameter.
  • x can be determined as follows:
  • psnr frame is the quality indicator of the image frame, which is the target quality indicator of the image frame in the model inference stage
  • the resolution prediction model is pre-trained using corresponding sample data pairs, i.e., corresponding data pairs between quality indicators and resolutions.
  • the quality indicators are used as input, and the resolution is used to supervise the updating of its weights and bias parameters to make them converge, so that it can be used to predict the corresponding ideal resolution according to the target quality indicators.
  • FIG. 5 it is a diagram of a mapping curve obtained after the resolution prediction model solidifies the weights and bias parameters, wherein the horizontal axis represents the normalized target quality index and the vertical axis represents the normalized ideal resolution. It can be seen from Figure 5 that there is a roughly linear relationship between the target quality index and its ideal resolution, which conforms to objective facts, and thus the resolution prediction model can be used to determine the ideal resolution corresponding to the target quality index.
  • Step S1212 Apply a preset conversion formula to calculate the target resolution that the image frame needs to achieve based on the encoding resolution scaling under its existing target quality index.
  • a conversion formula may be set according to an empirical rule to determine the target resolution that needs to be achieved by scaling the image frame corresponding to its target quality index on the basis of its encoding resolution.
  • the following exemplary conversion formula is applied to determine the target resolution:
  • psnr frame is the quality indicator of the image frame, which is the target quality indicator of the image frame in the model inference stage, and psnr lowest is the minimum threshold of the quality indicator; res frame is the encoding resolution of the image frame, and res lowest is the minimum threshold of the resolution, for example, the value is 180, and n is the weight. In one embodiment, the value is 2. Technical personnel in this field can make fine adjustments according to actual conditions.
  • the corresponding target resolution y real can be determined.
  • Step S1213 compare the target resolution with the ideal resolution, and when the target resolution is greater than the ideal resolution, determine that the interface type of the image frame scaling operation is a hardware interface type, otherwise it is a software interface type.
  • the two are compared, that is, whether y real >y ideal is established. If so, it indicates that the target resolution to be achieved is greater than the ideal resolution.
  • the interface type corresponding to the scaling operation of the image frame is determined to be a hardware interface type, so that the hardware scaling interface can be used to scale the image frame to save system overhead and improve computing efficiency. If it is not true, it means that the target resolution is less than or equal to the ideal resolution.
  • the technical advantages of the software algorithm can be used to ensure high-quality imaging effects after the scaling operation, and the interface type of the scaling operation of the image frame is determined to be a software interface type accordingly.
  • the corresponding ideal resolution can be predicted based on the target quality index of the image frame according to the acquired empirical knowledge.
  • the target resolution that the image frame needs to be scaled to reach can be determined based on the encoding resolution of the image frame and its target quality index. Both the ideal resolution and the target resolution are determined based on the target quality index of the image frame, wherein the ideal resolution plays the role of a decision threshold.
  • the target resolution is greater than the ideal resolution, it is decided to use the hardware interface type to implement the scaling operation, and vice versa, the software interface type is used to implement the scaling operation.
  • the corresponding scenarios of using hardware scaling operations and software scaling operations can be distinguished, and the system overhead and imaging quality are balanced to ensure that the live video stream can obtain a relatively stable image quality effect as a whole by utilizing limited hardware computing resources during playback.
  • determining the interface type corresponding to the scaling operation of the image frame according to the encoding resolution of the image frame and its corresponding target quality index includes:
  • Step S1221 Based on the coding resolution of the image frame, a mapping subset corresponding to the coding resolution is searched in a preset interface mapping table, where the mapping subset includes different target quality indicators and their corresponding interface types;
  • an interface mapping table is used to determine the corresponding interface type for the image frame.
  • the interface mapping table is a mapping table that characterizes the interface types corresponding to a given coding resolution and target quality index, and contains multiple data records. Through each data record, the interface types corresponding to the same coding resolution and different target quality indexes, and the interface types corresponding to different coding resolutions and the same target quality indexes are given.
  • An exemplary interface mapping table is as follows:
  • mapping table For an image frame, using its encoding resolution, for example, 360, the corresponding mapping subset is queried from the above mapping table as follows:
  • the mapping subset only includes the interface types corresponding to different target quality indicators at a specific encoding resolution.
  • the interface type is represented by 0 and 1, where 0 indicates that the interface type is a hardware interface type and 1 indicates that the interface type is a software interface type.
  • Step S1222 Based on the target quality indicator of the image frame, query the interface type corresponding to the target quality indicator from the mapping subset.
  • the corresponding interface type can be determined. For example, when the target quality index of the image frame is 34, the output interface type is determined to be a value of 0, that is, the hardware interface type is used. When it is 45, the output interface type is determined to be a value of 1, that is, the software scaling type is used.
  • an approximate determination method can be adopted to use the corresponding encoding resolution and target quality index with the smallest difference between the encoding resolution and the target quality index to determine the interface type used for the scaling operation of the image frame.
  • the interface type required for scaling the image frames in the live video stream is determined based on a table lookup, and the query process is more efficient and convenient with low computational complexity.
  • calling the zoom interface corresponding to the interface type to perform a zoom operation on the image frame and then rendering and displaying it in the video playback window of the live studio interface includes:
  • Step S1310 When the interface type is a hardware interface type, calling a hardware scaling interface to perform a scaling operation on the image frame;
  • Different terminal devices have different hardware scaling capabilities, but they all provide corresponding hardware scaling interfaces for external calls. Accordingly, when the interface type required for the image frame is a hardware interface type, it is only necessary to call the hardware scaling interface on the corresponding terminal device to perform the scaling operation on the image frame.
  • Step S1320 When the interface type is a software interface type, calling a software zoom interface to perform a zoom operation on the image frame;
  • the interface type required for the image frame is a software interface type
  • the live broadcast room application can come with various software algorithms, or the operating system of the terminal device may also provide relevant software algorithms, in this case, the preset software scaling interface is called accordingly to perform scaling operations on the image frame, so as to apply the corresponding software algorithm to scale the image frame to a specific resolution.
  • Optional software algorithms include but are not limited to a first software scaling algorithm with an added post-filtering processing link, a second software scaling algorithm based on a deep learning model (such as a super-resolution model), a third software scaling algorithm based on bilinear implementation (bilinear interpolation algorithm), and a fourth software scaling algorithm based on bicubic implementation (bicubic interpolation algorithm).
  • a first software scaling algorithm with an added post-filtering processing link a second software scaling algorithm based on a deep learning model (such as a super-resolution model), a third software scaling algorithm based on bilinear implementation (bilinear interpolation algorithm), and a fourth software scaling algorithm based on bicubic implementation (bicubic interpolation algorithm).
  • Different software scaling algorithms have different requirements for hardware computing power levels and can be used flexibly as needed.
  • Step S1330 Render and display the image frame after the zoom operation in the video playback window of the live broadcast room interface.
  • the image rendering algorithm is applied to render and display it in the video playback window of the live studio interface, so that the image frame can be played in the video playback window.
  • the image frames in the live video stream are rendered and displayed in sequence, and can be presented in the video playback window of the live studio.
  • the playback content of the live video stream is not limited to
  • calling a software scaling interface to perform a scaling operation on the image frame includes:
  • Step S1321 determining whether the scaling ratio of the image frame is a double ratio, and when it is a double ratio, using a first software scaling interface to implement a scaling operation on the image frame to eliminate moiré patterns generated during the scaling process of the image frame, wherein the scaling ratio is a ratio of a target resolution of the image frame to a coding resolution;
  • the target resolution of the image frame determined in the previous embodiment can be divided by the inherent encoding resolution of the image frame to determine the scaling ratio, and then it can be determined whether the scaling ratio is a double ratio. If it is a double ratio, the image frame can be scaled using the first software scaling interface provided by the first software scaling algorithm with an added post-filtering processing link, so that the image frame obtained after scaling can eliminate the moiré pattern generated during the scaling process, thereby ensuring the image quality of the image frame.
  • Step S1322 When the zoom ratio is not a double ratio, determine the hardware computing power level of the current device, and call a software zoom interface of a corresponding level according to the hardware computing power level to perform a zoom operation on the image frame.
  • multiple software scaling algorithms corresponding to different hardware computing powers can be provided in the live broadcast room application, such as the second, third, and fourth software scaling algorithms mentioned above. Then, by determining the hardware computing power level of the terminal device where the live broadcast room application is located to match the corresponding software scaling algorithm, the software scaling interface provided by the corresponding software scaling algorithm is called to implement the scaling operation.
  • the hardware computing power level in one embodiment, is pre-planned into three levels, corresponding to high-end, medium-end, and low-end terminal devices in terms of computing power performance, and then the brands, models, or CPUs, flash memories, etc. of these terminal devices are mapped with the corresponding hardware computing power levels and stored as a table for query. Different hardware computing power levels are also pre-associated with different software scaling algorithms.
  • high-end performance terminal devices can use the second software scaling algorithm exemplified in this application to implement scaling operations on image frames based on deep learning models;
  • mid-end performance terminal devices can use the third software scaling algorithm exemplified in this application, namely the bilinear algorithm, to balance the scaling effect and computing power requirements;
  • low-end performance terminal devices can use the fourth software scaling algorithm exemplified in this application, namely the bicubic algorithm, etc., to take advantage of the low computational requirements of the algorithm.
  • the scaling operation of the image frame is determined to be implemented by the software interface type
  • multiple situations can be distinguished, including double ratio and non-double ratio, and different hardware computing power levels
  • the corresponding software scaling algorithm is used to overcome various problems that may be caused by the scaling operation. It solves potential problems, such as the appearance of moiré in 2x scaling and the problem of live video streaming playback being stuck due to insufficient hardware computing power, making the live video streaming playback process smoother and the image quality more stable.
  • the method before rendering and displaying the image frame after the zoom operation into the video playback window of the live studio interface, the method includes:
  • Step S2100 determining whether the image frame after the zoom operation exceeds the boundary of the video playback window, and if it exceeds the boundary, performing face detection on the image frame to determine the face area;
  • the media server dynamically determines the encoding resolution of the image frames during the encoding process of the live video stream, when the terminal device scales different image frames of the same live video stream, the scaled images of each image frame may be different while keeping the target quality index roughly the same.
  • the final image size is still subject to the video stream playback window, some preprocessing can be performed to highlight the key points. Even if the live video stream produces resolution fluctuations, the key area is still the main display area of the video playback window, so that users are basically unaware of the impact of the resolution change.
  • the face detection model determines the face area where the face image is located. It is not difficult to understand that the face image in the face area is usually the face of the live user, which is the key content that the audience user expects to see.
  • the face area determined by the face detection model is usually a coordinate information, and a selection box is represented by the coordinates of the upper left corner and the lower right corner.
  • Step S2200 cropping the image frame with the face region as the center so that the size of the image frame fits the size of the video playback window.
  • the geometric center of the selection box of the face area is used as the center of the frame to be cropped, and the scaled image frame is cropped to a corresponding scale corresponding to the size of the video playback window.
  • the cropped image frame is centered on the face image covered by the face area. When it is rendered and displayed in the video playback window, the face image is displayed relatively centered.
  • the face image When the encoding resolution of consecutive image frames changes, the face image remains centered due to cropping, and the user is not easily aware of the impact of the resolution change.
  • the face image in the image frame is determined by face detection, and the image is cropped and then rendered and displayed with the face image in the center. This can filter out the impact of excessive image changes caused by significant changes in encoding resolution in the live video stream, thereby ensuring that users can obtain a good video viewing experience.
  • Step S3100 determining a subjective evaluation parameter of the current image frame, wherein the subjective evaluation parameter includes a reference resolution suitable for encoding the current image frame and a subjective quality index that can be expected to be obtained by encoding according to the reference resolution;
  • the media server When the media server obtains the live video stream uploaded by the live user, it can decode it to obtain each image frame, and call each image frame one by one as the current image frame for encoding processing. For each current image frame in the video stream, its corresponding subjective quality index can be determined by referring to the image subjective quality evaluation method.
  • a subjective image quality evaluation method is applied in advance to provide multiple video contents with different resolutions and play them for viewers to watch. Then, the scores of each viewer are obtained from multiple preset dimensions, and weighted and quantized into a subjective quality index according to the scores given by the viewer.
  • a mapping relationship table is established for the subjective quality indexes corresponding to the multiple resolutions, and then used as a reference for the application to determine the reference resolution and subjective quality index that matches the current image frame.
  • the reference resolution corresponding to the previous image frame is adjusted accordingly based on the change in image complexity of the current image frame relative to its previous image frame, so that the reference resolution corresponding to the current image frame can be evaluated. Based on the evaluated reference resolution, the corresponding subjective quality index can be determined using the mapping relationship data, thereby obtaining a reference resolution and a corresponding subjective quality index, which constitute the subjective evaluation parameters corresponding to the current image frame.
  • the reference resolution is intermediate data used as a reference, which constitutes an indication of the resolution encoding condition of the current image frame.
  • the subjective quality index refers to the subjective evaluation result of the image quality that can be predicted when the reference resolution is used as the encoding condition.
  • the mapping relationship table determined according to the subjective image quality evaluation method is fitted to obtain the corresponding subjective curve function, and then the corresponding subjective quality index can be quickly obtained according to the reference resolution determined corresponding to each current image frame to obtain its subjective evaluation parameters.
  • the subjective curve function has a linear characteristic after fitting, the subjective evaluation parameters determined thereby can be smooth.
  • Step S3200 Input the output bandwidth of the encoder into the parameter optimization model to obtain an approximate function that characterizes the mapping relationship between the encoding resolution and the target quality index under the output bandwidth;
  • the output bandwidth of the encoder is an objective condition, which is limited by network conditions and/or hardware conditions, and therefore has relative certainty.
  • the output bandwidth can be input into the parameter optimization model, and the control parameter optimization model simulates an approximate function for solving the optimal resolution of the encoder.
  • the parameter optimization model is implemented by pre-modeling. For example, considering that the frame rate of live video stream transmission is relatively stable under the same network conditions, and the encoding resolution and objective quality index are more flexible with the change of the encoder's export bandwidth, the objective quality index and encoding resolution of different video streams under a specific export bandwidth are fitted. From the fitting results, it can be seen that the encoding resolution and objective quality index of the video stream follow the natural logarithmic distribution. According to this principle, the parameter optimization model can be modeled and implemented.
  • ypsnr is an exemplary objective quality indicator
  • resolution is the resolution
  • A is a learnable weight parameter
  • B is a learnable bias parameter, which constitute the learnable parameters of the model.
  • the parameter optimization model can modify its weight parameters and bias parameters according to the objective quality indicators of the image frames output by the encoder, realize iterative training, and make it continuously approach convergence.
  • the core of the parameter optimization model is to construct a data curve corresponding to the mapping relationship between the export bandwidth, coding resolution, and objective quality indicators, so that the mapping relationship between the coding resolution and the objective quality indicators can be predicted according to the export bandwidth, that is, an approximate function is determined. Further determining a corresponding point on the determined approximate function can determine the specific coding resolution and objective quality indicators. It can be seen that the data curve corresponding to a specific export bandwidth essentially reflects the mapping relationship data set between the coding resolution and the objective quality indicators under the specific export bandwidth. The mapping relationship data set is represented by the approximate function obtained by the parameter optimization model. Based on this, this approximate function can be obtained according to an export bandwidth.
  • step S3100 and step S3200 can be executed concurrently, and the execution order can be swapped, and are not limited by the order listed in this application.
  • parameter optimization model can be modeled as a machine learning model with low computing resource utilization, or it can be modeled as a more intelligent deep learning model, as long as the functions disclosed in this application can be implemented according to the above principles.
  • Step S3300 Apply the approximate function to obtain the numerical approximation point corresponding to the subjective evaluation parameter, and obtain the coding score corresponding to the numerical approximation point. resolution and target quality indicators;
  • the approximate function represents the mapping relationship data of the linear relationship between the coding resolution and the objective quality index under a specific export bandwidth, which is reflected as a data curve in the rectangular coordinate system, and the subjective evaluation parameters corresponding to the current image frame, that is, its reference resolution and the subjective quality index corresponding to the reference resolution, can be reflected as a point in the rectangular coordinate system.
  • the point with the shortest distance on the data curve is determined to be the numerical approximation point on the data curve.
  • the numerical approximation point corresponds to the mapping relationship data between a coding resolution and an objective quality indicator.
  • the coding resolution can be determined as the optimal resolution for encoding the corresponding image frame, and the objective quality indicator is regarded as the target quality indicator expected for the corresponding encoding.
  • the subjective curve function is obtained in advance by data fitting.
  • the intersection of the subjective curve function and the approximate function can be calculated, and the intersection is the numerical approximation point, thereby the corresponding encoding resolution can also be obtained as the optimal resolution.
  • the optimal resolution determined in the present application is input into the encoder to control the encoder to encode the current image frame at the optimal resolution so as to encode the current image frame in the live video stream.
  • Step S3400 Use the target quality indicator as additional enhancement information corresponding to the current image frame, encode the current image frame into a live video stream according to the encoding resolution and push it to the live broadcast room.
  • the image frame After determining the encoding resolution corresponding to the current image frame, the image frame is subjected to corresponding resolution conversion processing according to the encoding resolution, and the converted image frame is encoded into a live video stream so as to be pushed to the live broadcast room.
  • the target quality index of the current image frame is further included in the additional enhancement information of the live video stream, a corresponding relationship is established with the current image frame, and the target quality index is delivered to the terminal device in the live broadcast room along with the live video stream.
  • the present application determines the subjective evaluation parameters corresponding to the current image frame in the live video stream, wherein the subjective evaluation parameters include a reference resolution suitable for encoding the current image frame and a subjective quality index that can be expected to be obtained by encoding according to the reference resolution rate, and evaluates the corresponding relationship between the resolution change and the image quality change;
  • the parameter optimization model predicts an approximate function that characterizes the mapping relationship between the encoding resolution and the objective quality index, and on this basis, obtains the numerical approximation point corresponding to the subjective evaluation parameter from the approximate function, and then determines the optimal resolution required for encoding according to the numerical approximation point to control the encoder to encode and output the current image frame, and uses the associated determined objective quality index as the target quality index corresponding to the current image frame, which is sent to the terminal device along with the live video stream, so that the objective
  • a video display control device includes a data acquisition module 1100, an interface matching module 1200, and a zoom display module 1300, wherein the data acquisition module 1100 is configured to acquire a live video stream, wherein the live video stream includes a plurality of image frames and their target quality indicators; the interface matching module 1200 is configured to determine an interface type corresponding to a zoom operation of the image frame according to an encoding resolution of the image frame and its corresponding target quality indicator, wherein the interface type is a hardware interface type or a software interface type; the zoom display module 1300 is configured to call the interface matching module 1200 to determine an interface type corresponding to a zoom operation of the image frame according to an encoding resolution of the image frame and its corresponding target quality indicator; The zoom interface corresponding to the port type zooms the image frame and then renders it into the video playback window of the live broadcast room interface.
  • the data acquisition module 1100 is configured to acquire a live video stream, wherein the live video stream includes a plurality of image frames and their target quality indicators
  • the interface matching module 1200 includes: a model inference unit, configured to input the target quality index of the image frame into a resolution prediction model to determine an ideal resolution; a target conversion unit, configured to apply a preset conversion formula to calculate the target resolution that the image frame needs to achieve based on its encoding resolution scaling under its existing target quality index; an interface selection unit, configured to compare the target resolution with the ideal resolution, and when the target resolution is greater than the ideal resolution, determine that the interface type of the scaling operation of the image frame is a hardware interface type, otherwise it is a software interface type.
  • the interface matching module 1200 includes: a resolution query unit, which is configured to query a mapping subset corresponding to the coding resolution of the image frame in a preset interface mapping table, and the mapping subset includes different target quality indicators and their corresponding interface types; an indicator query unit, which is configured to query the interface type corresponding to the target quality indicator from the mapping subset based on the target quality indicator of the image frame.
  • the zoom display module 1300 includes: a hardware zoom unit, which is configured to call the hardware zoom interface to perform a zoom operation on the image frame when the interface type is a hardware interface type; a software zoom unit, which is configured to call the software zoom interface to perform a zoom operation on the image frame when the interface type is a software interface type; and a rendering display unit, which is configured to render and display the image frame after the zoom operation into the video playback window of the live broadcast room interface.
  • the software scaling unit includes: a conventional processing subunit, configured to determine whether the scaling ratio of the image frame is a double ratio, and when it is a double ratio, using a first software scaling interface to implement a scaling operation on the image frame to eliminate the moiré pattern generated by the image frame during the scaling process, and the scaling ratio is the ratio of the target resolution of the image frame to the encoding resolution; an exception processing subunit, configured to determine the hardware computing power level of the current device when the scaling ratio is not a double ratio, and call the software scaling interface of the corresponding level according to the hardware computing power level to implement the scaling operation on the image frame.
  • the zoom display module 1300 includes: a boundary detection unit, configured to determine whether the image frame after the zoom operation exceeds the boundary of the video playback window, and when it exceeds the boundary, performing face detection on the image frame to determine the face area; an image cropping unit, configured to crop the image frame with the face area as the center so that the size of the image frame adapts to the size of the video playback window.
  • the video display control device of the present application further includes:
  • a parameter determination module configured to determine a subjective evaluation parameter of a current image frame, wherein the subjective evaluation parameter includes a reference resolution suitable for encoding the current image frame and a subjective quality index that can be expected to be obtained by encoding according to the reference resolution;
  • a function acquisition module is configured to input the output bandwidth of the encoder into the parameter optimization model to obtain an approximate function representing the mapping relationship between the encoding resolution and the target quality index under the output bandwidth;
  • a numerical determination module is configured to apply the approximate function to obtain a numerical approximation point corresponding to the subjective evaluation parameter, and obtain a coding resolution and a target quality index corresponding to the numerical approximation point;
  • the encoding push module is configured to use the target quality indicator as additional enhancement information corresponding to the current image frame, encode the current image frame into a live video stream according to the encoding resolution, and push it to the live broadcast room.
  • FIG. 12 a schematic diagram of the internal structure of the video display control device.
  • the video display control device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus.
  • the computer-readable non-volatile readable storage medium of the video display control device stores an operating system, a database, and computer-readable instructions.
  • the database may store an information sequence.
  • the processor of the video display control device is used to provide computing and control capabilities to support the operation of the entire video display control device.
  • the memory of the video display control device may store computer-readable instructions, which, when executed by the processor, may enable the processor to execute the video display control method of the present application.
  • the network interface of the video display control device is used to connect and communicate with a terminal.
  • FIG. 12 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the video display control device to which the scheme of the present application is applied.
  • the video display control device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
  • the processor is used to execute the specific functions of each module in Figure 11, and the memory stores the program code and various data required to execute the above modules or submodules.
  • the network interface is used to realize data transmission between user terminals or servers.
  • the non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the video display control device of this application, and the server can call the program code and data of the server to execute the functions of all modules.
  • the present application also provides a non-volatile readable storage medium storing computer-readable instructions.
  • the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the video display control method of any embodiment of the present application.
  • the present application also provides a computer program product, including a computer program/instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.
  • the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
  • the present application uses the target quality index of the image frame given when pushing the live video stream as the standard, matches the interface type based on the target quality index, and implements the scaling operation of the image frame by the scaling interface corresponding to the interface type, thereby ensuring that the image effect of the image frame after scaling can meet the standard established by the target quality index to the greatest extent, so that when the live video stream is played in the live broadcast room interface, a relatively high-quality and stable picture effect can be obtained, which can enhance the user experience of online live broadcast users.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本申请实施例提供了一种视频显示控制方法及其装置、设备、介质、产品，所述方法包括：获取直播视频流，所述直播视频流中包含多个图像帧及其目标质量指标；根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，所述接口类型为硬件接口类型或软件接口类型；调用所述接口类型相对应的缩放接口对所述图像帧进行缩放操作后渲染显示到直播间界面的视频播放窗口中。本申请以推送直播视频流时给定的图像帧的目标质量指标为标准，基于该目标质量指标确定相对应缩放接口实施对所述图像帧的缩放操作，使直播视频流在直播间界面中播放时，能够获得相对应较为优质、稳定的画面效果，能够提升网络直播用户的使用体验。

Description

视频显示控制方法及其装置、设备、介质、产品
本申请要求在2023年01月18日提交中国专利局，申请号202310084572.1的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请涉及直播视频处理技术领域，尤其涉及一种视频显示控制方法及其装置、设备、介质、产品。
背景技术
网络直播传输的直播视频流，会经过编码传输，在编码过程中，考虑到接收端的终端设备的显示质量的问题，一般是关联带宽、质量指标、码率、分辨率等多个维度的关联变动，来确定一组编码控制参数，通过这些编码控制参数来保证终端设备所获得的直播视频流的质量，使直播视频流播放时能够获得优质的显示效果。
但是，相关技术在处理直播视频流的显示时，往往忽视对终端设备侧的硬件条件的考虑，而任由终端设备一侧按照其固有的业务逻辑，使用相应的解码方式，将直播视频流中的图像帧渲染显示到直播程序的视频流播放窗口中。
直播视频流到达终端设备后，通常都需要适应终端设备中视频流播放窗口的分辨率做相应的缩放操作，相应的缩放算法主要分为软件缩放和硬件缩放两大类，其中，硬件缩放主要依赖于终端设备的硬件条件，可以节省设备功耗和CPU的算力开销，但其缩放后的图像效果相对一般；而软件缩放则有多种，不同的软件缩放算法对算力的要求不同，效果也不同，一般来说，算力要求高的软件算法，其缩放后的图像效果更佳。
所以，缩放算法的匹配也能决定直播视频流在终端的显示效果，而传统技术往往忽略对这一技术关键的应对方案，导致虽然在服务端对直播视频流实施了编码控制，但直播视频流在不同的终端设备中显示的时候，其图像显示效果差别较大。
发明内容
本申请的目的在于解决上述问题而提供一种视频显示控制方法及其相应的装置、设备、非易失性可读存储介质，以及计算机程序产品。
根据本申请的一个方面，提供一种视频显示控制方法，包括如下步骤：
获取直播视频流，所述直播视频流中包含多个图像帧及其目标质量指标；
根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，所述接口类型为硬件接口类型或软件接口类型；
调用所述接口类型相对应的缩放接口对所述图像帧进行缩放操作后渲染显示到直播间界面的视频播放窗口中。
根据本申请的另一方面，提供一种视频显示控制装置，包括：
数据获取模块，设置为获取直播视频流，所述直播视频流中包含多个图像帧及其目标质量指标；
接口匹配模块，设置为根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，所述接口类型为硬件接口类型或软件接口类型；
缩放显示模块，设置为调用所述接口类型相对应的缩放接口对所述图像帧进行缩放操作后渲染显示到直播间界面的视频播放窗口中。
根据本申请的另一方面，提供一种视频显示控制设备，包括中央处理器和存储器，所述中央处理器用于调用运行存储于所述存储器中的计算机程序以执行本申请所述的视频显示控制方法的步骤。
根据本申请的另一方面，提供一种非易失性可读存储介质，其以计算机可读指令的形式存储有依据所述的视频显示控制方法所实现的计算机程序，所述计算机程序被计算机调用运行时，执行该方法所包括的步骤。
根据本申请的另一方面，提供一种计算机程序产品，包括计算机程序/指令，所述计算机程序/指令被处理器执行时实现本申请任意一种实施例中所述方法的步骤。
相对于现有技术，本申请在获取直播视频流时同步获取直播视频流中的各个图像帧相对应的目标质量指标，然后根据所述图像帧的编码分辨率及目标质量指标来选择缩放操作相对应的接口类型，调用所述接口类型相对应的缩放接口实施对所述图像帧的缩放操作后渲染显示到直播间界面的视频播放窗口中，以推送直播视频流时给定的图像帧的目标质量指标为标准，基于该目标质量指标匹配接口类型，由接口类型相对应缩放接口实施对所述图像帧的缩放操作，确保所述图像帧缩放后的图像效果能够最大化程度地符合所述目标质量指标所建立的标准，使直播视频流在直播间界面中播放时，能够获得相对应较为优质、稳定的画面效果，能够提升网络直播用户的使用体验。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请所应用的网络直播环境相对应的网络架构示意图；
图2为本申请中对直播视频流进行编码所依据的技术框架的原理示意图；
图3为本申请的视频显示控制方法的实施例的流程示意图；
图4为本申请在一种实施例中，借助分辨率预测模型确定图像帧缩放操作相对应的接口类型的流程示意图；
图5为用于揭示给定的目标质量指标与预测的理想分辨率之间的映射关系所构造的数据曲线的示意图；
图6为本申请在另一实施例中，借助接口映射表格查询确定图像帧缩放操作相对应的接口类型的流程示意图；
图7为本申请的实施例中区别不同接口类型处理图像帧的流程示意图；
图8为本申请实施例中区别不同情形调用不同软件缩放接口对图像帧实施缩放操作的流程示意图；
图9为本申请实施例中对缩放后的图像帧调整画幅的流程示意图；
图10为本申请实施例中媒体服务器侧编码直播视频流的流程示意图；
图11为本申请的视频显示控制装置的原理框图；
图12为本申请所采用的一种视频显示控制设备的结构示意图。
具体实施方式
请参阅图1，本申请示例性的应用场景所采用的网络架构，其可用于部署网络直播服务，网络直播服务的直播视频流的经编码处理后传输至终端设备。图1所示的应用服务器81可用于支持网络直播的直播间的运行，而媒体服务器82可用 于处理各个直播用户推送的直播视频流的解编码过程而实现中继，其中的计算机83、移动电话84之类的终端设备，作为客户端，一般提供给终端用户使用。终端设备接收所述直播视频流之后，可以应用根据本申请的视频显示控制方法编程实现的计算机程序产品，用于实现对直播视频流中的图像帧的缩放操作，并将其显示到直播间界面的视频播放窗口中。
一种实施例中，根据本申请的视频显示控制方法编程实现的计算机程序产品可以实现为直播间应用程序的一个插件，随同直播间的加载而运行，以便为直播间的直播视频流实现缩放操作。
图2所示的原理框图揭示了网络直播服务的媒体服务器关于推送直播视频流的实现原理，由该图可以看出，为了向编码器提供最佳分辨率参数，自适应控制编码器对视频流的图像帧进行编码，由一个参数优化模型根据编码器的出口带宽而模拟出表征编码分辨率与目标质量指标之间映射关系的近似函数，由一个图像检测模块根据直播视频流确定当前图像帧参考图像主观质量评价方法所确定的主观评估参数，该主观评估参数包括参考分辨率和主观质量指标。
显然，近似函数与主观评估参数均包含了分辨率与质量指标之间的对应关系，其中近似函数由参数优化模型推理而得，且受编码后的图像帧的客观质量指标所优化，故该近似函数推理所得的质量指标可按照客观质量指标的形式进行量化确定，而主观评估参数中的质量指标是参考根据图像主观质量评价方法量化用户主观感受而得的数据确定的，故其质量指标可按照主观质量指标的形式进行量化确定。
在一实施例中，由一个分辨率确定模块通过求取所述近似函数与所述主观评估参数的数值逼近点，获得该数值逼近点相对应的编码分辨率作为最佳分辨率，并确定该编码分辨率对应的目标质量指标，以该最佳分辨率控制编码器对图像帧进行编码，编码后的直播视频流可被远程推送到参与直播间的终端设备。
此外，通过一个质量评价模块，参考原始的当前图像帧对解码器解码所述直播视频流中相对应的当前图像帧计算确定后者实际获得的客观质量指标，再以该客观质量指标及编码时的最佳分辨率反向传播修正所述参数优化模型的权重参数，使该参数优化模型迭代提升其仿真近似函数以求解最佳分辨率的预判能力，从而在视频流编码过程中持续优化编码效率，获得优质的编码效果。
本申请中，除了通过所述挖函数确定出最佳分辨率之外，还确定出该最佳分辨率相对应的目标质量指标，其中的最佳分辨率用于编码对应的图像帧，而其中的目标质量指标则可以作为附加增强信息(SEI，Supplemental enhancement information)随同所述图像帧嵌入直播视频流中，跟随直播视频流推送至相应的终端设备供其实施缩放操作之用。所述附加增强信息，是H.264之后协议版本中增加的内容，方便在视频流中加载额外数据，因而可加以利用。
请参阅图3，根据本申请的一个方面提供的一种视频显示控制方法，在其一个实施例中，包括如下步骤：
步骤S1100、获取直播视频流，所述直播视频流中包含多个图像帧及其目标质量指标；
请参考图1所示的网络架构，当媒体服务器编码了直播视频流之后，将发送到相应的直播间的各个用户的终端设备，用户的终端设备便可接收到所述直播视频流，然后对其解码，获得其中的各个图像帧及相应的附加增强信息。
所述直播视频流由一系列的图像帧构成，每个图像帧具有确定的编码分辨率，可以直接确定该编码分辨率以备后用，此外，通过直播视频流中的附加增强信息，还可以确定出所述图像帧相对应的目标质量指标，该目标质量指标是媒体服务器在对所述图像帧编码时嵌入到直播视频流中的。
所述目标质量指标，是媒体服务器在编码所述图像帧的过程中，为图像帧确定出的能够获得较优画质相对应的质量指标，质量指标具体类型可以是峰值信噪比(PSNR，Peak Signal to Noise Ratio)、均方误差、结构相似性中任意一项，具体由媒体服务器在编码过程中所采用的具体质量指标而确定。
峰值信噪比的算法公式如下：
其中，bits即每个像素点存储所占的位数。
峰值信噪比，即峰值信号的能量与噪声的平均能量之比，通常表示的时候取log变成分贝(dB)，由于均方误差(MSE)为真实图像与含噪图像之差的能量均值，而两者的差即为噪声，因此PSNR即峰值信号能量与MSE之比。
不难理解，由于PSNR的计算复杂度较低，因而更适合在终端设备一侧计算，当然，也可部署于服务器侧，由本领域技术人员灵活确定即可。
MSE为均方误差指标，其算法相对应的公式如下：
其中，fij，f'ij分别代表真实图像和含噪图像，M,N分别表示图像帧的高度和宽度，由于真实图像和含噪图像是等尺度的，因而其彼此高度和宽度相同。
步骤S1200、根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，所述接口类型为硬件接口类型或软件接口类型；
媒体服务器对任意一个图像帧进行相应的编码处理时，按照其所追求的目标质量指标，采用了相应的编码分辨率对该图像帧实施相应的编码操作，但是，在终端设备一侧，不同的终端设备有其固有的分辨率，例如2K显示屏、4K显示屏等，其显示分辨率各不相同，但图像帧的编码分辨率可能是360P、720P、1080P等不同分辨率，所以需要对直播视频流中的图像帧进行相应的缩放操作，使其尽量与终端设备的实际分辨率相匹配，从而获得更佳的显示效果。
对所述图像帧实施缩放操作是通过调用缩放接口来实施的，所述缩放接口既可以是硬件缩放接口，属于硬件接口类型，也可以是一种或多种软件缩放接口。属于软件接口类型。所述硬件缩放接口通常是由终端设备的硬件执行缩放操作相对应的接口，调用硬件缩放接口，可以借助独立于终端设备的CPU之外的硬件能力来实施对所述图像帧的缩放操作，从而节省系统开销。所述软件缩放接口是通过软件算法来实现对图像帧的超分辨率增强、插值等各种形式的处理而实现在放大或缩小图像帧的尺度的同时，确保缩放后仍能获得较优的画质，各种软件算法可以集成在终端设备的操作系统中，也可以集成到直播间应用程序中以供调用。
一种实施例中，考虑到一般目标质量指标较高的图像帧，其编码分辨率相应也较高，也考虑到硬件缩放接口受限于硬件性能而软件缩放接口具有更为优质的缩放效果的现实，可以按照分辨率和目标质量指标是否均高于相应的预设阈值确定对应的接口类型，当均高于相应的预设阈值时，确定所述图像帧的缩放操作相对应的接口类型为软件接口类型，否则为硬件接口类型。
参考以上实施例的原则，还可以有多种具体实施方式来实现为所述图像帧确定其相应的接口类型。
另一实施例中，可以将编码分辨率、目标质量指标与接口类型构造成映射表格，该映射表格中，已经预存了特定编码分辨率和特定目标质量指标组合的情况下所应采用的接口类型为硬件接口类型或软件接口类型，因而，通过查询该映射表格，确定出与所述图像帧的编码分辨率和目标质量指标最接近的一组数据记录，从中确定相应的接口类型即可。
再一实施例中，可以将目标质量指标与其理想分辨率之间的映射关系拟合为一个函数，根据该函数生成相应的分辨率预测模型，将所述图像帧的目标质量指标输入所述分辨率预测模型预测出其理想分辨率，再根据缩放目标相对应的目标分辨率与理想分辨率比较来确定相应的接口类型。在目标分辨率大于理想分辨率时，使用硬件接口类型以提升运算效率，否则使用软件接口类型以提升画质。所述的目标分辨率，既可以是终端设备的显示屏固有的分辨率，也可以是为了通配不同显示屏而拟制的分辨率。
步骤S1300、调用所述接口类型相对应的缩放接口对所述图像帧进行缩放操作后渲染显示到直播间界面的视频播放窗口中。
当确定了所述图像帧的接口类型后，便可根据该接口类型调用相应的的缩放接口来实施对所述的图像帧的缩放操作，通过缩放操作，将所述图像帧缩放到适应终端设备的显示屏的目标分辨率，渲染到直播间界面的视频播放窗口中显示。
不难理解，当所述接口类型为硬件接口类型时，缩放操作由终端设备的负责处理图像的硬件例如显卡芯片所实施；当所述接口类型为软件接口类型时，由终端设备的CPU调用执行相应软件算法的代码指令，而执行相应的软件算法，通过软件算法的执行而实施对所述图像帧的缩放操作。
可以看出，图像帧的显示画幅是受所述视频播放窗口制约的，所述视频播放窗口既可以是全屏窗口，也可以是局部窗口，但其实际分辨率始终受制于显示屏固有的分辨率。由于经过以上过程的处理，根据所述图像帧的编码分辨率和目标质量指标而为图像帧的缩放操作匹配出其相适应的接口类型，使得不同编码分辨率和不同目标质量指标条件下的图像帧都能够以更为合适的缩放接口来实现图像缩放，因而，当直播视频流的各个图像帧陆续显示到视频播放窗口后，能够获得总体上较为稳定的显示画质。
根据以上实施例可知，本申请在获取直播视频流时同步获取直播视频流中的各个图像帧相对应的目标质量指标，然后根据所述图像帧的编码分辨率及目标质量指标来选择缩放操作相对应的接口类型，调用所述接口类型相对应的缩放接口实施对所述图像帧的缩放操作后渲染显示到直播间界面的视频播放窗口中，以推送直播视频流时给定的图像帧的目标质量指标为标准，基于该目标质量指标匹配接口类型，由接口类型相对应缩放接口实施对所述图像帧的缩放操作，确保所述图像帧缩放后的图像效果能够最大化程度地符合所述目标质量指标所建立的标准，使直播视频流在直播间界面中播放时，能够获得相对应较为优质、稳定的画面效果，能够提升网络直播用户的使用体验。
在本申请任意实施例的基础上，请参阅图4，根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，包括：
步骤S1211、将所述图像帧的目标质量指标输入分辨率预测模型确定出理想分辨率；
可以根据直播视频流中的图像帧的目标质量指标，来预测所述图像帧在保持相应目标质量指标之下，可以达到的分辨率，本申请称之为理想分辨率。
为了根据目标质量指标预测其相应的理想分辨率，本申请预先建模一个数学模型作为分辨率预测模型来实施，该分辨率预测模型的建模公式如下所示：
yideal＝ω*ln(x)+b
其中，x表示质量指标的经验转换数值，其转换关系可以预先确定，yideal表示分辨率，ω为可学习权重，而b则为可学习的偏置参数。
一个实施例中，x可按如下方式确定：
其中，psnrframe为所述图像帧的质量指标，在模型推理阶段即为所述图像帧的目标质量指标，psnrlowest为最低质量指标的最低门槛，例如取值为20，α、β均为权重，一个实施例中，α＝2，β＝25，本领域技术人员可以根据实际效果进行微调。
所述分辨率预测模型预先采用相对应的样本数据对，即质量指标与分辨率之间的对应数据对进行训练，在训练过程中以质量指标为输入，以分辨率监督其权重和偏置参数的更新，使其收敛，从而可以用于根据目标质量指标预测出相应的理想分辨率。
如图5所示是所述分辨率预测模型固化权重和偏置参数后获得的一条映射曲线的图示，其中的横轴表示归一化的目标质量指标，纵轴表示归一化的理想分辨率。由图5可以看出，目标质量指标与其理想分辨率之间具有大致呈线性的关系，符合客观事实，因而可以采用所述分辨率预测模型来确定目标质量指标相对应的理想分辨率。
步骤S1212、应用预设转换公式计算出所述图像帧在其既有目标质量指标下需要基于其编码分辨率缩放达到的目标分辨率。
在给定了图像帧的编码分辨率以及相应的目标质量指标的情况下，可以按照经验法则设定一个转换公式，用来确定所述图像帧对应其目标质量指标在其编码分辨率的基础上进行缩放所需要达到的目标分辨率，为此，对应所述媒体服务器在编码过程中确定所述图像帧的编码分辨率及相应的目标质量指标的逻辑，应用如下的示例性转换公式确定所述的目标分辨率：
其中，psnrframe为所述图像帧的质量指标，在模型推理阶段即为所述图像帧的目标质量指标，psnrlowest为质量指标的最低门槛；resframe为所述图像帧的编码分辨率，reslowest为分辨率的最低门槛，例如取值为180，n为权重，一个实施例中取值为2，本领域技术人员可以根据实际情况进行微调。
至此，将所述图像帧的编码分辨率及目标质量指标输入以上的公式，便可确定出其相应的目标分辨率yreal。
步骤S1213、比较所述目标分辨率与所述理想分辨率的大小，当所述目标分辨率大于所述理想分辨率时，确定所述图像帧的缩放操作的接口类型为硬件接口类型，否则为软件接口类型。
确定了所述图像帧相对应的目标分辨率及理想分辨率之后，将两者进行比较，即比较yreal>yideal是否成立，若成立，表明需要达到的目标分辨率大于所述理想分辨率，在两者基于相同目标质量指标确定的情况下，目标分辨率越大，往 往相应的成像画质越差，因而更适于采用硬件缩放接口来对图像帧实施缩放，所以确定所述图像帧的缩放操作相对应的接口类型为硬件接口类型，以便采用硬件缩放接口来对所述图像帧进行缩放操作以节省系统开销、提升运算效率；若不成立，表明目标分辨率小于或等于所述理想分辨率，在两者基于相同目标质量指标确定的情况下，目标分辨率越小，往往相应的成像画质越佳，因而可以借助软件算法的技术优势来确保缩放操作后获得优质的成像效果，相应将所述图像帧的缩放操作的接口类型确定为软件接口类型。
根据以上实施例，一方面，利用预先训练出的分辨率预测模型，可以根据习得的经验知识，基于图像帧的目标质量指标，预测出其相对应的理想分辨率，另一方面，可以基于图像帧的编码分辨率及其目标质量指标确定出图像帧所需缩放达到的目标分辨率，理想分辨率与目标分辨率均基于图像帧的目标质量指标来确定，其中理想分辨率起到决策阈值的作用，在目标分辨率大于理想分辨率时决策使用硬件接口类型实施缩放操作，反之采用软件接口类型实施缩放操作，可以区分出选用硬件缩放操作和软件缩放操作相对应的场景，平衡系统开销和成像画质，确保直播视频流在播放过程中，利用有限的硬件运算资源，在整体上能获得较为稳定的画质效果。
在本申请任意实施例的基础上，请参阅图6，根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，包括：
步骤S1221、基于所述图像帧的编码分辨率，在预设的接口映射表中查询出所述编码分辨率相对应的映射子集，所述映射子集包含不同目标质量指标及其相适配的接口类型；
不同于前一实施例，本实施例中，采用接口映射表来为所述图像帧确定其相应的接口类型。所述的接口映射表是表征了给定的编码分辨率、目标质量指标相对应的接口类型的映射表格，包含有多个数据记录，通过各个数据记录，给出相同编码分辨率不同目标质量指标之下对应的接口类型，以及不同编码分辨率相同目标质量指标之下对应的接口类型，示例性的接口映射表如下：
参考以上示例性表格可知，对于一个图像帧来说，利用其编码分辨率例如360，从以上的映射表格中查询出其相对应的映射子集如下：
该映射子集便仅包含特定的编码分辨率下，不同目标质量指标相对应的接口类型。在以上示例性表格中，所述接口类型采用0和1表示，其中0表示接口类型为硬件接口类型，1表示接口类型为软件接口类型。
步骤S1222、基于所述图像帧的目标质量指标，从所述映射子集中查询出与所述目标质量指标相对应的接口类型。
进一步，采用所述图像帧的目标质量指标继续在以上得到的映射子集中查询，便可确定出其相应的接口类型，例如，当所述图像帧的目标质量指标为34时，则确定出接口类型为值0，即使用硬件接口类型，为45时，即确定出接口类型为值1，即使用软件缩放类型。
本实施例中，当所述图像帧的编码分辨率和目标质量指标并不等于接口映射表格中的具体分辨率数值时，可以采用近似确定的方式，采用与所述编码分辨率和目标质量指标差值最小的对应编码分辨率和目标质量指标来为所述图像帧确定其缩放操作所采用的接口类型即可。
根据以上实施例可以看出，基于查表的方式来确定直播视频流中的图像帧进行缩放操作时所需的接口类型，其查询过程更为高效便捷，运算量低。
在本申请任意实施例的基础上，请参阅图7，调用所述接口类型相对应的缩放接口对所述图像帧进行缩放操作后渲染显示到直播间界面的视频播放窗口中，包括：
步骤S1310、当所述接口类型为硬件接口类型时，调用硬件缩放接口对所述图像帧进行缩放操作；
不同的终端设备，其硬件的缩放能力不同，但都会提供相应的硬件缩放接口供外部调用，据此，当所述图像帧所需采用的接口类型为硬件接口类型时，只需对应调用相应终端设备上的硬件缩放接口，对所述图像帧实施缩放操作即可。
步骤S1320、当所述接口类型为软件接口类型时，调用软件缩放接口对所述图像帧进行缩放操作；
相反，当所述图像帧所需采用的接口类型为软件接口类型时，由于直播间应用程序可以自带各种软件算法，或者终端设备的操作系统中也可能提供相关软件算法，这种情况下，便相应调用预设的软件缩放接口对所述图像帧进行缩放操作，以便应用相应的软件算法，将所述图像帧缩放到特定分辨率。
可选的软件算法，包括但不限于增加了后滤波处理环节的第一软件缩放算法、基于深度学习模型的第二软件缩放算法(例如超分辨率模型)，基于bilinear实现的第三软件缩放算法(双线性插值算法)、基于bicubic实现的第四软件缩放算法(双三次插值算法)，不同的软件缩放算法对硬件算力等级要求不同，可以按需灵活使用。
步骤S1330、将缩放操作后的所述图像帧渲染显示到直播间界面的视频播放窗口中。
在实现了对所述图像帧的缩放操作后，应用图像渲染算法，将其渲染显示到直播间界面的视频播放窗口中，便实现将所述图像帧在视频播放窗口的播放，直播视频流中的各个图像帧顺次被渲染显示，便可在直播间的视频播放窗口中呈现 所述直播视频流的播放内容。
根据以上实施例可知，适应不同接口类型，调用相应的硬件接口类型或软件接口类型对直播视频流中的图像帧进行缩放操作后，均可显示到直播间应用程序中，从而构成对直播视频流的播放，使直播视频流在自适应选定的接口类型的缩放操作下，能够获得稳定的画质。
在本申请任意实施例的基础上，请参阅图7，当所述接口类型为软件接口类型时，调用软件缩放接口对所述图像帧进行缩放操作，包括：
步骤S1321、判断所述图像帧的缩放比率是否为二倍率，当为二倍率时，采用第一软件缩放接口实施对所述图像帧的缩放操作以消除所述图像帧在缩放过程中产生的摩尔纹，所述缩放比率为所述图像帧的目标分辨率与编码分辨率的比值；
软件缩放算法面对一个复杂的问题，二倍率的缩放效果通常不会太差，但是非二倍率的缩放算法在很多情况下会有严重的摩尔纹，为了处理摩尔纹需要做后处理。同时，如果视频本身没有摩尔纹，后处理会让视频编码更平坦，观看体验更差。所以软件缩放算法的选择，是否进行后处理是一个比较大问题。
为此，可以先将所述图像帧在前文实施例中确定的目标分辨率与所述图像帧固有的编码分辨率相除确定出缩放比率，然后判断该缩放比率是否为二倍率，如果是二倍率，采用增加了后滤波处理环节的第一软件缩放算法所提供的第一软件缩放接口对所述图像帧进行缩放处理，以便缩放后获得的图像帧，能够消除缩放过程中产生的摩尔纹，确保图像帧的画质。
步骤S1322、当所述缩放比率为非二倍率时，判断当前设备所属的硬件算力等级，根据硬件算力等级调用相应等级的软件缩放接口实施对所述图像帧的缩放操作。
当所述缩放比率并不是二倍率时，无需考虑摩尔纹的影响，这种情况下，可以按需选用其他软件缩放算法来实施对所述图像帧的缩放操作。
考虑到不同的软件缩放算法对硬件算力的要求不同，可以在直播间应用程序中提供多种对应不同硬件算力的软件缩放算法，例如前文所述的第二、三、四软件缩放算法，然后，通过确定直播间应用程序所在的终端设备的硬件算力等级来匹配相应的软件缩放算法，调用相应的软件缩放算法所提供的软件缩放接口来实施缩放操作。
具体而言，可以先获取当前终端设备的品牌、型号，然后查表确定其对应的硬件算力等级，或者先获取当前终端设备的CPU和闪存的型号再查表确定其对应的硬件算力等级均可。
所述硬件算力等级，一种实施例中，预先将其规划为三个等级，分别对应算力性能上属于高、中、低端的终端设备，然后将这些终端设备的品牌、型号，或者CPU、闪存等，与相应的硬件算力等级相映射存储为表格，以供查询。将不同硬件算力等级与不同软件缩放算法也预先建立关联，例如，高端性能的终端设备可以采用本申请示例性给出的第二软件缩放算法，以便基于深度学习模型实施对图像帧的缩放操作；中端性能的终端设备可以采用本申请示例性给出的第三软件缩放算法，即bilinear算法，以便平衡缩放效果与算力要求；低端性能的终端设备可以采用本申请示例性给出的第四软件缩放算法，即bicubic算法等，以利用该算法运算量要求较低的特点。
根据以上实施例可知，对于所述图像帧的缩放操作被确定为通过软件接口类型来实施缩放时，还可以区别多种情形，包括二倍率和非二倍率，以及不同硬件算力等级等条件，决策使用相应的软件缩放算法，从而克服缩放操作可能导致的各 种潜在问题，例如二倍率缩放情况下的出现摩尔纹的问题，硬件算力不足导致直播视频流播放卡顿的问题等，使直播视频流播放过程更为流畅，画质也更为稳定。
在本申请任意实施例的基础上，请参阅图9，将缩放操作后的所述图像帧渲染显示到直播间界面的视频播放窗口中之前，包括：
步骤S2100、判断缩放操作后的所述图像帧是否超出所述视频播放窗口的边界，当超过所述边界时，对所述图像帧进行人脸检测以确定出人脸区域；
由于媒体服务器在编码直播视频流的图像帧的过程中会动态决定图像帧的编码分辨率，导致终端设备针对同一直播视频流的不同图像帧进行缩放时，在保持目标质量指标维持大致相同的情况下，各个图像帧的缩放后的画幅可能有所不同，虽然最终画幅仍受制于视频流播放窗口，但也可进行一些预处理，通过预处理来突出重点，使得直播视频流即使产生分辨率的波动，其中的重点区域仍然是视频播放窗口的主要显示区域，使用户基本察觉不到分辨率变化所带来的影响。
为此，当对直播视频流的图像帧实施缩放操作后，可以先计算出缩放后的图像帧的画幅是否超出直播间界面中的视频播放窗口的边界，当超出其边界时，借助人脸检测模型对其实施人脸检测，确定出人脸图像所在的人脸区域，不难理解，人脸区域所在的人脸图像，通常是直播用户的面部，是观众用户期望看到的重点内容。人脸检测模型所确定出的人脸区域，通常是一个坐标信息，通过左上角和右下角的坐标表示出一个选择框。
步骤S2200、以所述人脸区域为中心对所述图像帧进行裁剪，使所述图像帧的尺寸适配所述视频播放窗口的尺寸。
进一步，以所述人脸区域的选择框的几何中心作为要裁剪的画幅的中心，对应所述视频播放窗口的尺寸，对所述缩放后的图像帧进行相应尺度的裁剪，裁剪出的图像帧便以所述人脸区域所覆盖的人脸图像为中心，当其被渲染显示到所述视频播放窗口时，所述人脸图像相对置中显示，当连续各个图像帧的编码分辨率发生变化时，由于裁剪的关系，人脸图像保持居中，用户不易察觉到分辨率变化所带来的影响。
根据以上实施例可知，在将直播视频流中的图像帧缩放后，通过人脸检测确定图像帧中的人脸图像，以人脸图像居中进行图像裁剪再渲染显示，可以过滤掉直播视频流中因编码分辨率大幅变动所导致的图像变化幅度过大的影响，确保用户能够获得良好的视频观看体验。
在本申请任意实施例的基础上，请参阅图10，获取直播视频流之前，包括：
步骤S3100、确定当前图像帧的主观评估参数，所述主观评估参数包括适于对当前图像帧编码的参考分辨率，及按照所述参考分辨率编码可预期获得的主观质量指标；
当媒体服务器获得直播用户上传的直播视频流后，可对其进行解码，从而获得其中的各个图像帧，逐一调用其中的各个图像帧作为当前图像帧进行编码处理。对于视频流中的每个当前图像帧，可以参考图像主观质量评价方法确定其相应的主观质量指标。
一种实施例中，预先应用图像主观质量评价方法，提供多个不同分辨率的视频内容，播放给观众观看，然后从多个预设维度获取各个观众的评分，根据观众给定的评分进行加权量化为一个主观质量指标，并将多个分辨率相对应的主观质量指标建立映射关系表格，然后用做本申请确定当前图像帧相匹配的参考分辨率和主观质量指标的参考。据此，当需要确 定当前图像帧相匹配的参考分辨率及其相应的主观质量指标时，根据当前图像帧相对于其在先图像帧的图像复杂度的变动，在其在先图像帧相对应的参考分辨率的基础上做相应调整，便可评估出当前图像帧相对应的参考分辨率，根据评估出的参考分辨率，便可利用所述映射关系数据确定出其相对应的主观质量指标，从而获得一个参考分辨率与一个对应的主观质量指标，构成当前图像帧相对应的主观评估参数。
由此可见，所述的参考分辨率，是作为参考的中间数据，构成指示当前图像帧的分辨率编码条件，相应的，所述的主观质量指标，是指在以该参考分辨率为编码条件的情况下，可预测获得的图像质量主观评价结果。
一种实施例中，将根据图像主观质量评价方法确定的映射关系表格进行数据拟合获得相对应的主观曲线函数，后续便可根据每一当前图像帧相应确定的参考分辨率快速获得其相对应的主观质量指标而获得其主观评估参数。这种情况下，由于主观曲线函数经拟合后具有线性的特点，可使据此确定的主观评估参数可为平滑。
步骤S3200、将编码器的出口带宽输入参数优化模型，获得所述出口带宽下表征编码分辨率与目标质量指标之间映射关系的近似函数；
编码器的出口带宽是客观条件，受网络条件和/或硬件条件所限制，因此具有相对确定性。本实施例中，可将该出口带宽输入参数优化模型中，控制参数优化模型仿真出求解编码器的最佳分辨率的近似函数。
所述参数优化模型通过预先建模实现。示例而言，考虑到同一网络条件下直播视频流传输过程中的帧率相对稳定，而编码分辨率与客观质量指标跟随编码器的出口带宽变化的弹性较大，故通过对特定出口带宽之下，不同视频流的客观质量指标和编码分辨率进行数据拟合，由拟合结果可知视频流的编码分辨率与客观质量指标之间服从自然对数分布。根据该原理，可建模实现所述的参数优化模型。
示例性的参数优化模型按照如下公式构建：
ypsnr＝A*ln(resolution)+B
其中，ypsnr为示例性的客观质量指标，resolution为分辨率，A为可学习权重参数，B为可学习偏置参数，两者构成模型的可学习参数。根据该公式可知，该参数优化模型可以编码器输出的图像帧的客观质量指标对其权重参数和偏置参数进行修正，实现迭代训练，使其不断逼近收敛。
根据参数优化模型的示例性建模原理可知，该参数优化模型的核心是构建出口带宽、编码分辨率、客观质量指标之间的映射关系相对应的数据曲线，从而可以根据出口带宽预测编码分辨率与客观质量指标的映射关系，也即确定一个近似函数。在确定出的近似函数上进一步确定一个相应的点，便可确定出具体的编码分辨率与客观质量指标。由此可见，特定出口带宽相对应的数据曲线本质上反映着编码分辨率与客观质量指标之间在该特定出口带宽之下的映射关系数据集合，该映射关系数据集合通过参数优化模型获得的近似函数表示，据此，根据一个出口带宽可获得这一近似函数。
需要指出的是，步骤S3100与步骤S3200是可以并发执行，也可交换执行顺序，并不因本申请中所列的先后而受限。
此外，所述参数优化模型，可建模为运算资源利用率较低的机器学习模型，也可建模为更具智能的深度学习模型，只要能根据以上原理实现本申请所揭示的功能即可。
步骤S3300、应用所述近似函数求取所述主观评估参数相对应的数值逼近点，获得所述数值逼近点相对应的编码分 辨率和目标质量指标；
形象而言，所述近似函数表征在特定出口带宽下的编码分辨率与客观质量指标之间的线性关系的映射关系数据，反映在直角坐标系上为一条数据曲线，而当前图像帧相对应的主观评估参数，即其参考分辨率及该参考分辨率相对应的主观质量指标，则可反映为直角坐标系上的一个点。
据此，一种实施例中，通过计算直角坐标系上主观参数模型相对应的点到所述近似函数的数据曲线之间的距离最短处，确定坐落在该数据曲线上的最短距离的点，便为该数据曲线上的数值逼近点，该数值逼近点对应着一个编码分辨率与一个客观质量指标之间的映射关系数据，该编码分辨率便可被确定为对相应的图像帧进行编码的最佳分辨率，该客观质量指标便被视为对应编码预期的目标质量指标。
另一实施例中，根据图像主观质量评价方法确定的参考分辨率与主观质量指标之间的映射关系数据，预先经数据拟合而获得主观曲线函数，这种情况下，可以计算出该主观曲线函数与该近似函数的交点，该交点即为所述的数值逼近点，由此也可获得相应的编码分辨率作为最佳分辨率。
本申请所确定的最佳分辨率，被输入所述编码器中，用于控制该编码器以该最佳分辨率对当前图像帧进行编码，以便将当前图像帧编码于所述直播视频流中。
步骤S3400、将所述目标质量指标作为所述当前图像帧相对应的附加增强信息，按照所述编码分辨率将所述当前图像帧编码到直播视频流中推送至直播间。
确定出当前图像帧相对应的编码分辨率后，便按照该编码分辨率对所述图像帧进行相应的分辨率转换处理，将转换后的图像帧编码到直播视频流中，以便推送至直播间，为了方便接收该直播视频流的终端设备能够根据所述编码分辨率和目标质量指标决策对所述当前图像帧进行缩放操作所使用的接口类型，进一步将所述当前图像帧的目标质量指标包含于所述直播视频流的附加增强信息中，与所述当前图像帧建立对应关系，随同直播视频流送达直播间的终端设备。
根据以上的实施例，可以看出，相对于现有技术，本申请在编码器的出口带宽的约束下，一方面通过确定直播视频流中的当前图像帧相对应的主观评估参数，所述主观评估参数包括适于对当前图像帧编码的参考分辨率，及按照所述参考分辨率率编码可预期获得的主观质量指标，评估出分辨率变动与图像质量变动之间的对应关系；另一方面由参数优化模型预测出表征编码分辨率与客观质量指标之间映射关系的近似函数，在此基础上，从该近似函数中获取所述主观评估参数相对应的数值逼近点，然后根据该数值逼近点确定出编码所需的最佳分辨率以控制编码器对当前图像帧进行编码输出，并将关联确定的客观质量指标作为所述当前图像帧相对应的目标质量指标，随直播视频流送达终端设备，使编码生成的视频流的客观质量指标服从主观质量指标的调谐，并且还能根据目标质量指标指导图像帧的缩放显示效果，从而，在确保视频流稳定流畅输出的同时，改善视频图像的总体质量观感。
请参阅图11，根据本申请的一个方面提供的一种视频显示控制装置，包括数据获取模块1100、接口匹配模块1200，以及缩放显示模块1300，其中，所述数据获取模块1100，设置为获取直播视频流，所述直播视频流中包含多个图像帧及其目标质量指标；所述接口匹配模块1200，设置为根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，所述接口类型为硬件接口类型或软件接口类型；所述缩放显示模块1300，设置为调用所述接 口类型相对应的缩放接口对所述图像帧进行缩放操作后渲染显示到直播间界面的视频播放窗口中。
在本申请任意实施例的基础上，所述接口匹配模块1200，包括：模型推断单元，设置为将所述图像帧的目标质量指标输入分辨率预测模型确定出理想分辨率；目标换算单元，设置为应用预设转换公式计算出所述图像帧在其既有目标质量指标下需要基于其编码分辨率缩放达到的目标分辨率；接口选择单元，设置为比较所述目标分辨率与所述理想分辨率的大小，当所述目标分辨率大于所述理想分辨率时，确定所述图像帧的缩放操作的接口类型为硬件接口类型，否则为软件接口类型。
在本申请任意实施例的基础上，所述接口匹配模块1200，包括：分辨率查询单元，设置为基于所述图像帧的编码分辨率，在预设的接口映射表中查询出所述编码分辨率相对应的映射子集，所述映射子集包含不同目标质量指标及其相适配的接口类型；指标查询单元，设置为基于所述图像帧的目标质量指标，从所述映射子集中查询出与所述目标质量指标相对应的接口类型。
在本申请任意实施例的基础上，所述缩放显示模块1300，包括：硬件缩放单元，设置为当所述接口类型为硬件接口类型时，调用硬件缩放接口对所述图像帧进行缩放操作；软件缩放单元，设置为当所述接口类型为软件接口类型时，调用软件缩放接口对所述图像帧进行缩放操作；渲染显示单元，设置为将缩放操作后的所述图像帧渲染显示到直播间界面的视频播放窗口中。
在本申请任意实施例的基础上，所述软件缩放单元，包括：常规处理子单元，设置为判断所述图像帧的缩放比率是否为二倍率，当为二倍率时，采用第一软件缩放接口实施对所述图像帧的缩放操作以消除所述图像帧在缩放过程中产生的摩尔纹，所述缩放比率为所述图像帧的目标分辨率与编码分辨率的比值；例外处理子单元，设置为当所述缩放比率为非二倍率时，判断当前设备所属的硬件算力等级，根据硬件算力等级调用相应等级的软件缩放接口实施对所述图像帧的缩放操作。
在本申请任意实施例的基础上，所述缩放显示模块1300，包括：边界检测单元，设置为判断缩放操作后的所述图像帧是否超出所述视频播放窗口的边界，当超过所述边界时，对所述图像帧进行人脸检测以确定出人脸区域；图像裁剪单元，设置为以所述人脸区域为中心对所述图像帧进行裁剪，使所述图像帧的尺寸适配所述视频播放窗口的尺寸。
在本申请任意实施例的基础上，本申请的视频显示控制装置，还包括：
参数确定模块，设置为确定当前图像帧的主观评估参数，所述主观评估参数包括适于对当前图像帧编码的参考分辨率，及按照所述参考分辨率编码可预期获得的主观质量指标；
函数获取模块，设置为将编码器的出口带宽输入参数优化模型，获得所述出口带宽下表征编码分辨率与目标质量指标之间映射关系的近似函数；
数值确定模块，设置为应用所述近似函数求取所述主观评估参数相对应的数值逼近点，获得所述数值逼近点相对应的编码分辨率和目标质量指标；
编码推送模块，设置为将所述目标质量指标作为所述当前图像帧相对应的附加增强信息，按照所述编码分辨率将所述当前图像帧编码到直播视频流中推送至直播间。
本申请的另一实施例还提供一种视频显示控制设备。如图12所示，视频显示控制设备的内部结构示意图。该视频显示控制设备包括通过系统总线连接的处理器、计算机可读存储介质、存储器和网络接口。其中，该视频显示控制设备的计算机可读的非易失性可读存储介质，存储有操作系统、数据库和计算机可读指令，数据库中可存储有信息序列，该计算机可读指令被处理器执行时，可使得处理器实现一种视频显示控制方法。
该视频显示控制设备的处理器用于提供计算和控制能力，支撑整个视频显示控制设备的运行。该视频显示控制设备的存储器中可存储有计算机可读指令，该计算机可读指令被处理器执行时，可使得处理器执行本申请的视频显示控制方法。该视频显示控制设备的网络接口用于与终端连接通信。
本领域技术人员可以理解，图12中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的视频显示控制设备的限定，在一实施例中视频显示控制设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
本实施方式中处理器用于执行图11中的各个模块的具体功能，存储器存储有执行上述模块或子模块所需的程序代码和各类数据。网络接口用于实现用户终端或服务器之间的数据传输。本实施方式中的非易失性可读存储介质中存储有本申请的视频显示控制装置中执行所有模块所需的程序代码及数据，服务器能够调用服务器的程序代码及数据执行所有模块的功能。
本申请还提供一种存储有计算机可读指令的非易失性可读存储介质，计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行本申请任一实施例的视频显示控制方法的步骤。
本申请还提供一种计算机程序产品，包括计算机程序/指令，该计算机程序/指令被一个或多个处理器执行时实现本申请任一实施例所述方法的步骤。
本领域普通技术人员可以理解，实现本申请上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，该计算机程序可存储于一非易失性可读存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，前述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)等计算机可读存储介质，或随机存储记忆体(Random Access Memory，RAM)等。
综上所述，本申请以推送直播视频流时给定的图像帧的目标质量指标为标准，基于该目标质量指标匹配接口类型，由接口类型相对应缩放接口实施对所述图像帧的缩放操作，确保所述图像帧缩放后的图像效果能够最大化程度地符合所述目标质量指标所建立的标准，使直播视频流在直播间界面中播放时，能够获得相对应较为优质、稳定的画面效果，能够提升网络直播用户的使用体验。

Claims (11)

  1. 一种视频显示控制方法，其中，包括：
    获取直播视频流，所述直播视频流中包含多个图像帧及其目标质量指标；
    根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，所述接口类型为硬件接口类型或软件接口类型；
    调用所述接口类型相对应的缩放接口对所述图像帧进行缩放操作后渲染显示到直播间界面的视频播放窗口中。
  2. 根据权利要求1所述的视频显示控制方法，其中，根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，包括：
    将所述图像帧的目标质量指标输入分辨率预测模型确定出理想分辨率；
    应用预设转换公式计算出所述图像帧在其既有目标质量指标下需要基于其编码分辨率缩放达到的目标分辨率。
    比较所述目标分辨率与所述理想分辨率的大小，当所述目标分辨率大于所述理想分辨率时，确定所述图像帧的缩放操作的接口类型为硬件接口类型，否则为软件接口类型。
  3. 根据权利要求1所述的视频显示控制方法，其中，根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，包括：
    基于所述图像帧的编码分辨率，在预设的接口映射表中查询出所述编码分辨率相对应的映射子集，所述映射子集包含不同目标质量指标及其相适配的接口类型；
    基于所述图像帧的目标质量指标，从所述映射子集中查询出与所述目标质量指标相对应的接口类型。
  4. 根据权利要求1至3中任意一项所述的视频显示控制方法，其中，调用所述接口类型相对应的缩放接口对所述图像帧进行缩放操作后渲染显示到直播间界面的视频播放窗口中，包括：
    当所述接口类型为硬件接口类型时，调用硬件缩放接口对所述图像帧进行缩放操作；
    当所述接口类型为软件接口类型时，调用软件缩放接口对所述图像帧进行缩放操作；
    将缩放操作后的所述图像帧渲染显示到直播间界面的视频播放窗口中。
  5. 根据权利要求4所述的视频显示控制方法，其中，当所述接口类型为软件接口类型时，调用软件缩放接口对所述图像帧进行缩放操作，包括：
    判断所述图像帧的缩放比率是否为二倍率，当为二倍率时，采用第一软件缩放接口实施对所述图像帧的缩放操作以消除所述图像帧在缩放过程中产生的摩尔纹，所述缩放比率为所述图像帧的目标分辨率与编码分辨率的比值；
    当所述缩放比率为非二倍率时，判断当前设备所属的硬件算力等级，根据硬件算力等级调用相应等级的软件缩放接口实施对所述图像帧的缩放操作。
  6. 根据权利要求1至4中任意一项所述的视频显示控制方法，其中，将缩放操作后的所述图像帧渲染显示到直播间界面的视频播放窗口中之前，包括：
    判断缩放操作后的所述图像帧是否超出所述视频播放窗口的边界，当超过所述边界时，对所述图像帧进行人脸检测以确定出人脸区域；
    以所述人脸区域为中心对所述图像帧进行裁剪，使所述图像帧的尺寸适配所述视频播放窗口的尺寸。
  7. 根据权利要求1至6中任意一项所述的视频显示控制方法，其中，获取直播视频流之前，包括：
    确定当前图像帧的主观评估参数，所述主观评估参数包括适于对当前图像帧编码的参考分辨率，及按照所述参考分辨率编码可预期获得的主观质量指标；
    将编码器的出口带宽输入参数优化模型，获得所述出口带宽下表征编码分辨率与目标质量指标之间映射关系的近似函数；
    应用所述近似函数求取所述主观评估参数相对应的数值逼近点，获得所述数值逼近点相对应的编码分辨率和目标质量指标；
    将所述目标质量指标作为所述当前图像帧相对应的附加增强信息，按照所述编码分辨率将所述当前图像帧编码到直播视频流中推送至直播间。
  8. 一种视频显示控制装置，其中，包括：
    数据获取模块，设置为获取直播视频流，所述直播视频流中包含多个图像帧及其目标质量指标；
    接口匹配模块，设置为根据所述图像帧的编码分辨率及其相应的目标质量指标确定所述图像帧的缩放操作相对应的接口类型，所述接口类型为硬件接口类型或软件接口类型；
    缩放显示模块，设置为调用所述接口类型相对应的缩放接口对所述图像帧进行缩放操作后渲染显示到直播间界面的视频播放窗口中。
  9. 一种视频显示控制设备，包括中央处理器和存储器，其中，所述中央处理器用于调用运行存储于所述存储器中的计算机程序以执行如权利要求1至7中任意一项所述的方法的步骤。
  10. 一种非易失性可读存储介质，其中，其以计算机可读指令的形式存储有计算机程序，所述计算机程序被计算机调用运行时，执行如权利要求1至7中任意一项所述的方法的步骤。
  11. 一种计算机程序产品，其中，包括计算机程序/指令，所述计算机程序/指令被处理器调用时，执行如权利要求1至7中任意一项所述的方法的步骤。
PCT/CN2024/070219 2023-01-18 2024-01-02 视频显示控制方法及其装置、设备、介质、产品 Ceased WO2024152893A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310084572.1 2023-01-18
CN202310084572.1A CN116095355B (zh) 2023-01-18 2023-01-18 视频显示控制方法及其装置、设备、介质、产品

Publications (1)

Publication Number Publication Date
WO2024152893A1 true WO2024152893A1 (zh) 2024-07-25

Family

ID=86204223

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/070219 Ceased WO2024152893A1 (zh) 2023-01-18 2024-01-02 视频显示控制方法及其装置、设备、介质、产品

Country Status (2)

Country Link
CN (1) CN116095355B (zh)
WO (1) WO2024152893A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116095355B (zh) * 2023-01-18 2024-06-21 百果园技术(新加坡)有限公司 视频显示控制方法及其装置、设备、介质、产品
CN118450195A (zh) * 2024-04-30 2024-08-06 广州光晨信息科技有限公司 视频图像画质调节方法、系统、计算机设备及存储介质
CN119421047A (zh) * 2024-11-04 2025-02-11 维沃移动通信有限公司 图像处理方法、装置、设备及介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106713937A (zh) * 2016-12-30 2017-05-24 广州虎牙信息科技有限公司 视频播放控制方法、装置及终端设备
CN107707976A (zh) * 2017-09-29 2018-02-16 维沃移动通信有限公司 一种视频解码方法及移动终端
US20180082429A1 (en) * 2016-09-16 2018-03-22 Dolby Laboratories Licensing Corporation Dynamic adjustment of frame rate conversion settings
CN108632667A (zh) * 2017-03-21 2018-10-09 腾讯科技（深圳）有限公司 一种多媒体文件的解码方法及终端
CN113055744A (zh) * 2020-08-31 2021-06-29 惠州华阳通用电子有限公司 一种视频解码方法及装置
CN114900692A (zh) * 2022-04-28 2022-08-12 有半岛(北京)信息科技有限公司 视频流帧率调整方法及其装置、设备、介质、产品
CN116095355A (zh) * 2023-01-18 2023-05-09 百果园技术(新加坡)有限公司 视频显示控制方法及其装置、设备、介质、产品

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100713012B1 (ko) * 2004-10-18 2007-05-02 한국정보통신대학교 산학협력단 다차원 비트율 조절에 따른 비디오 트랜스코딩 방법 및 그장치
CN101103632A (zh) * 2005-01-07 2008-01-09 皇家飞利浦电子股份有限公司 利用动态地基于法向流的量化步长来处理视频信号的方法
KR100954303B1 (ko) * 2008-01-22 2010-04-21 갤럭시아커뮤니케이션즈 주식회사 영상의 공간 해상도 제어 시스템 및 이를 이용한 영상 전송제어 방법
US9516305B2 (en) * 2012-09-10 2016-12-06 Apple Inc. Adaptive scaler switching
CN103281510A (zh) * 2012-11-06 2013-09-04 王彤 将数码成像装置的数码图像传输至主机的方法
TWI496114B (zh) * 2012-11-23 2015-08-11 Univ Nat Taiwan 影像追蹤裝置及其影像追蹤方法
CN106797439B (zh) * 2014-09-12 2020-02-07 索尼半导体解决方案公司 图像处理装置，图像处理方法，以及存储介质
US10313417B2 (en) * 2016-04-18 2019-06-04 Qualcomm Incorporated Methods and systems for auto-zoom based adaptive video streaming
CN110515452B (zh) * 2018-05-22 2022-02-22 腾讯科技（深圳）有限公司 图像处理方法、装置、存储介质和计算机设备
CN109862388A (zh) * 2019-04-02 2019-06-07 网宿科技股份有限公司 直播视频集锦的生成方法、装置、服务器及存储介质
CN113744117B (zh) * 2020-05-29 2025-06-06 Oppo广东移动通信有限公司 多媒体处理芯片、电子设备及动态图像处理方法
CN112634136B (zh) * 2020-12-24 2023-05-23 华南理工大学 一种基于图像特征快速拼接的图像超分辨率方法及其系统
CN114679591A (zh) * 2021-12-30 2022-06-28 广州方硅信息技术有限公司 直播间的视频比例切换方法、装置、介质以及计算机设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180082429A1 (en) * 2016-09-16 2018-03-22 Dolby Laboratories Licensing Corporation Dynamic adjustment of frame rate conversion settings
CN106713937A (zh) * 2016-12-30 2017-05-24 广州虎牙信息科技有限公司 视频播放控制方法、装置及终端设备
CN108632667A (zh) * 2017-03-21 2018-10-09 腾讯科技（深圳）有限公司 一种多媒体文件的解码方法及终端
CN107707976A (zh) * 2017-09-29 2018-02-16 维沃移动通信有限公司 一种视频解码方法及移动终端
CN113055744A (zh) * 2020-08-31 2021-06-29 惠州华阳通用电子有限公司 一种视频解码方法及装置
CN114900692A (zh) * 2022-04-28 2022-08-12 有半岛(北京)信息科技有限公司 视频流帧率调整方法及其装置、设备、介质、产品
CN116095355A (zh) * 2023-01-18 2023-05-09 百果园技术(新加坡)有限公司 视频显示控制方法及其装置、设备、介质、产品

Also Published As

Publication number Publication date
CN116095355B (zh) 2024-06-21
CN116095355A (zh) 2023-05-09

Similar Documents

Publication Publication Date Title
CN114554211B (zh) 内容自适应视频编码方法、装置、设备和存储介质
CN114900692B (zh) 视频流帧率调整方法及其装置、设备、介质、产品
WO2024152893A1 (zh) 视频显示控制方法及其装置、设备、介质、产品
TWI743919B (zh) 視訊處理裝置及視訊串流的處理方法
CN112437301B (zh) 一种面向视觉分析的码率控制方法、装置、存储介质及终端
WO2021068598A1 (zh) 共享屏幕的编码方法、装置、存储介质及电子设备
CN110620924B (zh) 编码数据的处理方法、装置、计算机设备及存储介质
WO2022000298A1 (en) Reinforcement learning based rate control
WO2020042269A1 (zh) 一种编码过程中的码率调整方法及装置
CN105991364B (zh) 一种用户感知评估方法及装置
CN112672149B (zh) 一种视频处理方法、装置、存储介质及服务器
CN114245209B (zh) 视频分辨率确定、模型训练、视频编码方法及装置
CN114071121B (zh) 影像品质评估装置及其影像品质评估方法
CN111524110B (zh) 视频质量的评价模型构建方法、评价方法及装置
CN111476866B (zh) 视频优化与播放方法、系统、电子设备及存储介质
CN116471262A (zh) 视频质量评估方法、装置、设备、存储介质及程序产品
CN110677728B (zh) 播放视频的方法、装置、设备及存储介质
CN117425047A (zh) 客户端视频码率的动态调整方法、存储介质及电子设备
CN117201792A (zh) 视频编码方法、装置、电子设备及计算机可读存储介质
WO2018192518A1 (zh) 数据处理方法、装置及存储介质
CN121442112A (zh) 一种视频编码方法、装置、设备及存储介质
HK40092255A (zh) 视频处理方法、装置、存储介质及计算机设备
HK40086889A (zh) 一种图像处理方法、装置及设备、存储介质、产品
WO2026011968A1 (zh) 视频编码方法、装置、设备以及可读存储介质
CN120881281A (zh) 视频转码处理方法及装置、计算机可读存储介质、设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24744069

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 24744069

Country of ref document: EP

Kind code of ref document: A1