WO2020038128A1 - Video processing method and device, electronic device and computer readable medium - Google Patents

Abstract

The present application relates to the technical field of video processing, and disclosed thereby are a video processing method and device, an electronic device and a computer readable medium. The method comprises: acquiring multiple frames of image data to be rendered that correspond to a video file; storing the multiple frames of image data in an off-screen rendering buffer; optimizing the multiple frames of image data in the off-screen rendering buffer according to a preset video enhancement algorithm; transmitting the optimized multiple frames of image data to a frame buffer corresponding to a screen; and the frame buffer reading the optimized multiple frames of image data and displaying the same on the screen. Therefore, the image quality when a video file is played back may be improved by means of optimizing the video file in another buffer, thus improving the user experience.

Description

Video processing method, device, electronic equipment and computer-readable medium

Cross-reference to related applications

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on August 23, 2018 under the application number CN201810969497.6 and entitled "Video Processing Method, Apparatus, Electronic Equipment, and Computer-readable Media". Citations are incorporated in this application.

Technical field

The present application relates to the technical field of video processing, and more particularly, to a video processing method, device, electronic device, and computer-readable medium.

Background technique

With the development of electronic technology and information technology, more and more devices can play video. During the video playback process, the device needs to perform operations such as decoding, rendering, and compositing the video, and then display it on the display screen. However, in the existing video playback technology, the picture quality effect of the played video can no longer meet the user's requirements. Demand, resulting in poor user experience.

Summary of the Invention

The present application proposes a video processing method, apparatus, electronic device, and computer-readable medium to improve the above defects.

In a first aspect, an embodiment of the present application provides a video processing method, which is applied to an image processor of an electronic device. The electronic device further includes a screen. The method includes: acquiring multi-frame image data to be rendered corresponding to a video file. Storing the multi-frame image data in an off-screen rendering buffer; optimizing the multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm; sending the optimized multi-frame image data to all The frame buffer corresponding to the screen is read; the optimized multi-frame image data is read from the frame buffer and displayed on the screen.

In a second aspect, an embodiment of the present application further provides a video processing apparatus, which is applied to an image processor of an electronic device, and the electronic device further includes a screen. The video processing device includes an acquisition unit, a first storage unit, an optimization unit, a second storage unit, and a display unit. An obtaining unit is configured to obtain multi-frame image data to be rendered corresponding to a video file. The first storage unit is configured to store the multi-frame image data in an off-screen rendering buffer. The optimization unit is configured to optimize the multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm. The second storage unit is configured to send the optimized multi-frame image data to a frame buffer corresponding to the screen. The display unit is configured to read the optimized multi-frame image data from the frame buffer and display the optimized multi-frame image data on the screen.

In a third aspect, an embodiment of the present application further provides an electronic device, including: an image processor, a memory, a screen, and one or more clients. Wherein the one or more clients are stored in the memory and configured to be executed by the image processor, and the one or more programs are configured to perform the above method.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the foregoing method.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to explain the technical solutions in the embodiments of the present application more clearly, the drawings used in the description of the embodiments are briefly introduced below. Obviously, the drawings in the following description are just some embodiments of the application. For those skilled in the art, other drawings can be obtained based on these drawings without paying creative labor.

FIG. 1 shows a block diagram of a video playback architecture provided by an embodiment of the present application;

FIG. 2 shows a block diagram of an image rendering architecture provided by an embodiment of the present application;

FIG. 3 shows a method flowchart of a video processing method according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a video list interface of a client provided in an embodiment of the present application; FIG.

FIG. 5 shows a specific method flowchart of S302 to S305 in the method corresponding to FIG. 3; FIG.

FIG. 6 shows a method flowchart of a video processing method according to another embodiment of the present application;

7 shows a method flowchart of a video processing method according to another embodiment of the present application;

8 is a block diagram of a video playback architecture provided by another embodiment of the present application;

FIG. 9 shows a module block diagram of a video processing apparatus according to an embodiment of the present application;

FIG. 10 is a structural block diagram of an electronic device according to an embodiment of the present application; FIG.

FIG. 11 illustrates a storage unit for storing or carrying a program code for implementing a video processing method according to an embodiment of the present application.

detailed description

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

Please refer to FIG. 1, which shows a block diagram of a video playback architecture. Specifically, when the operating system obtains the data to be played, the next job is to parse the audio and video data. General video files are composed of two parts: video stream and audio stream. Different video formats have different audio and video packaging formats. The process of synthesizing audio and video streams into a file is called muxer, while the process of separating audio and video streams from media files is called a demoxer. Playing video files requires separating audio and video streams from the file stream. It performs decoding. The decoded video frames can be directly rendered, and the audio frames can be sent to the buffer of the audio output device for playback. Of course, the timestamps of video rendering and audio playback must be synchronized.

Specifically, video decoding may include hard decoding and soft decoding. The hardware decoding is to hand a part of the video data that was originally processed by a central processing unit (CPU) to a graphics processing unit (GPU). To do, and the GPU's parallel computing capacity is much higher than the CPU, which can greatly reduce the load on the CPU, after the CPU usage is low, you can run some other programs at the same time, of course, for the better For processors, such as the i5 2320, or any AMD quad-core processor, the difference between hardware and software is just a matter of personal preference.

Specifically, as shown in FIG. 1, Media Framework obtains a video file to be played by the client through an API interface with the client, and delivers the video file to Video Decode. Among them, the Multimedia Framework is the multimedia in the Android system. Framework, MediaPlayer, MediaPlayerService and Stagefrightplayer constitute the basic framework of Android multimedia. The multimedia framework adopts the C / S structure. MediaPlayer acts as the client of the C / S structure, and MediaPlayerService and Stagefrightplayer serve as the server of the C / S structure. They are responsible for playing multimedia files. Through Stagefrightplayer, the server completes the client request Respond. VideoDecode is a super decoder that integrates the most commonly used audio and video decoding and playback for decoding video data.

Soft decoding means that the CPU is used to decode the video by software. After decoding, the GPU is called to merge and render the video and display it on the screen. Hard decoding refers to the independent completion of video decoding tasks through a dedicated daughter card device without resorting to the CPU.

No matter hard decoding or soft decoding, after decoding the video data, the decoded video data will be sent to SurfaceFlinger, and SurfaceFlinger will render and synthesize the decoded video data and display it on the display. Among them, SurfaceFlinger is an independent service that receives all the Surface of the Window as input, calculates the position of each Surface in the final composite image according to the parameters of ZOrder, transparency, size, position, etc., and then sends it to HWComposer or OpenGL to generate the final Display buffer, and then display to a specific display device.

As shown in Figure 1, in soft decoding, the CPU decodes the video data and passes it to SurfaceFlinger for rendering and compositing, and after hard decoding is decoded by the GPU, it passes to SurfaceFlinger for rendering and compositing. The SurfaceFlinger will call the GPU to render and composite the image, and display it on the display.

Specifically, the image rendering process is shown in Figure 2. The CPU obtains the video file to be played sent by the client, decodes the decoded video data, and sends the video data to the GPU. After the GPU rendering is complete, the rendering result is placed in The frame buffer (such as FrameBuffer in Figure 2), then the video controller will read the data of the frame buffer line by line according to the HSync signal, and then pass it to the display through digital-to-analog conversion.

However, in the existing video playback, the quality of the video played is poor, and the inventors have studied the reason for the poor quality because of the lack of enhancement and optimization of the video data. Therefore, in order to solve the technical problem, an embodiment of the present application provides a video processing method. The method is applied to an image processor of an electronic device to improve the image quality effect during video playback. Specifically, refer to FIG. 3. The video processing method shown below includes: S301 to S305.

S301: Obtain multi-frame image data to be rendered corresponding to a video file.

Specifically, when the client of the electronic device plays a video, the electronic device can obtain the video file to be played, and then decode the video file. Specifically, the above-mentioned soft decoding or hard decoding can be used to decode the video file, and then decode the video file. After that, the multi-frame image data to be rendered corresponding to the video file can be obtained, and then the multi-frame image data needs to be rendered before being displayed on the display screen.

Specifically, the electronic device includes a central processing unit and an image processing unit, and a specific implementation manner of acquiring multi-frame image data to be rendered corresponding to a video file. The central processing unit acquires a video file to be played sent by a client as an implementation manner. The central processing unit obtains a video playback request sent by the client. The video playback request includes a video file to be played. Specifically, the video playback request may include identity information of the video file to be played. The identity information may be a video file. Name, based on the identity information of the video file, the video file can be found in the storage space where the video file is stored.

Specifically, the video playback request can be obtained for the touch state of the play buttons corresponding to different video files on the client's interface. Specifically, as shown in FIG. 4, the client's video list interface displays multiple video corresponding Display content, as shown in FIG. 1. The display content corresponding to multiple videos includes a thumbnail corresponding to each video. The thumbnail can be used as a touch button. When the user clicks the thumbnail, the client can detect the user's selection. Click on the thumbnail to determine the video file you want to play.

The client responds to the video selected by the user in the video list, enters the video playback interface, and clicks the play button of the playback interface. The client can detect the video file that the user currently clicks by monitoring the user ’s touch operation. Then, The client sends the video file to the CPU, and the CPU selects hard decoding or soft decoding to decode the video file.

In the embodiment of the present application, the central processing unit obtains a video file to be played, and processes the video file according to a soft decoding algorithm to obtain multi-frame image data corresponding to the video file.

Then, the image processor obtains the multi-frame image data corresponding to the video file and stores the multi-frame image data in the off-screen rendering buffer. The specific implementation manner may be: intercepting the multi-frame corresponding to the video file sent by the central processor to the frame buffer Frame image data, storing the intercepted multi-frame image data to the off-screen rendering buffer.

Specifically, a program plug-in may be provided in the image processor, and the program plug-in detects a video file to be rendered sent by the central processor to the image processor. When the central processing unit decodes the video file to obtain the image data to be rendered, the image data to be rendered is sent to the GPU, and then intercepted by the program plug-in and stored in the off-screen rendering buffer.

S302: Store the multi-frame image data in an off-screen rendering buffer.

As an implementation manner, an off-screen rendering buffer is set in the GPU in advance. Specifically, the GPU will call the rendering client module to render and synthesize the multi-frame image data to be rendered and send it to the display screen for display. Specifically, the rendering The client module can be an OpenGL module. The final position of the OpenGL rendering pipeline is in the frame buffer. Frame buffer is a series of two-dimensional pixel storage array, including color buffer, depth buffer, template buffer and accumulation buffer. By default OpenGL uses the frame buffer provided by the window system.

OpenGL's GL_ARB_framebuffer_object extension provides a way to create additional FrameBuffer Objects (FBOs). Using the frame buffer object, OpenGL can redirect the frame buffer originally drawn to the window to the FBO.

Then set another buffer outside the frame buffer through FBO, that is, the off-screen rendering buffer. Then, the acquired multi-frame image data is stored in an off-screen rendering buffer. Specifically, the off-screen rendering buffer may be a storage space corresponding to the image processor, that is, the off-screen rendering buffer itself does not have a space for storing the image, but is mapped to a storage space in the image processor, and the actual image is It is stored in a storage space in the image processor corresponding to the off-screen rendering buffer.

By binding the multi-frame image data to the off-screen rendering buffer, the multi-frame image data can be stored in the off-screen rendering buffer, that is, the multi-frame image data can be found in the off-screen rendering buffer.

S303: Optimize the multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm.

As an implementation manner, the optimization of the multi-frame image data may include adding a new special effect in the image data, for example, adding a special effect layer to the image data to achieve the effect of the special effect.

As another implementation manner, optimizing the multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm includes: optimizing image parameters of the multi-frame image data in the off-screen rendering buffer. Wherein, the image parameter optimization includes at least one of enhancement of exposure, denoising, edge sharpening, increase of contrast, or increase of saturation.

Specifically, since the decoded image data is RGBA format data, in order to optimize the image data, the RGBA format data needs to be converted to the HSV format. Specifically, the histogram of the image data is obtained, and the histogram statistics are obtained to obtain RGBA format data is converted to HSV format parameters, and according to this parameter, RGBA format data is converted to HSV format.

Among them, the exposure is enhanced. For increasing the brightness of the image, the brightness value can be increased in the area where the brightness value crosses through the histogram of the image. In addition, the brightness of the image can be increased through non-linear superposition. The darker image to be processed, T represents the lighter image after processing, then the way to increase the exposure is T (x) = I (x) + (1-I (x)) * I (x). Among them, T and I are both [0,1] values. If one effect is not good, the algorithm can iterate multiple times.

Among them, image data denoising is used to remove image noise. In particular, during image generation and transmission, images are often degraded due to the interference and influence of various noises, which affects subsequent image processing and image visual effects. Will have adverse effects. There are many types of noise, such as electrical noise, mechanical noise, channel noise, and other noise. Therefore, in order to suppress noise, improve image quality, and facilitate higher-level processing, the image must be denoised pre-processed. From the perspective of the probability distribution of noise, it can be divided into Gaussian noise, Rayleigh noise, gamma noise, exponential noise, and uniform noise.

Specifically, the image can be denoised by a Gaussian filter, where the Gaussian filter is a linear filter that can effectively suppress noise and smooth the image. Its working principle is similar to the mean filter, which takes the mean value of the pixels in the filter window as the output. The coefficients of the window template and the mean filter are different, and the template coefficients of the mean filter are all the same; while the template coefficient of the Gaussian filter decreases as the distance from the template center increases. Therefore, the Gaussian filter has less blurred image than the mean filter.

For example, a 5 × 5 Gaussian filter window is generated, and the center position of the template is used as the coordinate origin to sample. The coordinates of each position of the template are brought into the Gaussian function, and the value obtained is the coefficient of the template. Convolution of the Gaussian filter window and the image can denoise the image.

Among them, edge sharpening is used to make blurred images clearer. There are two methods of image sharpening: one is the differential method, and the other is the high-pass filtering method.

Among them, increasing the contrast is used to enhance the image quality of the image and make the colors in the image more vivid. Specifically, contrast stretching is a method of image enhancement and also belongs to a grayscale transformation operation. By gray scale transformation, the gray value is stretched to the entire range of 0-255, then its contrast is obviously greatly enhanced. You can use the following formula to map the gray value of a pixel to a larger gray space:

I (x, y) = [(I (x, y) -Imin) / (Imax-Imin)] (MAX-MIN) + MIN;

Among them, Imin and Imax are the minimum and maximum gray values of the original image, and MIN and MAX are the minimum and maximum gray values of the gray space to be stretched.

The video enhancement algorithm can increase the image quality. In addition, the corresponding video enhancement algorithm can be selected based on the video file. Specifically, the multi-frame image data in the off-screen rendering buffer is optimized according to a preset video enhancement algorithm. Before, the method further includes: obtaining a video type corresponding to the video file; and determining a video enhancement algorithm based on the video type.

Specifically, a preset number of images in the video file are acquired, and as an image sample, all objects in each image in the image sample are analyzed, and thus the proportion of each object in the image sample can be determined. Specifically, the object may include Animals, people, food, etc., can determine the category of the image based on the determined proportion of each object, thereby determining the category of the video file, where the category of the image includes a person category, an animal category, a food category, a landscape category, and the like.

The video enhancement algorithm corresponding to the video file is determined according to the correspondence between the video type and the video enhancement algorithm. Specifically, the video enhancement algorithm may include at least one of exposure enhancement, denoising, edge sharpening, increased contrast, or increased saturation. Type, the exposure enhancement, denoising, edge sharpening, contrast increase, or saturation increase of the corresponding types of videos are different. For example, as shown in Table 1:

Table 1

According to the corresponding relationship shown in Table 1, the video enhancement algorithm corresponding to the video file can be determined.

S304: Send the optimized multi-frame image data to a frame buffer corresponding to the screen.

The frame buffer corresponds to the screen and is used to store data to be displayed on the screen, such as the Framebuffer shown in FIG. 2. The Framebuffer is a driver interface that appears in the operating system kernel. Taking the Android system as an example, Linux works in protected mode, so user mode processes cannot use interrupt calls provided in the graphics card BIOS to directly write data and display on the screen like DOS systems. Linux abstracts Framebuffer This device is used by the user process to directly write data and display it on the screen. The Framebuffer mechanism mimics the functions of a graphics card, and can directly operate on the video memory by reading and writing Framebuffer. Specifically, the framebuffer can be regarded as an image of the display memory, and after it is mapped to the process address space, read and write operations can be performed directly, and the written data can be displayed on the screen.

The frame buffer can be regarded as a space for storing data. The CPU or GPU puts the data to be displayed in the frame buffer, and the Framebuffer itself does not have any ability to calculate data. The frame buffer is read by the video controller according to the screen refresh frequency. The data inside is displayed on the screen.

S305: Read the optimized multi-frame image data from the frame buffer and display it on the screen.

Specifically, after the optimized multi-frame image data is stored in the frame buffer, after the image processor detects the data written in the frame buffer, the optimized multi-frame image data is read from the frame buffer. And displayed on the screen.

As an implementation manner, the image processor reads the optimized multi-frame image data frame by frame from the frame buffer according to the refresh frequency of the screen, and displays the optimized multi-frame image data on the screen after rendering and synthesis processing.

The specific implementation of the video processing method is described below based on the FBO mechanism of the Android system. As shown in FIG. 5, specifically, the method is a further description of S302 to S305 in the method corresponding to FIG. 3, and the method includes: S501 to S516.

S501: Create a new temporary texture and bind it to the FBO.

Among them, FBO can be regarded as the above-mentioned off-screen rendering buffer.

Vertex cache, index cache, texture cache, and template cache in GPU's video memory. The texture cache is the storage space for texture data. Since FBO does not have real storage space, a new temporary texture is created and the temporary texture is stored. Binding to FBO can realize the mapping relationship between temporary texture and FBO. Because temporary texture as a variable has a certain storage space in video memory, the actual storage space of FBO is the storage space of temporary texture. Therefore, a certain amount of video memory can be allocated for the FBO.

S502: Bind the rendering object to the FBO.

The rendering object is the multi-frame image data to be rendered corresponding to the video file. Specifically, the multi-frame image data can be stored in the FBO through the rendering object, where the rendering object can be used as a variable to assign the multi-frame image data to Rendering the object, and then binding the rendering object to the FBO, can realize storing the multi-frame image data corresponding to the video file to the off-screen rendering buffer. For example, if a handle is set in the FBO, and the handle points to multiple frames of image data, the handle can be a rendering object.

S503: Clear the FBO.

Before rendering, you need to clear the old data in the FBO, including the color cache, depth cache, and template cache. It should be noted that, because the multi-frame image data to be rendered corresponding to the video file is stored in the storage space corresponding to the rendering object, the multi-frame image data is written to the FBO by mapping instead of being stored in the FBO. Therefore, clearing the FBO will not delete multi-frame image data.

S504: Bind the HQV algorithm to the Shader Program.

Shader is the code of the shader (including vertex shader, fragment shader, etc.). Shader program: The engine (program) responsible for executing the shader. Used to perform the operation specified by the previous shader code.

Among them, the HQV algorithm is the video enhancement algorithm described above. The video enhancement algorithm is bound to the Shader Program, and the program defines how to execute the video enhancement algorithm. That is, the specific algorithm execution process can be written into the Shader Program. Program so that the GPU can execute the video enhancement algorithm.

S505: Determine whether the optimization is performed for the first time.

Specifically, each optimization for the video file is recorded. For example, a number of times variable is set, and each time the optimization is performed, 1 is added to the number of times variable. Determine whether it is the first time to perform the optimization operation, that is, whether to use the video enhancement algorithm to optimize the image data of the video file for the first time. If yes, execute S506, and if not, execute S507.

S506: Binding the initial texture.

S507: Binding a temporary texture.

In addition to setting the temporary texture, an initial texture is also set. Specifically, the initial texture is used as a variable for inputting data into the temporary texture, and the content of the temporary texture is directly mapped into the FBO. The initial texture and the temporary texture are both used as data storage variables. Specifically, the feature data corresponding to the video enhancement algorithm is written into the data texture object, where the data texture object is the temporary texture.

Because when the optimization is performed for the first time, no data is stored in the temporary texture, because the temporary texture is emptied during initialization.

When it is determined that the optimization is performed for the first time, the video enhancement algorithm is assigned to the initial texture, and the feature data corresponding to the video enhancement algorithm is passed to the temporary texture by the initial texture. Specifically, the initial texture is assigned to the temporary texture. The feature data corresponding to the algorithm are the parameters of the video enhancement algorithm, for example, the values of various parameters of the median filtering in denoising.

If it is not the first time optimization, if any data is stored in the temporary texture, you do not need to obtain the feature data corresponding to the video enhancement algorithm from the initial texture. You can directly obtain the feature data corresponding to the previously stored video enhancement algorithm from the temporary texture.

S508: Convolution rendering.

The feature data corresponding to the video enhancement algorithm is convolved with the multi-frame image data to be rendered to optimize the multi-frame image data to be rendered. Specifically, the off-screen rendering is performed by rendering the rendering object and the data texture object. Multi-frame image data in the buffer is optimized. That is, the operation of rendering to texture (RTT) is performed.

S509: Does it need to iterate the next optimization.

If the next iteration is required, the number of times variable is incremented by 1 and the process returns to S505. If the next iteration is not required, the process continues to S509.

S510: Bind the rendering object to the Framebuffer.

The rendering object at this time has been optimized by the video enhancement algorithm, that is, the rendering object is optimized multi-frame image data. The optimized multi-frame image data is sent to the Framebuffer for storage.

S511: Clear the framebuffer.

S512: Bind drawing texture to Shader Program.

Among them, the drawing texture is a texture used to draw an image, and it stores effect parameters, specifically, used to increase the effect on the image data, such as shadows and the like.

S513: Texture rendering.

As above, the operation of rendering to texture is performed, except that the rendering object in this step is an optimized multi-frame image data, and the texture object is a drawing texture.

S514: Whether to draw the next frame image.

After drawing one frame of image data, if it is necessary to draw the next frame, return to execute S502, otherwise execute S515.

S515: Output the result.

S516: Recycling data.

After the rendered image data is recycled, the control screen displays the image data.

It should be noted that, for the parts that are not described in detail in the foregoing steps, reference may be made to the foregoing embodiments, and details are not described herein again.

In addition, considering that the video enhancement algorithm is used to optimize the image data, it will cause delay or even freeze in the video playback process. Therefore, for some clients playing video, you can reduce the screen refresh rate and reduce the delay. Please refer to FIG. 6, which illustrates a video processing method provided by an embodiment of the present application. The method includes: S601 to S607.

S601: Obtain a video playback request sent by a client, where the video playback request includes the video file.

S602: If the client meets a preset criterion, reduce the refresh frequency of the screen.

After the video playback request is obtained, the client requesting video playback is determined, so as to obtain the identity of the client. Specifically, the client is a client installed in an electronic device and has a video playback function. The client has an icon on the system desktop. The user can click the client's icon to open the client. For example, the client can confirm the package name of the application that the user clicks. The package name of the video application can be obtained from the code in the system background. The name format is: com.android.video.

Determine whether the client meets the preset criteria. If so, the refresh rate of the screen is reduced. If not, the operation of reducing the refresh rate of the screen is not performed.

Specifically, the preset standard may be a standard set by a user according to actual usage requirements. For example, it may be that the name of the client needs to conform to a certain category, or the installation time of the client needs to be within the preset time period, or it may be The client's developer belongs to the preset list. According to different application scenarios, different preset standards can be set.

If the client meets the preset criteria, it means that the video played by the client has a lower definition or a lower video file size, and does not require an approximate screen refresh frequency, so the screen refresh frequency can be reduced.

As an implementation manner, the refresh frequency of the screen corresponding to the client meeting the preset criteria is the preset frequency, the electronic device obtains the refresh frequency of the current screen, and if the refresh frequency of the current screen is greater than the preset frequency, the The refresh frequency of the current screen is reduced to a preset frequency. If the refresh frequency of the current screen is less than or equal to the preset frequency, the refresh frequency of the current screen is kept unchanged. Specifically, if the refresh frequency of the current screen is equal to the preset frequency, the refresh frequency of the current screen is kept unchanged, and if the refresh frequency of the current screen is less than the preset frequency, the refresh frequency of the current screen is increased to a preset frequency.

If the client does not meet the preset criteria, it will determine the size relationship between the current screen refresh frequency and the preset frequency. If the current screen refresh frequency is less than the default frequency, the current screen refresh frequency will be increased to Default frequency, where the default frequency is greater than the preset frequency.

Specifically, if the client meets the preset criteria, the specific implementation of reducing the refresh frequency of the screen is: obtaining the identity of the client; determining whether the identity of the client meets the preset identity; , Then reduce the refresh frequency of the screen.

The identity information of the client may be the name of the client or the package name, and a preset identifier is stored in the electronic device in advance. The preset identifier includes the identity identifiers of multiple preset clients, and the video played by the preset client The file size is small or the resolution is low, and the refresh frequency of the screen is not required to be too high, thereby reducing the power consumption of the electronic device by reducing the refresh frequency.

As another implementation manner, if the client meets a preset criterion, a specific implementation manner of reducing the refresh frequency of the screen is: obtaining a category of the client, and determining whether the category of the client is a preset category, and if it is , Then reduce the refresh frequency of the screen.

The preset category may be a category set by a user according to requirements, for example, it may be a self-media video client. The self-media video client has a smaller file size or lower resolution than a client for playing movies or a client for games, and it is necessary to determine whether the client is a video client.

Specifically, after the identity of the client is obtained, the type of the client is determined according to the identity, where the identity of the client may be the package name, name, etc. of the client. For example, the correspondence between the identification of the client and the type of the client is stored in the electronic device in advance, as shown in Table 2 below:

Table 2

Client ID	Client category
Apk1	game
Apk2	video

Apk3

Audio

Therefore, according to the identifier of the client shown in Table 2 above and the correspondence relationship with the type of the client, the type of the client corresponding to the video file can be determined.

As an implementation manner, the category of the client may be a category set by the developer of the client for the client when it is opened, or a category set by the user for the client after the client is installed on the electronic device. For example, when a user installs a client on an electronic device, after the installation is completed and the client is entered, a dialog box is displayed instructing the user to set a category for the client. The specific category to which the client belongs can be set by the user according to requirements. For example, the user can set a social software as an audio category, or a video category, or a social category.

In addition, client installation software is installed in the electronic device. A client list is set in the client installation software, in which the user can download the client and can update and open the client, and the client installation software can display different clients according to categories, such as audio Category, video category, or game category. Therefore, when the user uses the client installation software to install the client, the user can already know the category of the client.

Furthermore, if some clients can play video and audio, if the client supports video playback, set the client type to video type. If it does not support video playback, only audio For the playback function, the client type is set to the audio type. Specifically, whether the client supports the video playback function can be determined by the function description information contained in the client's function description information, for example, the supported playback formats to determine whether the video format playback is supported, or by detecting the client Whether a video playback module is played in the program module of the client, for example, a codec algorithm for playing a video, etc., can determine whether the client supports the video playback function.

As another implementation manner, if some clients can play videos and also play audio, for example, some video playback software can play pure audio files or videos, the category of the client can be determined according to the client ’s usage history That is, according to the usage record of the client within a certain period of time, it is determined whether the user tends to play video or audio.

Specifically, the operation behavior data of all users of the client within a preset period of time is obtained, where all users refer to all users who have installed the client, then the operation behavior data may be obtained from a server corresponding to the client, In other words, when the user uses the client, the user logs in to the client using the user account corresponding to the user, and the operation behavior data corresponding to the user account is sent to the server corresponding to the client, and the server compares the obtained operation behavior data with the User account corresponding storage. In some embodiments, the electronic device sends an operation behavior query request for the client to a server corresponding to the client, and the server sends the operation behavior data of all users within a preset time period to the electronic device.

The operation behavior data includes the name and time of the audio file being played and the name and time of the video file being played. By analyzing the operation behavior data, it is possible to determine the audio file played by the client within a certain preset time period. The number and total time can also be obtained by the number of video files played by the client and the total time, and then the type of the client is determined according to the proportion of the total playing time of the audio and video files in the predetermined time period, specifically To obtain the proportion of the total playing time of the audio and video files in the predetermined time period. For the convenience of description, the proportion of the total playing time of the audio file in the predetermined time period is recorded as the proportion of the audio playing, and the video file is recorded. The proportion of the total playback time in the predetermined time period is recorded as the proportion of video playback. If the proportion of video playback is greater than the proportion of audio playback, the client type is set to the video type. If the proportion of audio playback is greater than video playback Ratio, the client's category is set to audio type. For example, if the preset time period is 30 days, or 720 hours, and the total playback time of the audio file is 200 hours, the audio playback ratio is 27.8%, and the total playback time of the video file is 330 hours, and the video playback ratio is 45.8%, the video playback proportion is greater than the audio playback proportion, then the client type is set to the video type.

In other embodiments, the electronic device sends a category query request for the client to the server, and the server determines the foregoing audio playback proportion and video playback proportion according to the operation behavior data corresponding to the client obtained in advance, and according to the audio playback proportion The type of the client is determined by the size relationship between the ratio and the video playback ratio. Specifically, reference may be made to the foregoing description.

Therefore, by recording the playback data of the client, the clarity and type of the video played by the client most of the time can be determined, and thus it can be determined whether the client is a self-media video client. If so, then It is determined that the identity of the client meets a preset identity.

S603: Obtain multi-frame image data to be rendered corresponding to the video file.

S604: Store the multi-frame image data in an off-screen rendering buffer.

S605: Optimize the multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm.

S606: Send the optimized multi-frame image data to a frame buffer corresponding to the screen.

S607: Read the optimized multi-frame image data frame by frame in the frame buffer based on the refresh frequency of the screen, and display it on the screen after rendering and synthesis processing.

When the video is playing, the video controller in the GPU reads the optimized multi-frame image data frame by frame from the frame buffer according to the screen refresh frequency, and displays it on the screen after rendering and synthesis processing, where The refresh frequency of the screen can be regarded as a clock signal. Whenever a clock signal arrives, the optimized multi-frame image data is read frame by frame in the frame buffer, and is rendered on the screen after rendering and synthesis processing. On display.

Therefore, using off-screen rendering instead of on-screen rendering can avoid optimizing the image data in the frame buffer by on-screen rendering, which may result in the video data being screened by the video controller before it is optimized The refresh frequency is taken out from the frame buffer and displayed on the screen.

It should be noted that the above steps S601 and S602 are not limited to be performed before S603, but may be performed after S607, that is, the video can be played according to the current screen refresh frequency, and then the current screen refresh frequency Adjustment. In addition, for the parts that are not described in detail in the foregoing steps, reference may be made to the foregoing embodiments, and details are not described herein again.

Please refer to FIG. 7, which illustrates a video processing method provided by an embodiment of the present application. The method includes: S701 to S706.

S701: Obtain multi-frame image data to be rendered corresponding to a video file.

S702: Determine whether the video file meets a preset condition.

The preset condition is a condition set by the user according to actual use. For example, the preset condition may be a category of obtaining a video file. If the category of the video file is a preset category, it is determined that the video file meets the preset condition. Specifically, For a manner of determining the category of the video file, refer to the foregoing embodiment.

In addition, it is also possible to determine the real-time nature of the video file, because the method of the present application is used to optimize the video file for video enhancement. Since a new buffer is set outside the frame buffer, the video can be avoided if it is not enhanced. It is displayed on the screen. This process has certain requirements for the real-time performance of the video file playback. Therefore, it is possible to determine whether to perform a video enhancement algorithm according to the real-time performance. Specifically, determine the real-time performance level corresponding to the video file, Whether the real-time level of the video file satisfies a preset level. If so, S703 is performed; otherwise, the method ends.

Specifically, if a playback request for a video file is received, a real-time level of the video file is determined. As an implementation manner, the identifier of the client corresponding to the video file is determined, and then the real-time level of the video file is determined according to the identifier of the client. Specifically, the identifier of the client that sends the playback request of the video file is determined, and the type of the client corresponding to the identifier of the client is determined. Specifically, reference may be made to the foregoing embodiment.

Then, the real-time level corresponding to the video file is determined according to the type of the client. Specifically, the real-time level corresponding to the type of the client is stored in the electronic device, as shown in Table 3 below:

table 3

Client ID

Client category

Real-time level

Apk1	game	J1
Apk2	video	J2
Apk3	Audio	J3
Apk4	Social	J1

According to the above-mentioned correspondence relationship, the real-time level corresponding to the video file can be determined. For example, if the identifier of the client corresponding to the video file is Apk4, the corresponding category is social, and the corresponding real-time level is J1. Among them, J1 ranks highest, followed by J2 and J3.

Then, it is determined whether the real-time level of the video file belongs to a preset level.

The preset level is a preset real-time level corresponding to the required video enhancement algorithm, and may be set by a user according to requirements. For example, the preset level is J2 and below. If the real-time level corresponding to the video file is J3, the real-time level of the video file meets the preset level, that is, for video files with high real-time requirements, the video enhancement algorithm may not be executed to avoid video enhancement leading to video The delay of playback affects the user experience.

S703: Store the multi-frame image data in an off-screen rendering buffer.

For specific implementations, refer to the foregoing embodiments.

Further, an operation for determining whether the multi-frame image data needs to be stored in an off-screen rendering buffer may be added according to a user watching a video.

Specifically, the electronic device is provided with a camera, and the camera and the screen are disposed on the same side of the electronic device. Then, the person images collected by the camera are acquired, and the person images are determined to meet the preset person standards. The image data is stored in the off-screen rendering buffer. In some embodiments, the operation of determining whether the character image meets the preset character standard may replace step S702. In other embodiments, the operation of determining whether the character image meets the preset character standard may be the same as the above step S702. In combination, for example, it is determined whether the character image meets a preset character standard, if the preset character standard is met, whether the video file meets a preset condition, and if the preset condition is met, the multi-frame image data is stored To the off-screen rendering buffer. Alternatively, first determine whether the video file satisfies a preset condition, and if the preset condition is met, then determine whether the character image meets a preset character standard, and if the preset character standard is met, store the multi-frame image data To the off-screen rendering buffer.

The specific implementation manner for determining whether the character image meets a preset character standard may be:

In some embodiments, a face image in a person image may be extracted, the identity information corresponding to the face image may be determined, and then the identity information is matched with the preset identity information. If the identity information matches, it is determined that the person image meets the preset person standard. The preset identity information is pre-stored identity information, and the identity information is an identifier for distinguishing different users. Specifically, the face image is analyzed to obtain feature information, where the feature information may be facial features or facial contours, etc., and identity information is determined based on the feature information.

In other embodiments, the age stage of the user may also be determined based on the face image. Specifically, face recognition is performed on the obtained face image information to identify the facial features of the current user. The face image is preprocessed, that is, the position of the face is accurately marked in the image, and the contour, skin color, texture, texture, and color characteristics of the face are detected, and according to different pattern features such as histogram features, color features, and template features , Structural features, Haar features, etc. pick out the useful information in the above facial features and analyze the age of the current user. For example, using visual features, pixel statistical features, face image transformation coefficient features, face image algebraic features, etc., knowledge-based representation methods or algebraic features or statistical learning-based representation methods are used to model features of certain faces. , And judge the age category to which the current mobile terminal user belongs according to the characteristics.

Among them, the age stage can include children's stage, adolescent stage, youth stage, middle-aged stage and old age stage, etc., or it can start from the age of 10, and every 10 years old is divided into one age group, or it can be divided into two age groups. That is, the senile and non-senile stages. The requirements for video enhancement may be different at each age stage, for example, the display effect of the video at the age stage is not high.

After determining the user's age stage, determine whether the age stage falls within the preset stage range, and if so, perform the operation of storing the multi-frame image data in an off-screen rendering buffer and subsequent video enhancement algorithm operations. This method ends. Among them, the scope of the preset stage may be the youth stage and the middle-aged stage, that is, the child stage, the juvenile stage, and the senior stage may not require enhanced processing of the video.

S704: Optimize the multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm.

S705: Send the optimized multi-frame image data to a frame buffer corresponding to the screen.

S706: Read the optimized multi-frame image data from the frame buffer and display it on the screen.

As shown in Figure 8, an HQV algorithm module is added to the GPU. This HQV algorithm module is a module for users to execute this video processing method. Compared with Figure 2, when the image data to be rendered is sent to SurfaceFlinger after soft decoding. After being intercepted and optimized by the HQV algorithm module, it is sent to SurfaceFlinger for rendering and subsequent display operations on the screen.

In addition, for the parts that are not described in detail in the foregoing steps, reference may be made to the foregoing embodiments, and details are not described herein again.

Please refer to FIG. 9, which shows a structural block diagram of a video processing apparatus 800 according to an embodiment of the present application. The apparatus may include: an obtaining unit 901, a first storage unit 902, an optimization unit 903, a second storage unit 904 and Display unit 905.

The obtaining unit 901 is configured to obtain multi-frame image data to be rendered corresponding to a video file.

The first storage unit 902 is configured to store the multi-frame image data in an off-screen rendering buffer.

The optimization unit 903 is configured to optimize the multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm.

The second storage unit 904 is configured to send the optimized multi-frame image data to a frame buffer corresponding to the screen.

The display unit 905 is configured to read the optimized multi-frame image data from the frame buffer and display the optimized multi-frame image data on the screen.

Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices and modules described above may refer to the corresponding processes in the foregoing method embodiments, and are not repeated here.

In the several embodiments provided in this application, the coupling between the modules may be electrical, mechanical, or other forms of coupling.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist separately physically, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or software functional modules.

Please refer to FIG. 10, which is a structural block diagram of an electronic device according to an embodiment of the present application. The electronic device 100 may be an electronic device capable of running a client, such as a smart phone, a tablet computer, or an e-book. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, a screen 140, and one or more clients, where one or more clients may be stored in the memory 120 and configured For execution by one or more processors 110, one or more programs are configured to perform the method as described in the foregoing method embodiment.

The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts in the entire electronic device 100, and executes or executes instructions, programs, code sets, or instruction sets stored in the memory 120 by calling or executing data stored in the memory 120 to execute Various functions and processing data of the electronic device 100. Optionally, the processor 110 may use at least one of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). To implement a hardware form.

Specifically, the processor 110 may include one or a combination of a central processing unit 111 (Central Processing Unit, CPU), an image processor 112 (Graphics Processing Unit, GPU), and a modem. Among them, the CPU mainly handles the operating system, user interface, and client; the GPU is responsible for rendering and rendering of the displayed content; the modem is used for wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 110, and may be implemented by a communication chip alone.

The memory 120 may include Random Access Memory (RAM), and may also include Read-Only Memory. The memory 120 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 120 may include a storage program area and a storage data area, where the storage program area may store instructions for implementing an operating system and instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , Instructions for implementing the following method embodiments, and the like. The storage data area may also store data (such as phonebook, audio and video data, and chat history data) created by the terminal 100 during use.

The screen 120 is used to display information input by the user, information provided to the user, and various graphical user interfaces of the electronic device. These graphical user interfaces may be composed of graphics, text, icons, numbers, videos, and any combination thereof. In one example, a touch screen may be disposed on the display panel so as to be integrated with the display panel.

Please refer to FIG. 11, which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. The computer-readable medium 1100 stores program code, and the program code can be called by a processor to execute a method described in the foregoing method embodiment.

The computer-readable storage medium 1100 may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 1100 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 1100 has a storage space of a program code 1111 for performing any of the method steps in the above method. These program codes can be read from or written into one or more computer program products. The program code 1111 may be compressed in a suitable form, for example.

Finally, it should be noted that the above embodiments are only used to describe the technical solution of the present application, rather than limiting it. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still Modifications to the technical solutions described in the foregoing embodiments, or equivalent replacements of some of the technical features thereof; and these modifications or replacements do not drive the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (20)

1. A video processing method, characterized in that it is applied to an image processor of an electronic device, the electronic device further includes a screen, and the method includes:

   Obtaining multi-frame image data corresponding to a video file to be rendered;

   Storing the multi-frame image data in an off-screen rendering buffer;

   Optimizing multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm;

   Sending the optimized multi-frame image data to a frame buffer corresponding to the screen;

   The optimized multi-frame image data is read from the frame buffer and displayed on the screen.
2. The method according to claim 1, wherein the optimizing the multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm comprises:

   The image parameter optimization of the multi-frame image data in the off-screen rendering buffer is performed, wherein the image parameter optimization includes at least one of enhancement of exposure, denoising, edge sharpening, increase of contrast, or increase of saturation.
3. The method according to claim 2, characterized in that when the image parameter optimization includes an exposure enhancement, the optimization of the image parameters of the multi-frame image data in the off-screen rendering buffer comprises:

   Determining an area of low brightness value of each frame of image data in the off-screen rendering buffer;

   The brightness value of a region with a low brightness value of the image data is increased.
4. The method according to claim 2, wherein when the optimization of the image parameters includes denoising, the optimization of the image parameters of the multi-frame image data in the off-screen rendering buffer comprises:

   Denoise the multi-frame image data in the off-screen rendering buffer by a Gaussian filter.
5. The method according to any one of claims 1-4, wherein before optimizing the multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm, the method further comprises:

   Obtaining a video type corresponding to the video file;

   A video enhancement algorithm is determined based on the video type.
6. The method according to claim 5, wherein the obtaining a video type corresponding to the video file comprises:

   Determining the categories of all objects in each frame of the video file;

   Determining the category of the image in each frame according to the proportion of objects of each category in the image in each frame in all the objects;

   A video type corresponding to the video file is determined according to a category of the image in each frame in the video file.
7. The method according to any one of claims 1 to 6, characterized in that said storing the multi-frame image data to an off-screen rendering buffer; Multi-frame image data is optimized, including:

   Writing the multi-frame image data into a rendering object;

   Writing feature data corresponding to the video enhancement algorithm into a data texture object, wherein the rendering object and the data texture object are bound to the off-screen rendering buffer;

   By rendering the rendering object and the data texture object, the multi-frame image data in the off-screen rendering buffer is optimized.
8. The method according to any one of claims 1-7, wherein before the acquiring multi-frame image data to be rendered corresponding to a video file, further comprising:

   The central processing unit obtains a video file to be played, and processes the video file according to a soft decoding algorithm to obtain multi-frame image data corresponding to the video file.
9. The method according to any one of claims 1 to 8, wherein the optimized multi-frame image data is read frame by frame in the frame buffer, and displayed on the screen after rendering and synthesis processing. ,include:

   Based on the refresh frequency of the screen, the optimized multi-frame image data is read frame by frame in the frame buffer, and displayed on the screen after rendering and synthesis processing.
10. The method according to claim 9, further comprising:

    Obtaining a video playback request sent by a client, where the video playback request includes the video file;

    If the client meets a preset criterion, the refresh frequency of the screen is reduced.
11. The method according to claim 10, wherein if the client meets a preset criterion, reducing the refresh frequency of the screen comprises:

    Obtaining the identity of the client;

    Determining whether the identity of the client meets a preset identity;

    If so, reduce the refresh frequency of the screen.
12. The method according to claim 10, wherein if the client meets a preset criterion, reducing the refresh frequency of the screen comprises:

    Obtaining a category of the client;

    Determining whether the category of the client is a preset category;

    If so, reduce the refresh frequency of the screen.
13. The method according to claim 12, wherein the acquiring the category of the client comprises:

    If the client supports video file playback and audio file playback, obtain operation behavior data of all users of the client within a preset time period, where the operation behavior data includes the name and Time, and the name and time of the video file being played;

    Determining the total playing time of the audio file and the total playing time of the video file according to the operation behavior data;

    The category of the client is determined according to the total playing time of the audio file and the proportion of the total playing time of the video file in the predetermined time period.
14. The method according to claim 13, wherein the category of the client includes a video type and an audio type, and the total duration of the playback of the audio file and the total duration of the playback of the video file are within the predetermined period of time To determine the category of the client, including:

    Recording the proportion of the total playing time of the audio file within the predetermined time period as the proportion of audio playback, and recording the proportion of the total playing time of the video file within the predetermined time period as the proportion of video playback ratio;

    If the video playback ratio is greater than the audio playback ratio, setting the category of the client to a video type;

    If the audio playback ratio is greater than the video playback ratio, the category of the client is set to the audio type.
15. The method according to any one of claims 1-14, wherein the storing the multi-frame image data into an off-screen rendering buffer comprises:

    Judging whether the video file meets a preset condition;

    If so, the multi-frame image data is stored in an off-screen rendering buffer.
16. The method according to claim 15, wherein the determining whether the video file satisfies a preset condition comprises:

    Determining a real-time level corresponding to the video file;

    Determining whether the real-time level of the video file meets a preset level;

    If the preset level is satisfied, determining that the video file meets a preset condition;

    If the preset level is not satisfied, it is determined that the video file does not satisfy a preset condition.
17. The method according to claim 15, wherein the video file is a person image collected by a camera of the electronic device, and the determining whether the video file meets a preset condition comprises:

    Determining whether the character image meets a preset character standard;

    If the preset character criterion is satisfied, determining that the video file meets a preset condition;

    If the preset character criterion is not satisfied, it is determined that the video file does not satisfy a preset condition.
18. A video processing device is characterized in that it is applied to an image processor of an electronic device, and the electronic device further includes a screen, including:

    An obtaining unit, configured to obtain multi-frame image data to be rendered corresponding to a video file;

    A first storage unit, configured to store the multi-frame image data in an off-screen rendering buffer;

    An optimization unit, configured to optimize the multi-frame image data in the off-screen rendering buffer according to a preset video enhancement algorithm;

    A second storage unit, configured to send the optimized multi-frame image data to a frame buffer corresponding to the screen;

    The display unit is configured to read the optimized multi-frame image data from the frame buffer and display the optimized multi-frame image data on the screen.
19. An electronic device, comprising:

    Image processor

    Memory

    screen;

    One or more clients, wherein the one or more clients are stored in the memory and configured to be executed by the image processor, and the one or more programs are configured to execute as claimed in claim 1 -17. The method of any one of -17.
20. A computer-readable medium, characterized in that the computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method according to any one of claims 1-17.

Images