WO2021253141A1

WO2021253141A1 - Image data processing apparatus and method

Info

Publication number: WO2021253141A1
Application number: PCT/CN2020/096018
Authority: WO
Inventors: 张运强
Original assignee: 华为技术有限公司
Priority date: 2020-06-15
Filing date: 2020-06-15
Publication date: 2021-12-23
Also published as: CN116075804A

Abstract

An image data processing method and apparatus. According to the method and apparatus, the compositing and matting processing of a graphics layer and the processing of a video layer are performed in parallel in two threads. Therefore, the processing of the video layer is no longer affected by the composition of the graphics layer, such that the problem of frame loss being caused by the time-consuming composition of the graphics layer during the process of video playing can be effectively solved. The method comprises: in a first thread, compositing a multi-layer graphics layer and performing matting processing on at least one graphics layer of the multi-layer graphics layer, so as to obtain a composited graphics layer, wherein the composited graphics layer comprises a matting area; processing a video layer in a second thread so as to obtain a processed video layer; and superimposing the composited graphics layer and the processed video layer so as to obtain display data.

Description

Device and method for image data processing

Technical field

This application relates to the field of display technology, and in particular to a device and method for image data processing.

Background technique

The content that can be displayed by the display device includes graphic data and video data. The graphic data includes, for example, status bar, navigation bar, and icon data. Generally speaking, status bar, navigation bar, and icon data each correspond to a graphic layer, and video data corresponds to a video. Layer, when multiple graphics data and video data are displayed at the same time, the picture that the user sees on the display is the result of the synthesis of multiple graphics and video layers.

However, in complex scenes, the composition of the graphics layer is relatively time-consuming, and usually exceeds a vertical synchronization (Vsync) period, resulting in frame loss during video playback.

Summary of the invention

The embodiments of the present application provide an image data processing device and method, which are used to solve the problem of frame loss caused by time-consuming composition of graphics layers during video playback.

The first aspect of the present application provides a method for processing image data. The method includes: synthesizing multiple graphics layers in a first thread and digging holes for at least one graphics layer of the multiple graphics layers to obtain The synthesized graphics layer, the synthesized graphics layer includes the excavated area; the video layer is processed in the second thread to obtain the processed video layer, and the processed video layer can be displayed through the excavated area; The synthesized graphics layer and the processed video layer are superimposed to obtain display data.

It should be understood that the synthesized graphics layer data includes a digging area, and the digging area is usually set to be transparent so that when the graphics layer and the video layer are synthesized, the video layer can be displayed through the digging area. In an optional case, you can dig holes in the graphics layer below the video layer, and then combine multiple graphics layers into one graphics layer; you can also combine multiple graphics layers into one graphics layer, and then The synthesized graphics layer is digging holes.

In the embodiment of this application, the synthesis and digging of the graphics layer and the processing of the video layer are performed in parallel in two threads. Therefore, the processing of video images will no longer be affected by the synthesis of the graphics layer, and video playback will also It is no longer affected by the composition of the graphics layer, and can effectively solve the problem of frame loss caused by the time-consuming composition of the graphics layer during video playback.

In a possible implementation manner, before synthesizing multiple graphics layers and digging holes on at least one graphics layer in the first thread to obtain the synthesized graphics layer, the method further includes: in the first thread Setting information related to the size of the video layer in the video layer; the digging of at least one graphics layer specifically includes: digging the at least one graphics layer according to the information related to the size of the video layer; Processing the video layer in the second thread to obtain the processed video layer specifically includes: in the second thread, processing the video layer according to the size-related information of the video layer to obtain the processed video layer.

It should be understood that the size-related information of the video layer may include the size of the video layer and the position information of the video layer. Exemplarily, the size-related information of the video layer may include one vertex coordinate and two length information of the video layer. The length information indicates the length and width of the video layer; the information related to the size of the video layer can also include the coordinates of two vertices and one length information of the video layer, and the size and the length of the video layer can be uniquely determined according to the coordinates of the two vertices and one length information. Play position; if the position information of the video layer is the coordinates of the 4 vertices of the display video layer, the size of the video layer can be determined according to the coordinates of the 4 vertices. At this time, the information related to the size of the video layer may only include position information. The information related to the size of the video layer is calculated by SurfaceFlinger. Illustratively, the multimedia framework sets the initial size of the video layer through the system's own API and sends it to SurfaceFlinger. SurfaceFlinger can also capture or perceive the user's zoom in or zoom out of the video. Or rotation and other operations, the initial size of the video layer sent by the SurfaceFlinger integrated multimedia framework and operations such as zooming in, zooming out, or rotating, etc., calculate the information related to the size of the video layer.

In the embodiment of this application, since setting the size of the video layer and digging the graphics layer below the video layer are completed in the same thread successively, that is, the sizes of the processed video layer and the digging area are all based on The size of the video layer set by this setting can ensure that the size of the digging area in the graphics layer is exactly the same as the size of the video layer. The processed video layer and the "dug area" can be completely matched, so that the processed video layer It can be displayed simultaneously through the "dug area".

In a possible implementation manner, the multi-layer graphics layer is synthesized in the first thread based on the first vertical synchronization signal, and the at least one graphics layer is holed; the synthesized graphics layer is synthesized based on the second vertical synchronization signal It is superimposed with the processed video layer to obtain display data; wherein, the first vertical synchronization signal and the second vertical synchronization signal are independent of each other.

It should be understood that the first vertical synchronization signal and the second vertical synchronization signal are two independent periodic signals, and the first vertical synchronization signal and the second vertical synchronization signal may have different frame rates and different periods. Specifically, when the effective signal of the first vertical synchronization signal arrives, the multi-layer graphics layer is synthesized in the first thread and the hole processing is performed on at least one of the graphics layers; when the effective signal of the second vertical synchronization signal arrives , Superimpose the synthesized graphics layer and the processed video layer to obtain the display data. The first vertical synchronization signal and the second vertical synchronization signal may be high-level active or low-level active, and the first vertical synchronization signal and the second vertical synchronization signal may be level-triggered, rising-edge-triggered, or falling-edge-triggered. The arrival of the effective signal of the Vsync signal can be understood as: the arrival of the rising edge of the Vsync signal, the arrival of the falling edge of the Vsync signal, when the Vsync signal is a high-level signal, or when the Vsync signal is a low-level signal. Exemplarily, the first vertical synchronization signal may be Graphic Vsync in the foregoing embodiment, and the second vertical synchronization signal may be Display Vsync in the foregoing embodiment.

In a possible implementation manner, the frame rate of the first vertical synchronization signal is lower than the frame rate of the second vertical synchronization signal.

In the embodiment of the present application, since the first vertical synchronization signal is used to control the synthesis of multiple graphics layers, the signal that controls the synthesis of display video data and the refresh frame rate of the display device is the second vertical synchronization signal, and the first vertical synchronization signal and the first vertical synchronization signal The two vertical synchronization signals are independent of each other. The frame rate of the second vertical synchronization signal can be greater than that of the first vertical synchronization signal. Therefore, the image processing architecture can support high frame rate video with a video frame rate higher than the graphics refresh frame rate. Play.

In a possible implementation, when the effective signal of the first vertical synchronization signal arrives, in the first thread, the size-related information of the video layer is set first, and then the multi-layer graphics layer is synthesized and based on the video layer. The size-related information is used to dig holes for at least one layer of graphics.

In the embodiment of this application, setting the size-related information of the video layer in the first thread needs to wait for the effective signal of the vertical synchronization signal to arrive. It is carried out after the effective signal of the vertical synchronization signal arrives.

In a possible implementation manner, when the second thread obtains the first video buffer Video Buffer, the first notification information is sent to the first thread; in the first thread, according to the first Video The size of the Buffer sets the information related to the size of the video layer; or, when the second thread detects that the size of the Video Buffer changes, it sends the first notification information to the first thread; in the first thread, according to The size of the changed Video Buffer resets the information related to the size of the video layer.

It should be understood that each Video Buffer stores data of one video layer, and the size of the Video Buffer is related to the size of the stored video layer. Therefore, when the size of the video layer changes, the size of the Video Buffer also changes. The first notification information is used to notify the first thread that the size of the video layer has changed.

In a possible implementation manner, inter-thread communication can be performed between the first thread and the second thread. When the size of the first Video Buffer or Video Buffer is received, the second thread can notify the first thread. In order to reset the information related to the size of the video layer in the first thread, and re-dig holes in the graphics layer according to the changed size, so as to ensure that the video layer after the size change can be displayed through the digging area of the graphics layer , That is, to ensure that the video layer and the graphics layer can still be synchronized and matched display when the size of the Video Buffer changes. In an optional situation, when the size of the Video Buffer changes, the information related to the size of the video data calculated by SurfaceFlinger may not necessarily change.

In a possible implementation manner, synthesizing multiple graphics layers and digging holes on at least one graphics layer in the first thread to obtain a synthesized graphics layer specifically includes: in the first thread, The hardware synthesizer HWC synthesizes the multiple graphics layers and performs hole processing on the at least one graphics layer to obtain the synthesized graphics layer.

It should be understood that when calling the hardware resources of the HWC to perform graphics layer synthesis and digging processing, the SurfaceFlinger of the framework layer specifically calls the HWC driver by accessing the HWC abstraction of the hardware abstraction layer, so as to realize the call of the HWC hardware resources.

In a possible implementation manner, in the first thread, the indication information of multiple graphic buffers Graphic Buffers is sent to the hardware synthesizer HWC; among them, one Graphic Buffer stores a layer of graphics layer data; in the first thread, In one thread, the HWC obtains the multiple graphics layer data from the Graphic Buffers.

In a possible implementation manner, in the first thread, the multi-layer graphics layer is synthesized and the at least one graphics layer is digged to obtain the synthesized graphics layer, which specifically includes: in the first thread, the graphics The processor GPU synthesizes the multiple graphics layers and performs hole processing on the at least one graphics layer to obtain the synthesized graphics layer.

When HWC does not support graphics layer synthesis and graphics layer digging processing, GPU hardware resources can be used to perform graphics layer synthesis and graphics layer digging.

In a possible implementation manner, after setting the size-related information of the video layer in the first thread, the method further includes: in the first thread, sending the size-related information of the video layer to Media HW; In the second thread, Media HW processes the video layer according to the size-related information of the video layer to obtain the processed video layer.

In a possible implementation, Media HW first receives information about the size of the video layer in the first thread, and then Media HW receives the first frame of video layer data in the second thread, and then Media HW receives The information related to the size of the video layer processes the first frame of video layer data; in another case, Media HW first receives the first frame of video layer data in the second thread, and then Media HW receives it in the first thread. When it comes to the size-related information of the video layer, Media HW will not process the first frame of video layer data immediately, but will wait for the size-related information of the video layer to receive the first frame of video layer data. Perform processing to avoid processing errors in the first frame of video layer data.

In a possible implementation manner, the superimposing the synthesized graphics layer and the processed video layer to obtain display data specifically includes: the display driver superimposes the synthesized graphics layer and the processed video layer to obtain The display data.

In a possible implementation manner, the method further includes: SurfaceFlinger creates the first thread and the second thread in the initialization phase; when the SurfaceFlinger receives the Graphic Buffer, notifying the first thread to process the Graphic Buffer When the SurfaceFlinger receives the Video Buffer, it notifies the second thread to process the video layer data in the Video Buffer; among them, the Graphic Buffer stores a layer of graphics layer data, and the Video Buffer stores a layer Video layer data.

The second aspect of the present application provides a method for image data processing. The method includes: in a first thread, SurfaceFlinger calls graphics hardware to synthesize multiple graphics layers and dig holes for at least one graphics layer to obtain the synthesis In the second thread, SurfaceFlinger calls the media hardware to process the video layer to obtain the processed video layer; the synthesized graphics layer includes the excavated area; the processed video layer can be displayed through the excavated area Out; the display driver superimposes the synthesized graphics layer and the processed video layer to obtain display data.

In a possible implementation manner, the SurfaceFlinger invokes graphics hardware resources to synthesize multiple graphics layers and perform digging processing on at least one graphics layer to obtain the synthesized graphics layer, the method further includes: In the thread, the SurfaceFlinger sends the calculated information about the size of the video layer to the Media HW; the SurfaceFlinger calls graphics hardware resources to dig holes for at least one graphics layer, which specifically includes: in the first thread, the SurfaceFlinger calls the graphics hardware to dig holes in the at least one graphics layer according to the size-related information of the video layer to obtain the synthesized graphics layer; in the second thread, SurfaceFlinger calls the media hardware to process the video layer to obtain The processed video layer specifically includes: in the second thread, SurfaceFlinger calls the Media HW to process the video layer according to the size-related information of the video layer to obtain the processed video layer.

In a possible implementation, when the effective signal of the first vertical synchronization signal arrives, the SurfaceFlinger calls the hardware resource to synthesize the multi-layer graphics layer and perform the at least one graphics layer in the first thread. Digging processing; when the effective signal of the second vertical synchronization signal arrives, the drive display superimposes the synthesized graphics layer and the processed video layer to obtain the display data; wherein, the first vertical synchronization signal and the The second vertical synchronization signals are independent of each other.

In a possible implementation manner, when the effective signal of the first vertical synchronization signal arrives, in the first thread, the information related to the calculated size of the video layer is first sent to the Media HW, and then the SurfaceFlinger The hardware resource is called to synthesize the multi-layer graphics layer, and the at least one graphics layer is drilled according to the size-related information of the video layer.

In a possible implementation manner, the frame rate of the first vertical synchronization signal is less than the frame rate of the second vertical synchronization signal.

In a possible implementation manner, it further includes: when the second thread obtains the first video buffer Video Buffer, sending first notification information to the first thread; in the first thread, SurfaceFlinger obtains the The size of the first Video Buffer, and set the information related to the size of the video layer according to the size of the first Video Buffer; or, when the second thread detects that the size of the Video Buffer has changed, send it to the first The thread sends the first notification information; in the first thread, SurfaceFlinger obtains the size of the changed Video Buffer, and resets the size-related information of the video layer according to the size of the changed Video Buffer; where, the Video Buffer A piece of data of the video layer is stored in the video layer, and the size of the Video Buffer is related to the size of the video layer.

In a possible implementation manner, in the first thread, SurfaceFlinger calls graphics hardware to synthesize multiple graphics layers and dig holes for at least one graphics layer to obtain a synthesized graphics layer, which specifically includes: In the first thread, the SurfaceFlinger calls the hardware synthesizer HWC to synthesize the multiple graphics layers and dig holes for the at least one graphics layer to obtain the synthesized graphics layer; the display driver combines the synthesized graphics layer and Before superimposing the processed video layer to obtain display data, the method further includes: the HWC sends the synthesized graphics layer to the display driver; and the Media HW sends the processed video layer to the display driver.

It should be understood that when the HWC gets the synthesized graphics layer, the HWC abstraction sends the indication information of the FrameBuffer storing the synthesized graphics layer to the display driver; after the Media HW gets the processed video layer, the Media HW will store the processed video The instruction information of the Video Buffer of the layer is sent to the display driver.

In a possible implementation, in the first thread, SurfaceFlinger calls graphics hardware to synthesize multiple graphics layers and dig holes for at least one graphics layer to obtain a synthesized graphics layer, which specifically includes: In one thread, the SurfaceFlinger calls the graphics processor GPU; composites the multiple graphics layers and digs holes for the at least one graphics layer to obtain the composite graphics layer; the display driver performs the composite graphics layer Before superimposing the processed video layer to obtain the display data, the method further includes: the GPU returns the synthesized graphics layer to the SurfaceFlinger; the SurfaceFlinger sends the synthesized graphics layer to the HWC; and the HWC sends the synthesized graphics layer to the HWC. The synthesized graphics layer is sent to the display driver; the Media HW sends the processed video layer to the display driver.

It should be understood that if the graphics layer synthesis and graphics layer digging processing are performed by the GPU, the GPU needs to return the synthesized graphics layer to SurfaceFlinger first, and then SurfaceFlinge sends the instructions of the FrameBuffer storing the synthesized graphics layer to the HWC abstraction , And then the HWC abstracts and sends the instruction information of the FrameBuffer storing the synthesized graphics layer to the display driver.

In a possible implementation manner, it further includes: SurfaceFlinger creates the first thread and the second thread in the initialization phase; when the buffer received by the SurfaceFlinger is a graphic buffer Graphic Buffer, notifying the first thread to process the Graphic Buffer When the buffer received by the SurfaceFlinger is a Video Buffer, notify the second thread to process the video layer data in the Video Buffer; among them, one Graphic Buffer stores one layer of graphics layer data, and one Video Buffer stores One layer of video layer data.

In a possible implementation, the Media HW receives the first frame of video layer data in the second thread; the Media HW receives information about the size of the video layer in the first thread; the Media HW In the second thread, the first frame of video layer data is processed according to the information related to the size of the video layer.

It should be understood that if the Media HW receives the first frame of video layer data in the second thread before receiving the information about the size of the video layer, the Media HW receives the video layer data in the first thread. After the size-related information, the Media HW processes the first frame of video layer data in the second thread according to the size-related information of the video layer.

The third aspect of the present application provides an image data processing device. The device includes a processor on which software instructions run to form a framework layer, a hardware abstraction layer HAL, and a driver layer. The HAL includes graphics hardware abstraction and Media hardware Media HW abstraction, the driver layer includes graphics hardware drivers, media hardware drivers, and display drivers; this framework layer is used in the first thread to call graphics hardware through the graphics hardware abstraction and the graphics hardware driver, and it is used for multiple layers The graphics layer is synthesized and at least one graphics layer of the multi-layer graphics layer is digged to obtain a synthesized graphics layer. The synthesized graphics layer includes a digging area; the frame layer is used in the second thread , Through the media hardware abstraction and the media hardware driver calling the media hardware Media HW to process the video layer to obtain the processed video layer; the processed video layer can be displayed through the digging area; the display driver, It is used to superimpose the synthesized graphics layer and the processed video layer to obtain display data.

Optionally, the device further includes a transmission interface through which the processor receives data sent by other devices or sends data to other devices. The device can be coupled with hardware resources such as a display, media hardware, or graphics hardware through a connector, a transmission line, or a bus. The device can be a processor chip with image or video processing functions. In an optional case, the device, media hardware, and graphics hardware can be integrated on one chip. In another optional situation, the device, media hardware, graphics hardware, and display may be integrated on one terminal. Among them, the graphics hardware abstraction corresponds to the graphics hardware driver, the media hardware abstraction corresponds to the media hardware driver, the graphics hardware can be called through the graphics hardware abstraction and the graphics hardware driver, and the media hardware can be called through the media hardware abstraction and the media hardware driver. For example, the graphics hardware driver is abstractly called by accessing the graphics hardware to realize the calling of the graphics hardware, and the media hardware driver is abstractly called by accessing the media hardware to realize the calling of the media hardware.

It should be understood that the synthesized graphics layer is stored in the FrameBuffer. After the graphics hardware obtains the synthesized graphics layer, the graphics hardware abstracts the FrameBuffer instruction information to the display driver, and the display driver can read from the corresponding memory space according to the FrameBuffer instruction information Take the synthesized graphics layer data; after the media hardware obtains the processed video layer, the media hardware abstracts and sends the instruction information of the Video Buffer storing the processed video layer to the display driver, and the display driver can correspond to the instruction letter of the Video Buffer Read the processed video layer data in the memory space.

In a possible implementation, the framework layer is also used to: in the first thread, set the size-related information of the video layer, and send the size-related information of the video layer to the media hardware abstraction; the framework layer is specifically used for In the first thread, the graphics hardware is called to synthesize the multi-layer graphics layer according to the size-related information of the video layer, and at least one graphics layer is digged to obtain the synthesized graphics layer; in the second thread, call Media HW processes the video layer according to the size-related information of the video layer to obtain the processed video layer.

In a possible implementation manner, the framework layer is specifically used to call the graphics hardware based on the first vertical synchronization signal to synthesize the multi-layer graphics layer in the first thread and to dig holes in at least one graphics layer Processing: The display driver is specifically used to superimpose the synthesized graphics layer and the processed video layer based on the second vertical synchronization signal to obtain display data; wherein the first vertical synchronization signal and the second vertical synchronization signal are independent of each other.

In a possible implementation manner, the framework layer is specifically used to, when the effective signal of the first vertical synchronization signal arrives, in the first thread, the size-related information of the video layer is first sent to the media hardware abstraction, and then Calling graphics hardware to synthesize multiple graphics layers and digging holes for at least one graphics layer according to the size-related information of the video layer.

In a possible implementation manner, when the second thread obtains the first video buffer Video Buffer, it sends the first notification information to the first thread; the framework layer is also used to receive the first notification in the first thread After the information, in the first thread, the size of the first Video Buffer is obtained, and the size-related information of the video layer is set according to the size of the first Video Buffer.

Or, when the second thread detects that the size of the Video Buffer has changed, it sends the first notification information to the first thread; the framework layer is also used to, after the first thread receives the first notification information, in the first thread, Obtain the size of the changed Video Buffer, and set the information related to the size of the video layer according to the size of the changed Video Buffer; among them, the Video Buffer stores the data of a video layer, and the size of the Video Buffer is related to the size of the video layer.

In a possible implementation manner, the graphics hardware includes HWC and GPU. Correspondingly, the hardware abstraction layer includes the HWC abstraction, and the driver layer includes the HWC driver and the GPU driver.

The framework layer is specifically used for: in the first thread, through the HWC abstraction and the HWC driver to call the HWC, the multi-layer graphics layer is synthesized and at least one graphics layer is digged to obtain the synthesized graphics layer; the HWC abstraction is used Yu: Send the synthesized graphics layer to the display driver; the media hardware abstraction is also used to send the processed video layer to the display driver.

When HWC does not support graphics layer synthesis and graphics layer digging processing, the framework layer is specifically used to: in the first thread, call the GPU through the GPU driver to synthesize multiple graphics layers and dig holes for at least one graphics layer Processing to obtain the synthesized graphics layer; GPU is also used to: return the synthesized graphics layer to the framework layer; the framework layer is also used to send the synthesized graphics layer to the HWC abstraction; the HWC abstraction is used to send the synthesized graphics layer For the display driver; the Media HW abstraction is also used to send the processed video layer to the display driver.

It should be understood that the HWC abstraction sends the instruction information of the FrameBuffer storing the synthesized graphics layer to the display driver, and the media hardware abstraction sends the instruction information of the VideoBuffer storing the processed video layer to the display driver.

In a possible implementation, the SurfaceFlinger of the framework layer is used to create the first thread and the second thread in the initialization phase; SurfaceFlinger is also used to notify the first thread to process the Graphic Buffer when the Graphic Buffer is received. When receiving the Video Buffer, notify the second thread to process the video layer data in the Video Buffer; among them, a layer of graphics layer data is stored in the Graphic Buffer, and a layer of video layer data is stored in the Video Buffer.

In a possible implementation, the Media HW abstraction is specifically used to: receive the first frame of video layer data in the second thread; receive the size-related information of the video layer in the first thread; the frame layer is specifically used to : In the second thread, Media HW abstractly calls Media HW to process the first frame of video layer data according to the size-related information of the video layer.

It should be understood that the beneficial effects of the device side can be referred to the method side, which will not be repeated here.

The fourth aspect of the present application provides an image data processing device. The device includes: a framework layer for invoking the graphics hardware to synthesize the multi-layer graphics layer in the first thread and at least one of the multi-layer graphics layers A graphics layer is used for digging a hole to obtain a composite graphics layer, the composite graphics layer includes the digging area; the graphics hardware; media hardware Media HW; the framework layer is also used to call the Media in the second thread HW processes the video layer to obtain the processed video layer; the processed video layer can be displayed through the digging area; the display driver is used to superimpose the synthesized graphics layer and the processed video layer , Get the display data.

It should be understood that the framework layer and the display driver are part of the operating system formed by software instructions running on the processor. Graphics hardware and media hardware may be coupled with the processor through connectors, interfaces, transmission lines or buses, etc. These interfaces are usually electrical communication interfaces, but may also be mechanical interfaces or other forms of interfaces.

In a possible implementation manner, the software instructions running on the processor are also used to form a hardware abstraction layer, graphics hardware drivers, and media hardware drivers. The hardware abstraction layer includes graphics hardware drivers corresponding to graphics hardware and media Media hardware driver corresponding to the hardware. The framework layer is specifically used to call graphics hardware through graphics hardware abstraction and graphics hardware drivers, and the framework layer is specifically used to call media hardware through media hardware abstraction and media hardware drivers.

In a possible implementation, the framework layer is also used to: in the first thread, set the size-related information of the video layer, and send the size-related information of the video layer to Media HW; the framework layer is specifically used Therefore, in the first thread, the graphics hardware is called to synthesize the multi-layer graphics layer according to the information related to the size of the video layer and the at least one graphics layer is digged to obtain the synthesized graphics layer; In the second thread, the Media HW is called to process the video layer according to the size-related information of the video layer to obtain the processed video layer.

In a possible implementation, the framework layer is specifically used to, based on the first vertical synchronization signal, call the graphics hardware in the first thread to synthesize the multi-layer graphics layer and perform the at least one graphics layer. Digging processing; the display driver is specifically used to, when the second vertical synchronization signal arrives, superimpose the synthesized graphics layer and the processed video layer to obtain the display data; wherein, the first vertical synchronization signal and The second vertical synchronization signals are independent of each other.

In a possible implementation manner, the framework layer is specifically used to send information related to the size of the video layer to the Media in the first thread when the effective signal of the first vertical synchronization signal arrives. HW, and then call the graphics hardware to synthesize the multi-layer graphics layer, and perform digging processing on the at least one graphics layer according to the size-related information of the video layer.

In a possible implementation manner, when the second thread obtains the first video buffer Video Buffer, the first notification information is sent to the first thread; the framework layer is also used to receive in the first thread After the first notification information, in the first thread, obtain the size of the first Video Buffer, and set the size-related information of the video layer according to the size of the first Video Buffer; or, when the first When the second thread detects that the size of the Video Buffer has changed, it sends first notification information to the first thread; the framework layer is also used to: after the first thread receives the first notification information, in the first thread , Obtain the size of the changed Video Buffer, and set the size-related information of the video layer according to the size of the changed Video Buffer; wherein, the Video Buffer stores a piece of data of the video layer, and the size of the Video Buffer It is related to the size of the video layer.

In a possible implementation manner, the graphics hardware includes a hardware synthesizer HWC, and the framework layer is specifically used to: in the first thread, call the HWC to synthesize the multi-layer graphics layer and to compose at least one graphics layer Perform the digging process to obtain the synthesized graphics layer; the display driver superimposes the synthesized graphics layer and the processed video layer to obtain the display data, the HWC is also used to: send the synthesized graphics layer to the display Driver; The Media HW is also used to send the processed video layer to the display driver.

It should be understood that sending the synthesized graphics layer to the display driver is specifically implemented by the HWC abstraction in the hardware abstraction layer of the HWC, and sending the processed video layer to the display driver is implemented by the Media HW abstraction.

In a possible implementation manner, the graphics hardware includes a graphics processor GPU, and the framework layer is specifically used to: in the first thread, call the GPU to synthesize the multi-layer graphics layer and dig at least one graphics layer. Hole processing to obtain the synthesized graphics layer; the display driver superimposes the synthesized graphics layer and the processed video layer, and before the display data is obtained, the GPU is also used to: return the synthesized graphics layer to the framework Layer; the framework layer is also used to send the synthesized graphics layer to the HWC; the HWC is also used to send the synthesized graphics layer to the display driver; the Media HW is also used to send the processed graphics layer The video layer is sent to the display driver.

In a possible implementation manner, the framework layer includes SurfaceFlinger, which is used to: create the first thread and the second thread in the initialization phase; and the SurfaceFlinger is also used to, when receiving the Graphic Buffer, Notify the first thread to process the graphics layer data in the Graphic Buffer; when the Video Buffer is received, notify the second thread to process the video layer data in the Video Buffer; wherein the Graphic Buffer stores a layer of graphics layer data , The Video Buffer stores a layer of video layer data.

In a possible implementation manner, the Media HW is specifically used to: receive the first frame of video layer data in the second thread; receive the size-related information of the video layer in the first thread; the frame The layer is specifically used to: in the second thread, call the Media HW to process the first frame of video layer data according to the size-related information of the video layer.

The fifth aspect of the present application provides an image data processing device. The device includes a processor, graphics hardware, and media hardware. Software instructions run on the processor to form a framework layer and a display driver; the framework layer is used to In the first thread, the graphics hardware is called to synthesize multiple graphics layers and at least one graphics layer is digging holes to obtain a composite graphics layer. The composite graphics layer includes the digging area; the frame layer is used in the second In the thread, the media hardware is called to process the video layer to obtain the processed video layer; the display driver is used to superimpose the synthesized graphics layer and the processed video layer to obtain display data.

It should be understood that graphics hardware and media hardware may be coupled with the processor through connectors, interfaces, transmission lines or buses, etc. These interfaces are usually electrical communication interfaces, but may also be mechanical interfaces or other forms of interfaces.

As mentioned earlier, the framework layer calls Media HW through the Media HW abstraction and Media HW driver to realize the processing of the video layer.

The sixth aspect of the present application provides a computer-readable storage medium that stores instructions in the computer-readable storage medium, and when the instructions are run on a computer or a processor, the computer or the processor is caused to execute the first aspect as described above. Or the method in any of its possible implementations.

The seventh aspect of the present application provides a computer program product containing instructions. When the instructions run on a computer or a processor, the computer or the processor executes the above-mentioned first aspect or any one of its possible implementations. The method.

Description of the drawings

FIG. 1 is a schematic diagram of an exemplary terminal architecture provided by an embodiment of the application;

FIG. 2 is a hardware architecture diagram of an exemplary image processing device provided by an embodiment of the application;

FIG. 3a is a schematic diagram of an exemplary operating system architecture applicable to an embodiment of this application;

FIG. 3b is a schematic diagram of an exemplary graphic layer synthesis process provided by an embodiment of the application;

Figure 4 is a schematic diagram of a traditional image processing architecture;

FIG. 5 is a schematic diagram of an exemplary image processing architecture provided by an embodiment of the application;

FIG. 6 is a schematic diagram of another exemplary image processing architecture provided by an embodiment of the application;

FIG. 7 is a flowchart of an image processing method provided by an embodiment of the application;

FIG. 8 is a flowchart of another image processing method provided by an embodiment of the application;

FIG. 9 is a flowchart of another image processing method provided by an embodiment of the application;

FIG. 10 is a schematic structural diagram of an exemplary image processing apparatus provided by an embodiment of the application;

FIG. 11 is a schematic structural diagram of another exemplary image processing apparatus provided by an embodiment of the application;

FIG. 12 is a schematic structural diagram of another exemplary image processing apparatus provided by an embodiment of the application;

FIG. 13 is a schematic structural diagram of another exemplary image processing apparatus provided by an embodiment of the application;

FIG. 14 is a schematic structural diagram of another exemplary image processing apparatus provided by an embodiment of the application.

detailed description

The terms "first", "second", etc. in the specification embodiments and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusion, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those clearly listed steps or units, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

It should be understood that in this application, "at least one (item)" refers to one or more, and "multiple" refers to two or more. "And/or" is used to describe the association relationship of associated objects, indicating that there can be three types of relationships. For example, "A and/or B" can mean: only A, only B, and both A and B. , Where A and B can be singular or plural. The character "/" generally indicates that the associated objects before and after are in an "or" relationship. "The following at least one item (a)" or similar expressions refers to any combination of these items, including any combination of a single item (a) or a plurality of items (a). For example, at least one (a) of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c" ", where a, b, and c can be single or multiple.

First, in order to facilitate the understanding of the embodiments of the present application, the following introduces terms related to the embodiments of the present application.

Graphic Buffer (Graphic Buffer): used to save graphics data or graphics layer data, the graphics layer data can include, for example, bullet screen data, subtitle data, navigation bar, status bar, icon layer, floating window, application display interface or Information such as identification information, graphics layer data can also be some data information rendered after the application is started. The graphics data in the graphics buffer can come from multiple applications.

Frame Buffer (FrameBuffer): used to save the composite graphics layer data, the composite graphics layer data is synthesized from the multi-layer graphics layer data.

Video Buffer (Video Buffer): Used to store video data. The video data can come from Tencent Video, iQiyi Video, Youku Video, etc., for example. For example, it is used to store decoded data of the multimedia frame.

Surface: Represents the graphics data of a window of the application process, and a Surface corresponds to a graphics layer data.

Vsync signal: used to synchronize the time when the application starts to render, the time to wake up the SurfaceFlinger composite graphics layer, and the display refresh cycle (display refresh cycle) of the display device. The Vsync signal is periodic, the number of Vsync valid signals in a unit time is called the Vsync frame rate, the time interval between two adjacent Vsync valid signals is called the Vsync period, and the Vsync frame rate is the reciprocal of the Vsync period. Exemplarily, the common Vsync cycle is 16ms, and the Vsync frame rate is 1s/16ms=60. It should be understood that the Vsync signal can be active high or active low, and the Vsync signal can be level triggered, rising edge triggered or falling edge triggered. The arrival of the effective Vsync signal can be understood as: the arrival of the rising edge of the Vsync signal, the arrival of the falling edge of the Vsync signal, when the Vsync signal is a high-level signal, or when the Vsync signal is a low-level signal.

As shown in FIG. 1, it is a schematic structural diagram of an exemplary terminal 100 provided by an embodiment of this application. The terminal 100 may include an antenna system 110, a radio frequency (RF) circuit 120, a processor 130, a memory 140, a camera 150, an audio circuit 160, a display screen 170, one or more sensors 180, a wireless transceiver 190, and so on.

The antenna system 110 may be one or more antennas, and may also be an antenna array composed of multiple antennas. The radio frequency circuit 120 may include one or more analog radio frequency transceivers, the radio frequency circuit 120 may also include one or more digital radio frequency transceivers, and the RF circuit 120 is coupled to the antenna system 110. It should be understood that in the various embodiments of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, connection through various interfaces, transmission lines, buses, and the like. The radio frequency circuit 120 can be used for various types of cellular wireless communications.

The processor 130 may include a communication processor, and the communication processor may be used to control the RF circuit 120 to receive and send signals through the antenna system 110, and the signal may be a voice signal, a media signal, or a control signal. The processor 130 may include various general processing devices, such as a general central processing unit (Central Processing Unit, CPU), a system on chip (System on Chip, SOC), a processor integrated on the SOC, and a separate processor chip. Or a controller, etc.; the processor 130 may also include a dedicated processing device, such as an application specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA), or a digital signal processor (Digital Signal Processor). Processor, DSP), dedicated video or graphics processor, graphics processing unit (Graphics Processing Unit, GPU), neural network processing unit (Neural-network Processing Unit, NPU), etc. The processor 130 may be a processor group composed of multiple processors, and the multiple processors are coupled to each other through one or more buses. The processor may include an analog-to-digital converter (Analog-to-Digital Converter, ADC) and a digital-to-analog converter (Digital-to-Analog Converter, DAC) to realize signal connection between different components of the device. The processor 130 is used to process media signals such as images, audios, and videos.

The memory 140 is coupled to the processor 130. Specifically, the memory 140 may be coupled to the processor 130 through one or more memory controllers. The memory 140 can be used to store computer program instructions, including a computer operating system (Operation System, OS) and various user application programs. The memory 140 can also be used to store user data, such as graphics image data, video data, and video data rendered by application programs. Audio data, calendar information, contact information, or other media files, etc. The processor 130 may read computer program instructions or user data from the memory 140, or store computer program instructions or user data in the memory 140, so as to implement related processing functions. The memory 140 may be a non-power-down volatile memory, such as EMMC (Embedded Multi Media Card, embedded multimedia card), UFS (Universal Flash Storage), or read-only memory (Read-Only Memory, ROM), Or other types of static storage devices that can store static information and instructions, or volatile memory (volatile memory), such as Random Access Memory (RAM), or other types that can store information and instructions The type of dynamic storage device can also be Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM (Compact Disc Read-Only Memory, CD-ROM) or other optical disk storage, optical discs Memory (including compact discs, laser discs, digital universal discs or Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, but not limited to this. Optionally, the memory 140 may be independent of the processor 130, and the memory 140 may also be integrated with the processor 130.

The camera 150 is used to collect images or videos. Illustratively, the user can trigger the turning on of the camera 150 through an application program instruction to realize a photographing or camera function, such as taking pictures or videos of any scene. The camera can include components such as lenses, filters, and image sensors. Among them, the camera may be located in front of the terminal device or on the back of the terminal device. The specific number and arrangement of the cameras can be flexibly determined according to the requirements of the designer or manufacturer's strategy, which is not limited in this application.

The audio circuit 160 is coupled with the processor 130. The audio circuit 160 may include a microphone 161 and a speaker 162. The microphone 161 may receive sound input from the outside, and the speaker 162 may play audio data. It should be understood that the terminal 100 may have one or more microphones and one or more earphones, and the embodiment of the present application does not limit the number of microphones and earphones.

The display screen 170 is used to provide users with various display interfaces or various menu information for selection. Illustratively, the content displayed on the display screen 170 includes, but is not limited to, a soft keyboard, a virtual mouse, virtual keys and icons, etc. These display contents are associated with specific internal modules or functions. The display screen 170 may also accept user input. Optionally, the display screen 170 may also display information input by the user, such as accepting control information such as enabling or disabling. Specifically, the display screen 170 may include a display panel 171 and a touch panel 172. Among them, the display panel 171 may adopt a liquid crystal display (Liquid Crystal Display, LCD), an organic light emitting diode (Organic Light-Emitting Diode, OLED), a light emitting diode (Light Emitting Diode, LED) display device, or a cathode ray tube (Cathode Ray tube). Tube, CRT) etc. to configure the display panel. The touch panel 172, also known as a touch screen, a touch-sensitive screen, etc., can collect user contact or non-contact operations on or near it (for example, the user uses any suitable objects or accessories such as fingers, stylus, etc.) on the touch panel 172 Or the operation near the touch panel 172 may also include a somatosensory operation; the operation includes a single-point control operation, a multi-point control operation, etc.), and the corresponding connection device is driven according to a preset program. Optionally, the touch panel 172 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the signal brought by the user's touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into information that can be processed by the processor 130, and sends it to The processor 130 can receive and execute commands sent by the processor 130. Further, the touch panel 172 can cover the display panel 171, and the user can operate on or near the touch panel 172 according to the content displayed on the display panel 171. After the touch panel 172 detects the operation, it passes through the I/O subsystem 10 It is transmitted to the processor 130 to determine the user input, and then the processor 130 provides corresponding visual output on the display panel 171 through the I/O subsystem 10 according to the user input. Although in FIG. 1, the touch panel 172 and the display panel 171 are used as two independent components to realize the input and output functions of the terminal 100, but in some embodiments, the touch panel 172 and the display panel 171 are integrated in together.

The sensor 180 may include an image sensor, a motion sensor, a proximity sensor, an environmental noise sensor, a sound sensor, an accelerometer, a temperature sensor, a gyroscope, or other types of sensors, and various combinations of them. The processor 130 drives the sensor 180 to receive various information such as audio information, image information, or motion information through the sensor controller 12 in the I/O subsystem 10, and the sensor 180 transmits the received information to the processor 130 for processing.

The wireless transceiver 190 can provide wireless connection capabilities to other devices. The other devices can be peripheral devices such as wireless headsets, Bluetooth headsets, wireless mice, or wireless keyboards, or wireless networks, such as wireless fidelity (Wireless Fidelity). Fidelity, WiFi) network, wireless personal area network (Wireless Personal Area Network, WPAN), or other wireless local area network (Wireless Local Area Network, WLAN), etc. The wireless transceiver 190 may be a Bluetooth compatible transceiver, which is used to wirelessly couple the processor 130 to peripheral devices such as Bluetooth headsets and wireless mice. The wireless transceiver 190 may also be a WiFi compatible transceiver for processing The device 130 is wirelessly coupled to a wireless network or other devices.

The terminal 100 may also include other input devices 14 coupled to the processor 130 to receive various user inputs, such as receiving inputted numbers, names, addresses, and media selections. The other input devices 14 may include keyboards, physical buttons (press buttons, Rocker buttons, etc.), dials, slide switches, joysticks, click scroll wheels, and optical mice (optical mice are touch-sensitive surfaces that do not display visual output, or are extensions of touch-sensitive surfaces formed by touch screens).

The terminal 100 may also include the aforementioned I/O subsystem 10, and the I/O subsystem 10 may include other input device controllers 11 for receiving signals from other input devices 14 or sending the processor 130 to other input devices 190. For controlling or driving information, the I/O subsystem 10 may also include the aforementioned sensor controller 12 and display controller 13, which are respectively used to implement the exchange of data and control information between the sensor 180 and the display screen 170 and the processor 130.

The terminal 100 may further include a power source 101 to supply power to other components of the terminal 100 including 110-190, and the power source may be a rechargeable or non-rechargeable lithium ion battery or a nickel hydrogen battery. Further, when the power supply 101 is a rechargeable battery, it can be coupled with the processor 130 through a power management system, so that the management of charging, discharging, and power consumption adjustment can be realized through the power management system.

It should be understood that the terminal 100 in FIG. 1 is only an example, and does not limit the specific form of the terminal 100. The terminal 100 may also include other existing components that are not shown in FIG. 1 or that may be added in the future.

In an optional solution, the RF circuit 120, the processor 130, and the memory 140 may be partially or completely integrated on one chip, or may be independent chips. The RF circuit 120, the processor 130, and the memory 140 may include one or more integrated circuits arranged on a printed circuit board (PCB).

As shown in FIG. 2, a hardware architecture diagram of an exemplary image processing apparatus provided by an embodiment of the present application. The image processing apparatus 200 may be, for example, a processor chip. Exemplarily, the hardware architecture shown in FIG. 2 The figure may be an exemplary architecture diagram of the processor 130 in FIG. 1, and the image processing method and image processing architecture provided by the embodiment of the present application may be applied to the processor chip.

Referring to FIG. 2, the device 200 includes: at least one CPU, a memory, a microcontroller (Microcontroller Unit, MCU), a GPU, an NPU, a memory bus, a receiving interface, a sending interface, and so on. Although not shown in FIG. 2, the device 200 may also include an application processor (Application Processor, AP), a decoder, and a dedicated video or image processor.

The above-mentioned parts of the device 200 are coupled through connectors. Illustratively, the connectors include various interfaces, transmission lines, or buses. These interfaces are usually electrical communication interfaces, but may also be mechanical interfaces or other forms of interfaces. The embodiment does not limit this.

Optionally, the CPU can be a single-CPU processor or a multi-CPU processor; optionally, the CPU can be a processor group composed of multiple processors, between multiple processors Coupled to each other through one or more buses. The receiving interface may be a data input interface of the processor chip. In an optional case, the receiving interface and the transmitting interface may be High Definition Multimedia Interface (HDMI), V-By-One Interface, Embedded Display Port (eDP), Mobile Industry Processor Interface (MIPI) or Display Port (DP), etc. For the memory, reference may be made to the foregoing description of the memory 140.

In an optional case, the above-mentioned parts are integrated on the same chip; in another optional case, the CPU, GPU, decoder, receiving interface, and transmitting interface are integrated on one chip, and the chip is Each part of the access to external memory through the bus. The dedicated video/graphics processor can be integrated with the CPU on the same chip, or it can exist as a separate processor chip. For example, the dedicated video/graphics processor can be a dedicated ISP. In an optional situation, the NPU can also be used as an independent processor chip. The NPU is used to implement various neural network or deep learning related operations.

The chip involved in the embodiments of this application is a system manufactured on the same semiconductor substrate by an integrated circuit process, also called a semiconductor chip, which can be manufactured on a substrate using an integrated circuit process (usually a semiconductor such as silicon) Material) is a collection of integrated circuits formed on the surface, the outer layer of which is usually encapsulated by a semiconductor packaging material. The integrated circuit may include various types of functional devices. Each type of functional device includes transistors such as logic gate circuits, Metal-Oxide-Semiconductor (MOS) transistors, bipolar transistors or diodes, and may also include capacitors and resistors. Or inductance and other components. Each functional device can work independently or under the action of necessary driver software, and can realize various functions such as communication, calculation, or storage.

As shown in FIG. 3a, it is a schematic diagram of an exemplary operating system architecture to which this embodiment of the application is applicable. The operating system may run on the processor 130 shown in FIG. 1, and the code corresponding to the operating system may be stored in the memory 140 shown in FIG. 1, or the operating system may run on the image processing system shown in FIG.装置200上。 Device 200.

Exemplarily, the operating system may be, for example, an Android system, an iOS system, or a Linux system. The operating system architecture includes an application (APP) layer, a framework layer, a hardware abstraction layer (HAL), and a driver layer.

Optionally, the APP layer may include, for example, applications such as WeChat, iQiyi, Tencent Video, Taobao, or Camera.

The frame layer is the logical scheduling layer of the operating system architecture, and the frame layer can perform resource scheduling and policy allocation for the video processing process. Exemplarily, the framework layer includes:

The Graphics Framework is responsible for the layout of the graphics window and the rendering of graphics data, storing the rendered graphics data in the Graphics Buffer, and sending the graphics layer data in the Graphic Buffer to SurfaceFlinger.

The Multimedia Framework is responsible for decoding the video stream and sending the decoded data to SurfaceFlinger.

SurfaceFlinger is responsible for managing each layer (including graphics layer and video layer), receiving the Graphic Buffer and Video Buffer of each layer, and superimposing the graphics layer by using a graphics processing unit (GPU) or a hardware compositor. The graphics layer data superimposed by SurfaceFlinger is stored in the Framebuffer, and the data in the Framebuffer can be read and displayed on the display screen. Figure 3b is a schematic diagram of an exemplary graphics layer synthesis process. The example shown in Figure 3b includes three graphics layer data, namely the status bar graphics layer, the icon graphics layer, and the navigation bar graphics layer. Examples Sexually, the three graphics layer data can be rendered by the same application, where each graphics layer data corresponds to a Graphic Buffer, and SurfaceFlinger synthesizes the three graphics layer data into one image frame data, and combines the image frame The data is stored in the FrameBuffer. The image frame is the composite result of the status bar graphics layer, the icon graphics layer and the navigation bar graphics layer. The image frame data in the FrameBuffer can be read and displayed on the display screen. In an optional situation, the content to be displayed includes video data in addition to the graphic data, and the image frame data and the video data are synthesized before being displayed on the display screen.

The Open Graphics Library (OpenGL) provides an interface for graphics rendering and graphics layer overlay, and can be connected to the GPU driver.

HAL is the interface layer between operating system software and audio and video hardware devices. HAL provides an interface for the interaction between upper layer software and lower layer hardware. The HAL layer abstracts the underlying hardware into software that contains the corresponding hardware interfaces. By accessing the HAL layer, the settings of the underlying hardware devices can be realized. For example, the relevant hardware devices can be enabled or disabled at the HAL layer. The driver layer is used to directly control the underlying hardware devices according to the control information input by the HAL. Exemplarily, the HAL includes a hardware composer (Hardware Composer, HWC) abstraction and a media hardware (Media Hardware, Media HW) abstraction. Correspondingly, the driver layer includes a hardware synthesis driver and a media hardware driver. HWC is used for hardware synthesis of multiple graphics layers, which can provide support for SurfaceFlinger's hardware synthesis, and store the synthesized graphics layer in FrameBuffer and send it to the display driver. Media HW is responsible for processing the video layer data, and informing the display driver of the processed video layer and the position information of the video layer. Media HW is a dedicated hardware circuit that can be used to improve the video display effect. It should be understood that different vendors may refer to media hardware differently. It should be understood that the HWC abstraction of the HAL layer corresponds to the hardware synthesis driver of the driver layer, and the media hardware abstraction corresponds to the media hardware driver. By accessing the HWC abstraction of the HAL layer, the control of the underlying HWC hardware can be realized through the hardware synthesis driver. The control of the underlying Media HW is achieved by accessing the media hardware abstraction and media hardware driver of the HAL layer.

The driver layer also includes a GPU driver and a display driver. The GPU driver is responsible for the rendering and overlay of graphics, and the display driver is responsible for the synthesis of the video layer and the graphics layer, and sends the synthesis result to the display for display.

As shown in Figure 4, it is a schematic diagram of a traditional image processing architecture. In this image processing architecture, the graphics framework sends the instructions of multiple graphic buffers Graphic Buffers to SurfaceFlinger, and the multimedia framework sends the instructions of the video buffer Video Buffer to SurfaceFlinger, where one graphics layer corresponds to one Graphic Buffer, for example The graphics layer of the navigation bar and the graphics layer of the status bar correspond to different Graphic Buffers. SurfaceFlinger binds Graphic Buffers and Video Buffers to the corresponding layers. For example, bind Graphic Buffer 1 to the navigation bar graphics layer, and bind Graphic Buffer 2 to the state. Column graphics layer, bind Video Buffer to the video layer, etc. When the Vsync signal arrives, SurfaceFlinger sends the instructions of Graphic Buffers and Video Buffer to the hardware synthesizer HWC in the main thread. HWC synthesizes multiple graphics layers in all Graphic Buffers, and digs holes in the graphics layer below the video layer during the synthesis process to display the video layer. The graphics layer data synthesized by HWC is stored in FrameBuffer. Then, in the main thread, the HWC sends the instruction information of the Video Buffer to the Media HW so that the Media HW can read the video data from the Video Buffer and process the video data. It should be understood that although not shown, the hardware synthesizer and media hardware in Figure 4 both include the corresponding hardware abstraction layer and driver layer. When the hardware synthesizer is called, the hardware synthesizer of the hardware abstraction layer needs to be accessed to abstractly call the hardware synthesis driver. In order to implement the call to the hardware synthesizer; similarly, the media hardware abstraction layer and the media hardware driver need to be used to implement the call to the Media HW. Call. Exemplarily, SurfaceFlinger sends the indication information of Graphic Buffers and Video Buffer to the HWC abstraction layer of the abstraction layer in the main thread, and uses the hardware synthesis driver to call the HWC hardware to realize the synthesis of multiple graphics layers and the performance of the graphics layer. Digging treatment. HWC sends the synthesized graphics layer data to the display driver, and Media HW sends the processed video image to the display driver. When the new Vsync signal arrives, the display driver synthesizes the composite graphics layer data sent by the HWC and the video data sent by the Media HW, and sends the composite result to the display device for display. It should be understood that the indication information of the buffer is used to point to a memory area. Exemplarily, the indication information may be a file descriptor (fd).

In this image processing architecture, since the display frame rate of the video frame and the synthesis of multiple graphics layers are controlled by the same vertical synchronization signal, the supported video frame rate is limited by the speed of the graphics layer synthesis, so better The hardware to support the high frame rate refresh of graphics frames. In addition, because HWC's processing of multiple graphics layers and Media HW's processing of video data are performed sequentially in the same thread, video playback is easily affected by the synthesis of multiple graphics layers. Especially in some complex graphics scenes, graphics rendering and compositing are time-consuming, resulting in some graphics frames and video frames not being displayed in time, these graphics frames and video frames will be discarded by the application or SurfaceFlinger, even if the hardware is upgraded Performance may still cause frame loss during video playback.

The embodiment of the present application provides an image processing architecture, and the image processing architecture is shown in FIG. 5.

The graphics framework sends the instructions of Graphic Buffer to SurfaceFlinger, and the multimedia framework sends the instructions of Video Buffer to SurfaceFlinger.

When SurfaceFlinger is initialized, it will start two threads at the same time: the first thread and the second thread. In the first thread, the instruction information of Graphic Buffers is sent to HWC so that HWC can synthesize multiple graphics layers and control the graphics layer. In the second parallel thread, the instruction information of Video Buffer is sent to Media HW so that Media HW can process the video data. In this way, since Media HW's processing of video data and HWC's composite processing of multiple graphics layers are completed in parallel in two threads, Media HW's processing of video data will no longer be affected by the progress of graphics layer synthesis. . It should be understood that HWC can respectively dig holes in the graphics layer below the video layer, and then synthesize multiple graphics layers into one graphics layer; HWC can also synthesize multiple graphics layers into one graphics layer, and then the synthesized graphics Layers for digging. The synthesized graphics layer data obtained by the HWC processing is stored in the FrameBuffer, and the HWC sends the instruction information of the FrameBuffer to the display driver. Media HW stores the processed video layer data in Video Buffer, and sends the instruction information of Video Buffer to the display driver.

It should be understood that the first thread and the second thread are both circular threads. When SurfacFlinger receives the Graphic Buffer sent by the graphics framework, it will notify the first thread to process, but the first thread will not start to process until the graphics Vsync arrives. The graphics layer data in Graphic Buffer is processed. When SurfacFlinger receives the Video Buffer sent by the multimedia framework, it will notify the second thread to process, and the second thread will start processing after receiving the notification without waiting for the vertical synchronization signal.

Specifically, when the graphics Vsync arrives, in the first thread, the set size-related information of the video layer is sent to Media HW, so that Media HW can process the video data according to the size-related information of the video layer; similarly; In the first thread, the HWC digs a hole in the graphics layer according to the information related to the size of the video layer set, so that the size of the digging area is equal to the size of the video layer. Because setting the size of the video layer and digging the hole are performed in the same thread, the size of the hole is exactly the same as the size of the set video layer, which ensures that the video layer and the graphics layer are synchronized and displayed. Exemplarily, the information related to the size of the video layer may include position information and size information of the video layer, etc. Optionally, when the position information includes the positions of the four vertices of the video layer, the size of the video layer can be determined according to the position information , The size-related information of the video layer can only include the position information; when the position information only includes a certain vertex position of the video layer (for example, the vertex position in the upper right corner), the size-related information of the video layer also includes the size of the video information. The information related to the size of the video layer is calculated by SurfaceFlinger. For example, the multimedia framework sends the initial size of the video layer set by the system's own application programming interface (Application Programming Interface, API) to SurfaceFlinger, and SurfaceFlinger can also Capturing or perceiving the user's operations such as zooming in, zooming out, or rotating the video, the initial size of the video layer and operations such as zooming in, zooming out, or rotating sent by the SurfaceFlinger integrated multimedia framework, and calculate the information related to the size of the video layer.

However, because the size of the Video Buffer will affect the digging of the graphics layer, and the processing of the Video Buffer is performed in the second thread, when the second thread receives the first Video Buffer or detects that the size of the Video Buffer has changed At that time, a notification message is sent to the first thread of SurfaceFlinger to notify that the size of the video layer has changed. Exemplarily, the notification information may be carried on an identification bit. It should be understood that the size of the Video Buffer may include the size information of the Video Buffer and the rotation information of the Video Buffer. Each Video Buffer stores a frame of video data or data of a video layer. The size of the Video Buffer and the video data (or The size of the video layer is related. For example, the size of the Video Buffer may be equal to the size of the video data, and the rotation information of the Video Buffer is used to indicate the rotation information of the video data. After SurfaceFlinger receives the notification message, it obtains the size of the updated Video Buffer, and recalculates the information related to the size of the video data according to the size of the updated Video Buffer, so that the HWC in the first thread can be based on the changed size Re-dug the graphics layer to ensure that the video layer after the size change can be displayed through the graphics layer, so as to ensure that the video layer and the graphics layer can still be synchronized and displayed when the size of the Video Buffer changes. In an optional situation, when the size of the Video Buffer changes, the information related to the size of the video data calculated by SurfaceFlinger may not necessarily change.

It should be understood that the order of sending the size-related information of the video layer to Media HW in the first thread and sending the first frame of video data to Media HW in the second thread is not limited. In the case, Media HW first receives the information related to the size of the video layer, and then receives the first frame of video data; in another optional case, Media HW first receives the first frame of video data, and then receives the video Information about the size of the layer. In either case, Media HW needs to process the first frame of video data according to the received information about the size of the video layer before sending it to the display driver.

In an optional solution, SurfaceFlinger needs to refer to the size information of the Video Buffer when calculating the size of the video data. In this case, when Media HW receives the first Video Buffer, it will send it to SurfaceFlinger. The first thread sends notification information. After SurfaceFlinger receives the notification message, it obtains the size of the first Video Buffer, and calculates the size-related information of the video data according to the size of the first Video Buffer.

In addition, the image processing architecture introduces two vertical synchronization signals Graphic Vsync and Display Vsync. Graphic Vsync is used to trigger the synthesis of multiple graphics layers, and Display Vsync is used to trigger the display driver to synthesize the graphics layer and video layer and display device Refresh. It should be understood that Graphic Vsync and Display Vsync are two independent vertical synchronization signals, and the frame rates of Graphic Vsync and Display Vsync may be different. For example, the frame rate of Display Vsync can be set to be greater than the frame rate of Graphic Vsync, so that the refresh frame rate of the video can be greater than the actual refresh frame rate of the graphics. Exemplarily, Graphic Vsync is also used to trigger the application to render the graphics layer data. The graphics layer data in the Graphic Buffer can come from multiple applications. The application renders the graphics layer data to fill the Graphic Buffer and fill the Graphic Buffer and HWC pair. The synthesis of multiple graphics layer data corresponds to two different cycles of the Graphic Vsync signal.

In the image processing architecture provided by the embodiments of this application, the composition of the graphics layer by HWC and the processing of the video layer by Media HW are performed in parallel by threads. Therefore, the processing of video images will no longer be affected by the composition of the graphics layer. Video playback It is no longer affected by the composition of the graphics layer, and can effectively solve the problem of frame loss caused by the time-consuming composition of the graphics layer during the video playback process. Furthermore, setting the size of the video layer and digging holes in the graphics layer are completed in the same thread one after the other, and inter-thread communication can be carried out between the first thread and the second thread. When the size of the Video Buffer changes, The second thread may notify the first thread so that the digging size of the graphics layer is consistent with the size of the video data, so as to ensure the synchronization and matching display of the video layer and the graphics layer. In addition, because the vertical synchronization signal that controls the synthesis of multiple graphics layers and the vertical synchronization signal that controls the refresh frame rate of the display device are independent of each other, the refresh frame rate of the video can be greater than the actual refresh frame rate of the graphics, so the image processing architecture can support The playback of high frame rate video with a video frame rate higher than the graphics refresh frame rate.

In an optional situation, the graphics hardware included in the graphics processing system includes an HWC and a GPU. If the HWC does not support the overlay of the graphics layer, the hardware resources of the GPU can be called to implement the overlay of the multiple graphics layers.

As shown in FIG. 6, another exemplary image processing architecture is provided for this embodiment of the application.

In the image processing architecture shown in Figure 6, in the first thread, when the graphics Vsync arrives, the indication information of Graphic Buffers is sent to the GPU, and the GPU realizes the synthesis (or overlay) of the multi-layer graphics layer data and the matching In the process of digging holes in the graphics layer, the synthesized graphics layer data is stored in FrameBuffer. In other words, SurfaceFlinger uses the GPU's image processing function to realize the synthesis of multi-layer graphics layer data and the processing of digging holes in the graphics layer by calling the API interface of the GPU. The GPU returns the processing result to SurfaceFlinger, and SurfaceFlinger sends the FrameBuffer instruction information to the hardware synthesizer, and then the hardware synthesizer informs the display driver of the FrameBuffer instruction information, so that the display driver can read and process from the corresponding memory according to the FrameBuffer instruction information After the graphics layer data. Compared with the image processing architecture shown in Figure 5, the image processing architecture in Figure 6 is not done by HWC, but by GPU. Other processing is not done by HWC, but by GPU. The image processing architecture shown in FIG. 5 is the same, and reference may be made to the description of the image processing architecture shown in FIG. 5, which will not be repeated here.

It should be understood that although not shown, the hardware synthesizer and media hardware in Figures 5 and 6 all include the corresponding hardware abstraction layer and driver layer, and the hardware synthesizer needs to be called through the hardware synthesizer abstraction layer and hardware synthesis driver. , In other words, need to access the hardware synthesizer of the HAL to abstractly call the hardware synthesis driver to implement the call to the hardware synthesizer; similarly, the media hardware abstraction layer and the media hardware driver need to be used to call the Media HW, or in other words, through the access The media hardware of HAL abstractly calls the media hardware driver to realize the call to the media hardware. It should be understood that in Figures 5 and 6, the instruction information of Graphic Buffer is sent to the hardware synthesizer abstraction, and the instruction information of Video Buffer is sent to the media hardware abstraction; the hardware synthesizer abstraction sends the instruction information of FrameBuffer to the display driver, and the media hardware abstraction Send the instruction information of Video Buffer to the display driver. In other words, it can be considered that the transmission of related instruction information occurs at the hardware abstraction layer and the driver layer, and will not be sent to the hardware.

Based on the image processing architecture shown in FIG. 5 and FIG. 6, an embodiment of the present application also provides an image data processing method. As shown in FIG. 7, the method includes:

701. In the first thread, synthesize multiple graphics layers and perform digging processing on at least one graphics layer of the multiple graphics layers to obtain a composite graphics layer, where the composite graphics layer includes a digging area;

702. Process the video layer in the second thread to obtain a processed video layer.

It should be understood that the processed video layer can be displayed through the excavated area. In this method embodiment, a certain size difference between the processed video layer and the excavated area is allowed, that is, the sizes of the processed video layer and the excavated area may not be exactly the same.

703. Superimpose the synthesized graphics layer and the processed video layer to obtain display data.

In an optional situation, before step 701, the method further includes: setting information related to the size of the video layer in the first thread.

In this way, the size-related information of the video layer is set in the first thread, and then at least one layer of graphics layer is digged according to the size-related information of the video layer; in the first thread, the set video The information related to the size of the layer is sent to Media HW, and then the Media HW processes the video layer according to the information related to the size of the video layer in the second thread to obtain the processed video layer. It should be understood that the size-related information of the video layer may include the size of the video layer and the position information of the video layer. Exemplarily, the size-related information of the video layer may include one vertex coordinate and two length information of the video layer. The length information indicates the length and width of the video layer; the information related to the size of the video layer can also include the coordinates of two vertices and one length information of the video layer, and the size and the length of the video layer can be uniquely determined according to the coordinates of the two vertices and one length information. Play position; if the position information of the video layer is the coordinates of the 4 vertices of the display video layer, the size of the video layer can be determined according to the coordinates of the 4 vertices. At this time, the information related to the size of the video layer may only include position information. The information related to the size of the video layer is calculated by SurfaceFlinger. Illustratively, the multimedia framework sets the initial size of the video layer through the system's own API and sends it to SurfaceFlinger. SurfaceFlinger can also capture or perceive the user's zoom in or zoom out of the video. Or rotation and other operations, the initial size of the video layer sent by the SurfaceFlinger integrated multimedia framework and operations such as zooming in, zooming out, or rotating, etc., calculate the information related to the size of the video layer.

In an optional case, the multi-layer graphics layer is synthesized in the first thread based on the first vertical synchronization signal, and the at least one graphics layer is digged; the synthesized graphics layer is synthesized based on the second vertical synchronization signal It is superimposed with the processed video layer to obtain display data; wherein, the first vertical synchronization signal and the second vertical synchronization signal are independent of each other.

In an optional situation, the frame rate of the first vertical synchronization signal is less than the frame rate of the second vertical synchronization signal.

In an optional situation, when the effective signal of the first vertical synchronization signal arrives, in the first thread, the size-related information of the video layer is set first, and then the multi-layer graphics layer is synthesized and based on the video layer. The size-related information is used to dig holes for at least one layer of graphics.

In an optional situation, when the second thread obtains the first video buffer Video Buffer, the first notification information is sent to the first thread; in the first thread, according to the first Video The size of the Buffer sets the information related to the size of the video layer; or, when the second thread detects that the size of the Video Buffer changes, it sends the first notification information to the first thread; in the first thread, according to The size of the changed Video Buffer resets the information related to the size of the video layer.

In this embodiment of the application, inter-thread communication can be performed between the first thread and the second thread. When the size of the first Video Buffer or Video Buffer is received, the second thread can notify the first thread so that In the first thread, reset the information related to the size of the video layer, and re-dug the graphics layer according to the changed size, so as to ensure that the video layer after the size change can be displayed through the digging area of the graphics layer. That is to ensure that when the size of the Video Buffer changes, it can still realize the synchronization and matching display of the video layer and the graphics layer. In an optional situation, when the size of the Video Buffer changes, the information related to the size of the video data calculated by SurfaceFlinger may not necessarily change.

In an optional case, the HWC synthesizes multiple graphics layers and performs hole-digging processing on at least one graphics layer in the first thread.

In an optional situation, in the first thread, the GPU synthesizes multiple graphics layers and performs hole-digging processing on at least one graphics layer to obtain a synthesized graphics layer.

In an optional situation, after setting the size-related information of the video layer in the first thread, the method further includes: in the first thread, sending the size-related information of the video layer to Media HW; In the second thread, Media HW processes the video layer according to the size-related information of the video layer to obtain the processed video layer.

In an optional situation, Media HW first receives the information about the size of the video layer in the first thread, and then Media HW receives the first frame of video layer data in the second thread, and then Media HW receives The information related to the size of the video layer processes the first frame of video layer data; in another case, Media HW first receives the first frame of video layer data in the second thread, and then Media HW receives it in the first thread. When it comes to the size-related information of the video layer, Media HW will not process the first frame of video layer data immediately, but will wait for the size-related information of the video layer to receive the first frame of video layer data. Perform processing to avoid processing errors in the first frame of video layer data.

In an optional situation, the display driver superimposes the synthesized graphics layer and the processed video layer to obtain display data.

In an optional case, the method further includes:

SurfaceFlinger creates the first thread and the second thread in the initialization phase; when the SurfaceFlinger receives the Graphic Buffer, the first thread is notified to process the graphics layer data in the Graphic Buffer; when the SurfaceFlinger receives the Video Buffer , Notify the second thread to process the video layer data in the Video Buffer; wherein, the Graphic Buffer stores a layer of graphics layer data, and the Video Buffer stores a layer of video layer data.

Based on the image processing architecture shown in FIG. 5 and FIG. 6, an embodiment of the present application also provides another image data processing method. As shown in FIG. 8, the method includes:

801. Set information related to the size of the video layer in the first thread and send it to Media HW;

It should be understood that the size-related information of the video layer may include the size of the video layer and the position information of the video layer. The size-related information of the video layer is calculated by SurfaceFlinger. Illustratively, the multimedia framework sets the video through the system's own API. The initial size of the layer is sent to SurfaceFlinger. SurfaceFlinger can also capture or perceive the user's operations such as zooming in, zooming out or rotating the video. The initial size of the video layer sent by the SurfaceFlinger integrated multimedia framework and operations such as zooming in, zooming out or rotating are calculated Get information about the size of the video layer. Exemplarily, when a valid signal of Graphic Vsync arrives, step 801 is executed.

802. Send indication information of multiple Graphic Buffers to the HWC in the first thread.

It should be understood that SurfaceFlinger calls the hardware resources of the HWC in the first thread to realize the synthesis of the multi-layer graphics layer. In the specific implementation, the indication information of the Graphic Buffer storing the graphics layer data is sent to the HWC in the first thread. The indication information Pointing to a section of memory, HWC can obtain graphics layer data from the corresponding memory and process it according to the instruction information.

803. In the first thread, the HWC synthesizes the multi-layer graphics layer data, and performs hole processing on at least one graphics layer according to the size-related information of the video layer to obtain a synthesized graphics layer;

It should be understood that the synthesized graphics layer data includes a digging area, and the digging area is usually set to be transparent so that when the graphics layer and the video layer are synthesized, the video layer can be displayed through the digging area. In an optional case, HWC can dig holes in the graphics layer below the video layer, and then combine multiple graphics layers into one graphics layer; HWC can also combine multiple graphics layers into one graphics layer first, and then Then dig a hole in the synthesized graphics layer. Exemplarily, the composite graphics layer data obtained by the HWC processing can be stored in the FrameBuffer. It should be understood that step 803 will be executed only when the effective signal of Graphic Vsync arrives, and step 803 will be executed after step 801.

In an optional situation, if the HWC does not support the overlay of the graphics layer, SurfaceFlinger can call the GPU resource in the first thread to implement the overlay of the multi-layer graphics layer, which corresponds to the image processing architecture shown in Figure 4. In this case, 802 will send instructions for multiple Graphic Buffers to the GPU, so as to use the GPU's image processing function to realize the synthesis of multi-layer graphics layer data and the processing of digging holes in the graphics layer. The GPU returns the processing result to SurfaceFlinger, and SurfaceFlinger sends the FrameBuffer instruction information to the hardware synthesizer, and then the hardware synthesizer informs the display driver of the FrameBuffer instruction information.

804. Send the synthesized graphics layer to the display driver;

It should be understood that the HWC sends the instruction information of the FrameBuffer to the display driver, so that the display driver can obtain the synthesized graphics layer data from the corresponding memory according to the instruction information.

805. Send the instruction information of Video Buffer to Media HW in the second thread.

It should be understood that step 805 and step 801 are executed in parallel, that is, step 805 and step 801 can be executed at the same time.

806. When the first Video Buffer is received or when the size of the Video Buffer is detected to change, the second thread notifies the first thread.

Exemplarily, the second thread sends first notification information to the first thread, where the first notification information is used to indicate that the size of the video layer of the Main Thread has changed. After the first thread receives the notification information, SurfaceFlinger can obtain the updated Video Buffer size, and calculate the information related to the size of the video layer according to the updated Video Buffer size. Exemplarily, the size of the Video Buffer may include size information and rotation information.

807. In the second thread, Media HW processes the video layer data according to the information related to the size of the video layer received in the first thread.

In an optional case, Media HW first receives the information related to the size of the video layer in the first thread, and then Media HW receives the first frame of video layer data in the second thread, and then Media HW receives The information related to the size of the video layer processes the first frame of video layer data; in another case, Media HW first receives the first frame of video layer data in the second thread, and then Media HW receives it in the first thread. When it comes to the size-related information of the video layer, Media HW will not process the first frame of video layer data immediately, but will wait for the size-related information of the video layer to receive the first frame of video layer data. Perform processing to avoid processing errors in the first frame of video layer data.

808. Send the processed video layer to the display driver;

Media HW sends the instruction information of Video Buffer to the display driver so that the display driver can obtain the processed video layer data from the corresponding memory according to the instruction information.

809. Display Vsync arrives, and the display driver superimposes the video layer and the graphics layer to obtain display data and send it to the display device for display.

In the embodiments of this application, the composition of the graphics layer by HWC and the processing of the video layer by Media HW are performed in parallel by threads. Therefore, the processing of video images will no longer be affected by the composition of the graphics layer, and video playback will no longer be affected by graphics. The effect of layer composition can effectively solve the problem of frame loss caused by the time-consuming graphics layer composition during video playback. In addition, because setting the size of the video layer and digging holes in the graphics layer below the video layer are completed in the same thread, and inter-thread communication can be carried out between the first thread and the second thread, when the Video Buffer is When the size changes, the second thread can notify the first thread, which can ensure that the size of the hole in the graphics layer is exactly the same as the size of the video layer, so that the video layer and the graphics layer can be synchronized and displayed. Furthermore, since the composite that controls multiple graphics layers is Graphic Vsync, the signal that controls the composite of display video data and the refresh frame rate of the display device is Display Vsync. Graphic Vsync and Display Vsync are independent of each other, and the frame rate of Display Vsync can be greater than Graphic Vsync’s frame rate, so this image processing architecture can support the playback of high frame rate videos with a video frame rate higher than the graphics refresh frame rate.

The embodiment of the present application also provides another image data processing method. As shown in FIG. 9, the method includes:

S10. SurfaceFlinger creates a Video Thread in the initialization phase;

The Video Thread is a dedicated thread dedicated to processing video layer data, and the Video Thread may correspond to the aforementioned second thread. It should be understood that the thread that SurfaceFlinger receives the Video Buffer and the Video Thread are different threads. Illustratively, the thread that receives the Video Buffer can be called the first receiving thread. The Video Thread and the first receiving thread need to communicate between threads to facilitate the first A receiving thread can notify the Video Thread that a new Video Buffer is available.

S12. The Multimedia Framework sends the Video Buffer to the Buffer queue of the video layer;

Exemplarily, the buffer includes the Usage flag bit, which is used to indicate the type of the buffer. For example, when the Usage flag bit is the first indicator value, it means that the buffer is a Video Buffer, and when the Usage flag bit is the second indicator value. When, it means that the buffer is a Graphic Buffer. In an optional case, when the Usage flag bit is not occupied, it indicates that the buffer is a Graphic Buffer; when the Usage flag bit is occupied, it indicates that the buffer is a Video Buffer.

S14. SurfaceFlinger receives the buffer in the first receiving thread;

S16. When the received buffer is a Video Buffer, SurfaceFlinger informs the Video Thread that there is a new available Video Buffer;

Exemplarily, SurfaceFlinger can use the Usage flag to determine whether the received buffer is a Video Buffer. It should be understood that Video Thread is a cyclic thread. When SurfaceFlinger receives the Video Buffer sent by the media framework, it will notify Video Thread to process the video layer data. Video Thread will start processing after receiving the notification without waiting for the vertical synchronization signal.

S18. After the Video Thread receives the notification, it takes out the Video Buffer from the Buffer queue of the video layer and sends it to the Media HW for processing;

S20. Video Thread judges whether the received Video Buffer is the first Buffer received, if it is, then go to S24, if not, go to S22;

S22. Video Thread judges whether the size of the current Video Buffer has changed compared with the previous Video Buffer, and if there is a change, proceed to S24; if there is no change, no processing is performed, which can also be understood as the end of the branch;

It should be understood that S20 and S22 are two judging conditions in parallel, and any one of the conditions will be met, and S24 will be entered.

S24. Notify the Main Thread that the size of the video layer has changed;

In an optional situation, the Video Thread sends first notification information to the Main Thread. The first notification information is used to indicate that the size of the Main Thread video layer has changed so that the Main Thread can reset the video layer according to the updated size. Size-related information, for example, the first notification information may be carried on an identification bit. After Main Thread receives the notification information, SurfaceFlinger can obtain the size of the updated Video Buffer, and calculate the size of the video layer according to the size of the updated Video Buffer. It should be understood that the Main Thread is also created by SurfaceFlinger in the initialization phase, the Main Thread is a thread used to process graphics layer data, and the Main Thread is also used to set the size of the video layer. Main Thread can correspond to the aforementioned first thread. Therefore, when the Video Buffer received by the Video Thread is the first Video Buffer or the size of the Video Buffer has changed, the Main Thread needs to be notified so that the Main Thread can reset the size of the video layer. Main Thread sends the information related to the size of the re-set video layer to Media HW. It should be understood that the information related to the size of the set video layer is calculated by SurfaceFlinger. In an optional case, the multimedia framework sets the initial size of the video layer through the system's own API and transmits it to SurfaceFlinger. SurfaceFlinger can also capture or Perceive the user's operations such as zooming in, zooming out, or rotating the video, the initial size of the video layer and operations such as zooming in, zooming out, or rotating sent from the SurfaceFlinger integrated multimedia framework can calculate information about the size of the video layer.

S26. In the Main Thread, when Graphic Vsync arrives, the information related to the size of the set video layer is first sent to Media HW, and then the HWC synthesizes the multi-layer graphics layer and compares the video according to the information related to the size of the set video layer. The graphics layer below the layer is digging holes;

It should be understood that sending information related to the size of the set video layer to the Media HW can also be understood as setting the size of the video layer through the Media HW. The Main Thread is a cyclic thread. When SurfacFlinger receives the Graphic Buffer sent by the Graphic Framework, it notifies the Main Thread to process it, but the Main Thread does not start processing the graphics layer data in the Graphic Buffer until the Graphic Vsync arrives. The method also includes: in the Main Thread, SurfacFlinger sends the Graphic Buffers instruction information to the HWC, so that the HWC can synthesize the multi-layer graphics layer and perform hole processing on the related graphics layer. Optionally, if the HWC itself does not support superimposing graphics, when Graphic Vsync arrives, SurfacFlinger calls the GPU, and the GPU synthesizes the multi-layer graphics layer and performs hole processing on the related graphics layer.

Since setting the size of the video layer and digging the graphics layer below the video layer are completed in the same thread, it can ensure that the size of the digging hole in the graphics layer is exactly the same as the size of the video layer, so that the video layer and the video layer are exactly the same. Synchronous matching display of the graphics layer.

S28. Media HW sends the processed video layer to the display driver;

S30. The HWC sends the processed graphics layer to the display driver;

It should be understood that S28 and S30 can be performed synchronously, and S28 and S20-S24 are also parallel, and the order is not limited. For example, when Media HW in S18 obtains the processed video layer, it will execute S28 to convert the processed video layer. Send to the display driver, and when it is detected that the size of the first Video Buffer or Video Buffer is received, step S20 or S22 is executed, the processed video layer data is stored in the Video Buffer, and the Media HW will indicate the Video Buffer The information is sent to the display driver so that the display driver can obtain the processed video layer data from the corresponding memory; the processed graphics layer data is stored in the FrameBuffer, and the HWC sends the instruction information of the FrameBuffer to the display driver so that the display driver can Obtain the processed graphics layer data from the corresponding memory. Optionally, if the HWC itself does not support graphics overlay, the overlay of multiple graphics layers is done by the GPU, the GPU returns the FrameBuffer instruction information to SurfaceFlinger, and SurfaceFlinger then sends the FrameBuffer instruction information to HWC, so that HWC will The instruction information of FrameBuffer is sent to the display driver.

When S32, Display Vsync arrives, the display driver superimposes the processed video layer and the processed graphics layer to obtain display data and send it to the display device for display.

Display Vsync is also used to control the refresh frame rate of the display device. Since the processed graphics layer data contains the "dug area", and the size of the "dug area" is the same as the size of the processed video layer, after the display driver combines the two, the video layer and the "dug area" The area" can be completely matched, so that the video layer can be displayed simultaneously through the "dug area". In addition, since it is Graphic Vsync that controls the synthesis of multiple graphics layers, the signal that controls the synthesis of display video data and the refresh frame rate of the display device is Display Vsync. Graphic Vsync and Display Vsync are independent of each other, and the frame rate of Display Vsync can be greater than Graphic The frame rate of Vsync, so the image processing architecture can support the playback of high frame rate video with a video frame rate higher than the graphics refresh frame rate.

It should be understood that, for ease of understanding, the method embodiment corresponding to FIG. 9 describes the method in the form of steps, but the sequence number of the steps does not limit the execution order between the steps of the method. The steps performed in the Video Thread and the steps performed in the Main Thread are parallel.

FIG. 10 is a structural diagram of an exemplary image data processing apparatus provided by an embodiment of the application. The device includes a processor, and software instructions run on the processor to form a framework layer, a hardware abstraction layer, and a driver layer. Optionally, the device further includes a transmission interface through which the processor receives data sent by other devices or sends data to other devices. The transmission interface may be, for example, an HDMI interface, a V-By-One interface, an eDP interface, or MIPI. Interface, DP interface or Universal Serial Bus (Universal Serial Bus, USB) interface, etc. These interfaces are usually electrical communication interfaces, but may also be mechanical interfaces or other forms of interfaces, which are not limited in this embodiment. The device can be coupled with hardware resources such as a display, media hardware, or graphics hardware through a connector, a transmission line, or a bus. The device can be a processor chip with image or video processing functions. In an optional case, the device, media hardware, and graphics hardware can be integrated on one chip. In another optional situation, the device, media hardware, graphics hardware, and display may be integrated on one terminal. It should be understood that the image processing framework shown in FIGS. 5 and 6 may be run on the device shown in FIG. 10, and the device shown in FIG. 10 may be used to implement the method embodiments corresponding to FIGS. 7-9.

Exemplarily, the framework layer includes SurfaceFlinger, the hardware abstraction layer includes graphics hardware abstraction and media hardware abstraction, and the driver layer includes media hardware drivers, graphics hardware drivers, and display drivers. Among them, the graphics hardware abstraction corresponds to the graphics hardware driver, the media hardware abstraction corresponds to the media hardware driver, the graphics hardware can be called through the graphics hardware abstraction and the graphics hardware driver, and the media hardware can be called through the media hardware abstraction and the media hardware driver. For example, the graphics hardware driver is abstractly called by accessing the graphics hardware to realize the calling of the graphics hardware, and the media hardware driver is abstractly called by accessing the media hardware to realize the calling of the media hardware.

The framework layer is used in the first thread to call the graphics hardware through graphics hardware abstraction and graphics hardware drivers, to synthesize the multi-layer graphics layer, and to dig holes for at least one of the multi-layer graphics layers to obtain A synthesized graphic layer, the synthesized graphic layer including a digging area;

The framework layer is also used in the second thread to call Media HW through media hardware abstraction and media hardware driver to process the video layer to obtain the processed video layer; the processed video layer can pass through the digging area show;

The display driver is used to superimpose the synthesized graphics layer and the processed video layer to obtain display data.

In an optional case, the framework layer is also used to: in the first thread, set the size-related information of the video layer, and send the size-related information of the video layer to the media hardware abstraction;

The framework layer is specifically used to, in the first thread, call the graphics hardware to synthesize the multi-layer graphics layer according to the size-related information of the video layer, and to dig holes for at least one graphics layer to obtain the synthesized graphics layer; In the second thread, Media HW is called to process the video layer according to the size-related information of the video layer to obtain the processed video layer.

In an optional situation, the framework layer is specifically used to call the graphics hardware based on the first vertical synchronization signal to synthesize the multi-layer graphics layer in the first thread and to dig holes in at least one graphics layer Processing: The display driver is specifically used to superimpose the synthesized graphics layer and the processed video layer based on the second vertical synchronization signal to obtain display data; wherein the first vertical synchronization signal and the second vertical synchronization signal are independent of each other.

In an optional case, the framework layer is specifically used to send the size-related information of the video layer to the media hardware abstraction in the first thread when the effective signal of the first vertical synchronization signal arrives. Then the graphics hardware is called to synthesize the multi-layer graphics layer, and the at least one graphics layer is digged according to the size-related information of the video layer.

In an optional case, when the second thread obtains the first video buffer Video Buffer, it sends the first notification information to the first thread; the framework layer is also used to receive the first notification in the first thread After the information, in the first thread, the size of the first Video Buffer is obtained, and the size-related information of the video layer is set according to the size of the first Video Buffer.

In an optional situation, the graphics hardware includes an HWC and a GPU, as shown in FIG. 11 is a schematic structural diagram of another exemplary image processing apparatus provided in an embodiment of the present application. Correspondingly, the hardware abstraction layer includes the HWC abstraction, and the driver layer includes the HWC driver and the GPU driver.

In an optional case, the SurfaceFlinger of the framework layer is used to create the first thread and the second thread in the initialization phase; SurfaceFlinger is also used to notify the first thread to process the Graphic Buffer when the Graphic Buffer is received. When receiving the Video Buffer, notify the second thread to process the video layer data in the Video Buffer; among them, a layer of graphics layer data is stored in the Graphic Buffer, and a layer of video layer data is stored in the Video Buffer.

In an optional case, the Media HW abstraction is specifically used to: receive the first frame of video layer data in the second thread; receive the size-related information of the video layer in the first thread; the frame layer is specifically used to : In the second thread, Media HW abstractly calls Media HW to process the first frame of video layer data according to the size-related information of the video layer.

As shown in FIG. 12, it is a schematic structural diagram of another exemplary image processing apparatus provided by an embodiment of this application. The device includes: a framework layer, graphics hardware, media hardware, and a display driver. The framework layer and the display driver are part of the operating system formed by software instructions running on the processor. Graphics hardware and media hardware can be coupled with the processor through connectors, interfaces, transmission lines or buses. These interfaces are usually electrical communication interfaces, but may also be mechanical interfaces or other forms of interfaces. Exemplarily, the graphics hardware may include GPU and HWC.

The framework layer is used in the first thread to call the graphics hardware to synthesize the multi-layer graphics layer and to dig holes for at least one of the multi-layer graphics layers to obtain the synthesized graphics layer, the synthesized graphics layer Including the digging area;

This framework layer is also used in the second thread to call Media HW to process the video layer to obtain the processed video layer; the processed video layer can be displayed through the digging area;

In an optional situation, the software instructions running on the processor are also used to form a hardware abstraction layer, graphics hardware drivers, and media hardware drivers. The hardware abstraction layer includes graphics hardware drivers corresponding to graphics hardware and media Media hardware driver corresponding to the hardware. The frame layer is specifically used to call graphics hardware through graphics hardware abstraction and graphics hardware drivers, and the frame layer is specifically used to call media hardware through media hardware abstractions and media hardware drivers.

In an optional case, the framework layer is also used to: in the first thread, set the size-related information of the video layer, and send the size-related information of the video layer to Media HW; the framework layer is specifically used Therefore, in the first thread, the graphics hardware is called to synthesize the multi-layer graphics layer according to the information related to the size of the video layer and the at least one graphics layer is digged to obtain the synthesized graphics layer; In the second thread, the Media HW is called to process the video layer according to the size-related information of the video layer to obtain the processed video layer.

In an optional case, the framework layer is specifically used to, based on the first vertical synchronization signal, call the graphics hardware in the first thread to synthesize the multi-layer graphics layer and perform the at least one graphics layer Digging processing; the display driver is specifically used to, when the second vertical synchronization signal arrives, superimpose the synthesized graphics layer and the processed video layer to obtain the display data; wherein, the first vertical synchronization signal and The second vertical synchronization signals are independent of each other.

In an optional situation, the framework layer is specifically used to send the size-related information of the video layer to the Media in the first thread when the effective signal of the first vertical synchronization signal arrives. HW, and then call the graphics hardware to synthesize the multi-layer graphics layer, and perform digging processing on the at least one graphics layer according to the size-related information of the video layer.

In an optional situation, when the second thread obtains the first video buffer Video Buffer, the first notification information is sent to the first thread; the framework layer is also used to receive After the first notification information, in the first thread, obtain the size of the first Video Buffer, and set the size-related information of the video layer according to the size of the first Video Buffer; or, when the first When the second thread detects that the size of the Video Buffer has changed, it sends first notification information to the first thread; the framework layer is also used to: after the first thread receives the first notification information, in the first thread , Obtain the size of the changed Video Buffer, and set the size-related information of the video layer according to the size of the changed Video Buffer; wherein, the Video Buffer stores a piece of data of the video layer, and the size of the Video Buffer It is related to the size of the video layer.

In an optional case, the graphics hardware includes a hardware synthesizer HWC, and the framework layer is specifically used to: in the first thread, call the HWC to synthesize the multi-layer graphics layer and perform the synthesis of at least one graphics layer Perform the digging process to obtain the synthesized graphics layer; the display driver superimposes the synthesized graphics layer and the processed video layer to obtain the display data, the HWC is also used to: send the synthesized graphics layer to the display Driver; The Media HW is also used to send the processed video layer to the display driver.

In an optional case, the graphics hardware includes a graphics processor GPU, and the framework layer is specifically used to: in the first thread, call the GPU to synthesize the multi-layer graphics layer and dig at least one graphics layer. Hole processing to obtain the synthesized graphics layer; the display driver superimposes the synthesized graphics layer and the processed video layer, and before the display data is obtained, the GPU is also used to: return the synthesized graphics layer to the framework Layer; the framework layer is also used to send the synthesized graphics layer to the HWC; the HWC is also used to send the synthesized graphics layer to the display driver; the Media HW is also used to send the processed graphics layer The video layer is sent to the display driver.

In an optional case, the framework layer includes SurfaceFlinger, which is used to: create the first thread and the second thread in the initialization phase; and the SurfaceFlinger is also used to, when receiving the Graphic Buffer, Notify the first thread to process the graphics layer data in the Graphic Buffer; when the Video Buffer is received, notify the second thread to process the video layer data in the Video Buffer; wherein the Graphic Buffer stores a layer of graphics layer data , The Video Buffer stores a layer of video layer data.

In an optional case, the Media HW is specifically used to: receive the first frame of video layer data in the second thread; receive the size-related information of the video layer in the first thread; the frame The layer is specifically used to: in the second thread, call the Media HW to process the first frame of video layer data according to the size-related information of the video layer.

It should be understood that the image processing framework shown in FIGS. 5 and 6 may be run on the device shown in FIG. 12, and the device shown in FIG. 12 may be used to implement the method embodiments corresponding to FIGS. 7-9. For relevant detailed explanations, reference may be made to the descriptions of the foregoing method embodiments.

As shown in FIG. 13, it is a schematic structural diagram of another exemplary image processing apparatus provided by an embodiment of this application. The device includes a processor, graphics hardware, and media hardware, and software instructions run on the processor to form a framework layer and a display driver;

The framework layer is used for invoking the graphics hardware to synthesize multiple graphics layers and digging holes for at least one graphics layer in the first thread to obtain a synthesized graphics layer, and the synthesized graphics layer includes the digging area;

The framework layer is used to call the media hardware to process the video layer in the second thread to obtain the processed video layer;

As shown in Figure 13, the software instructions running on the processor are also used to form a hardware abstraction layer, which includes graphics hardware abstraction and media hardware abstraction. Correspondingly, although not shown in FIG. 13, the driver layer also includes graphics hardware drivers and media hardware drivers. Graphics hardware and media hardware can be coupled with the processor through connectors, interfaces, transmission lines, or buses. In an optional case, the graphics hardware includes HWC and GPU, as shown in FIG. 14. Correspondingly, the hardware abstraction layer includes the HWC abstraction, and the driver layer includes the HWC driver and the GPU driver.

It should be understood that the aforementioned image processing framework shown in Figures 5 and 6 can be run on the device shown in Figure 13 or Figure 14, and the device shown in Figure 13 or Figure 14 can be used to implement the aforementioned corresponding to Figures 7 to 9的方法实施例。 Example of the method. It is not limited here in detail.

The embodiments of the present application also provide a computer-readable storage medium that stores instructions in the computer-readable storage medium, and when it runs on a computer or a processor, the computer or the processor executes the method provided in the embodiments of the present application Part or all of the functions.

The embodiments of the present application also provide a computer program product containing instructions, which when run on a computer or a processor, enable the computer or the processor to perform part or all of the functions in the methods provided in the embodiments of the present application.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, a person of ordinary skill in the art should understand that it can still implement the foregoing The technical solutions recorded in the examples are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

A method for image data processing, characterized in that the method includes:

Synthesize the multi-layer graphics layer in the first thread and perform digging processing on at least one graphics layer of the multi-layer graphics layer to obtain a synthesized graphics layer, and the synthesized graphics layer includes a digging area;

Processing the video layer in the second thread to obtain a processed video layer, and the processed video layer can be displayed through the digging area;

The synthesized graphics layer and the processed video layer are superimposed to obtain display data.
The method according to claim 1, characterized in that, before the multi-layer graphics layer is synthesized in the first thread and the hole processing is performed on at least one graphics layer to obtain the synthesized graphics layer, the method further include:

Setting information related to the size of the video layer in the first thread;

The digging process for at least one layer of graphics specifically includes:

Digging holes for the at least one graphics layer according to information related to the size of the video layer;

The processing of the video layer in the second thread to obtain the processed video layer specifically includes:

In the second thread, the video layer is processed according to information related to the size of the video layer to obtain the processed video layer.
The method according to claim 1 or 2, characterized in that:

Synthesize the multi-layer graphics layer in the first thread based on the first vertical synchronization signal and perform digging processing on the at least one graphics layer;

Superimposing the synthesized graphics layer and the processed video layer based on a second vertical synchronization signal to obtain the display data;

Wherein, the first vertical synchronization signal and the second vertical synchronization signal are independent of each other.
The method of claim 3, wherein:

When the effective signal of the first vertical synchronization signal arrives, in the first thread, the size-related information of the video layer is first set, and then the multi-layer graphics layer is synthesized and based on the video layer The information related to the size of the at least one layer of graphics is processed by digging holes.
The method according to claim 3 or 4, wherein the frame rate of the first vertical synchronization signal is less than the frame rate of the second vertical synchronization signal.
The method according to any one of claims 1 to 5, further comprising:

When the second thread obtains the first video buffer Video Buffer, send the first notification information to the first thread;

In the first thread, information related to the size of the video layer is set according to the size of the first Video Buffer; or,

When the second thread detects that the size of the Video Buffer has changed, sending first notification information to the first thread;

In the first thread, the information related to the size of the video layer is reset according to the size of the changed Video Buffer;

Wherein, the Video Buffer stores one piece of data of the video layer, and the size of the Video Buffer is related to the size of the video layer.
The method according to any one of claims 1 to 6, characterized in that, in the first thread, the multi-layer graphics layer is synthesized and the at least one graphics layer is digged to obtain a synthesized graphics layer, Specifically:

In the first thread, the hardware synthesizer HWC synthesizes the multiple graphics layers and performs hole processing on the at least one graphics layer to obtain the synthesized graphics layer.
The method according to any one of claims 1 to 6, characterized in that, in the first thread, the multi-layer graphics layer is synthesized and the at least one graphics layer is digged to obtain a synthesized graphics layer, Specifically:

In the first thread, the graphics processor GPU synthesizes the multiple graphics layers and performs hole processing on the at least one graphics layer to obtain the synthesized graphics layer.
The method according to any one of claims 1 to 8, wherein after the setting of the size-related information of the video layer in the first thread, the method further comprises:

In the first thread, the information related to the size of the video layer is sent to the media hardware Media HW;

The processing of the video layer in the second thread to obtain the processed video layer specifically includes:

In the second thread, the Media HW processes the video layer according to the size-related information of the video layer to obtain the processed video layer.
The method according to any one of claims 1 to 9, characterized in that,

The superimposing the synthesized graphics layer and the processed video layer to obtain display data specifically includes:

The display driver superimposes the synthesized graphics layer and the processed video layer to obtain the display data.
The method according to any one of claims 1 to 8, further comprising:

SurfaceFlinger creates the first thread and the second thread in the initialization phase;

When the SurfaceFlinger receives the Graphic Buffer, notify the first thread to process the graphics layer data in the Graphic Buffer;

When the SurfaceFlinger receives the Video Buffer, notify the second thread to process the video layer data in the Video Buffer;

Wherein, a layer of graphics layer data is stored in the Graphic Buffer, and a layer of video layer data is stored in the Video Buffer.
An image data processing device, characterized in that, the device includes a processor on which software instructions run to form a framework layer, a hardware abstraction layer HAL, and a driver layer, and the HAL includes graphics hardware abstraction and Media hardware Media HW is abstracted, and the driver layer includes graphics hardware driver, media hardware driver and display driver;

The framework layer is used for invoking the graphics hardware through the graphics hardware abstraction and the graphics hardware driver in the first thread, to synthesize the multi-layer graphics layer, and to compose at least one layer of the graphics in the multi-layer graphics layer Performing digging processing on the layer to obtain a synthesized graphic layer, the synthesized graphic layer including the digging area;

The framework layer is used in the second thread to call Media HW through the media hardware abstraction and the media hardware driver to process the video layer to obtain the processed video layer; the processed video layer can Display through the digging area;

The display driver is used to superimpose the synthesized graphics layer and the processed video layer to obtain display data.
The device according to claim 12, wherein the frame layer is also used for:

In the first thread, setting information related to the size of the video layer, and sending the information related to the size of the video layer to the Media HW abstraction;

The frame layer is specifically used for,

In the first thread, the graphics hardware is called to synthesize the multi-layer graphics layer according to the size-related information of the video layer, and the at least one graphics layer is digged to obtain the synthesis Graphics layer

In the second thread, the Media HW is called to process the video layer according to the size-related information of the video layer to obtain the processed video layer.
The device according to claim 12 or 13, characterized in that:

The framework layer is specifically configured to call the graphics hardware to synthesize the multi-layer graphics layer in the first thread based on the first vertical synchronization signal and perform hole-digging processing on the at least one graphics layer;

The display driving is specifically configured to superimpose the synthesized graphics layer and the processed video layer based on a second vertical synchronization signal to obtain the display data;

Wherein, the first vertical synchronization signal and the second vertical synchronization signal are independent of each other.
The device of claim 14, wherein:

The framework layer is specifically used to, when the effective signal of the first vertical synchronization signal arrives, in the first thread, first send the size-related information of the video layer to the Media HW abstraction, and then Invoking the graphics hardware to synthesize the multi-layer graphics layer, and performing hole processing on the at least one graphics layer according to information related to the size of the video layer.
The device according to claim 14 or 15, wherein the frame rate of the first vertical synchronization signal is lower than the frame rate of the second vertical synchronization signal.
The device according to any one of claims 12 to 16, characterized in that:

When the second thread obtains the first video buffer Video Buffer, send the first notification information to the first thread;

The framework layer is also configured to, after the first thread receives the first notification information, in the first thread, obtain the size of the first Video Buffer, and according to the first The size of Video Buffer sets information related to the size of the video layer; or,

When the second thread detects that the size of the Video Buffer has changed, sending first notification information to the first thread;

The framework layer is also used to, after the first thread receives the first notification information, in the first thread, obtain the size of the changed Video Buffer, and according to the changed Video Buffer size The size of sets the information related to the size of the video layer;

Wherein, the Video Buffer stores one piece of data of the video layer, and the size of the Video Buffer is related to the size of the video layer.
The apparatus according to any one of claims 12 to 17, wherein the graphics hardware includes a hardware synthesizer HWC, the graphics hardware abstraction includes HWC abstraction, the graphics hardware driver includes an HWC driver, and the framework layer Specifically used for:

In the first thread, call the HWC through the HWC abstraction and the HWC driver, synthesize the multi-layer graphics layer, and perform hole processing on the at least one graphics layer to obtain the synthesis Graphics layer

The display driver superimposes the synthesized graphics layer and the processed video layer, and before the display data is obtained, the HWC abstraction is also used for:

Sending the synthesized graphics layer to the display driver;

The Media HW abstraction is also used to send the processed video layer to the display driver.
The device according to any one of claims 12 to 17, wherein the graphics hardware includes a graphics processor GPU, the graphics hardware driver includes a GPU driver, and the framework layer is specifically used for:

In the first thread, the GPU is invoked by the GPU driver, the multi-layer graphics layer is synthesized, and the at least one graphics layer is burrowed to obtain the synthesized graphics layer;

The display driver superimposes the synthesized graphics layer and the processed video layer, and before the display data is obtained, the GPU is also used for:

Returning the synthesized graphics layer to the frame layer;

The framework layer is also used to send the synthesized graphics layer to the HWC abstraction;

The HWC abstraction is also used to send the synthesized graphics layer to the display driver;

The Media HW abstraction is also used to send the processed video layer to the display driver.
The device according to any one of claims 12 to 19, wherein the frame layer comprises SurfaceFlinger, and the SurfaceFlinger is used for:

Creating the first thread and the second thread in the initialization phase;

The SurfaceFlinger is also used to notify the first thread to process the graphics layer data in the Graphic Buffer when receiving the Graphic Buffer Graphic Buffer;

When the Video Buffer is received, notify the second thread to process the video layer data in the Video Buffer;

Wherein, a layer of graphics layer data is stored in the Graphic Buffer, and a layer of video layer data is stored in the Video Buffer.
The device according to any one of claims 12 to 20, wherein the Media HW abstraction is specifically used for:

Receiving the first frame of video layer data in the second thread;

Receiving information about the size of the video layer in the first thread;

The framework layer is specifically used for:

In the second thread, the Media HW abstractly calls the Media HW to process the first frame of video layer data according to the size-related information of the video layer.
An image data processing device, characterized in that the device includes:

The framework layer is used in the first thread to call the graphics hardware to synthesize the multi-layer graphics layer and to perform digging processing on at least one of the multi-layer graphics layers to obtain a synthesized graphics layer, the synthesis The graphic layer of includes the digging area;

The graphics hardware;

Media HW;

The framework layer is also used to call the Media HW to process the video layer in the second thread to obtain the processed video layer; the processed video layer can be displayed through the excavated area;

The display driver is used to superimpose the synthesized graphics layer and the processed video layer to obtain display data.
The device according to claim 22, wherein the frame layer is also used for:

In the first thread, setting information related to the size of the video layer, and sending the information related to the size of the video layer to the Media HW;

The frame layer is specifically used for,

In the first thread, the graphics hardware is called to synthesize the multi-layer graphics layer according to the size-related information of the video layer, and the at least one graphics layer is digged to obtain the synthesis Graphics layer

In the second thread, the Media HW is called to process the video layer according to the size-related information of the video layer to obtain the processed video layer.
The device according to claim 22 or 23, wherein:

The framework layer is specifically configured to, based on a first vertical synchronization signal, call the graphics hardware in the first thread to synthesize the multi-layer graphics layer and perform hole-digging processing on the at least one graphics layer;

The display driving is specifically configured to, when a second vertical synchronization signal arrives, superimpose the synthesized graphics layer and the processed video layer to obtain the display data;

Wherein, the first vertical synchronization signal and the second vertical synchronization signal are independent of each other.
The device of claim 24, wherein:

The framework layer is specifically configured to, when the effective signal of the first vertical synchronization signal arrives, in the first thread, first send the size-related information of the video layer to the Media HW, and then call The graphics hardware synthesizes the multi-layer graphics layer and performs hole processing on the at least one graphics layer according to information related to the size of the video layer.
The device according to claim 24 or 25, wherein the frame rate of the first vertical synchronization signal is less than the frame rate of the second vertical synchronization signal.
The device according to any one of claims 22 to 26, characterized in that:

When the second thread obtains the first video buffer Video Buffer, send the first notification information to the first thread;

The framework layer is also configured to, after the first thread receives the first notification information, in the first thread, obtain the size of the first Video Buffer, and according to the first The size of Video Buffer sets information related to the size of the video layer; or,

When the second thread detects that the size of the Video Buffer has changed, sending first notification information to the first thread;

The framework layer is also used to, after the first thread receives the first notification information, in the first thread, obtain the size of the changed Video Buffer, and according to the changed Video Buffer size The size of sets the information related to the size of the video layer;

Wherein, the Video Buffer stores one piece of data of the video layer, and the size of the Video Buffer is related to the size of the video layer.
The device according to any one of claims 22 to 27, wherein the graphics hardware comprises a hardware synthesizer HWC, and the framework layer is specifically used for:

In the first thread, call the HWC to synthesize the multi-layer graphics layer and perform hole-digging processing on the at least one graphics layer to obtain the synthesized graphics layer;

The display driver superimposes the synthesized graphics layer and the processed video layer, and before the display data is obtained, the HWC is also used for:

Sending the synthesized graphics layer to the display driver;

The Media HW is also used to send the processed video layer to the display driver.
The device according to any one of claims 22 to 27, wherein the graphics hardware comprises a graphics processing unit (GPU), and the framework layer is specifically used for:

In the first thread, invoking the GPU to synthesize the multi-layer graphics layer and perform digging processing on the at least one graphics layer to obtain the synthesized graphics layer;

The display driver superimposes the synthesized graphics layer and the processed video layer, and before the display data is obtained, the GPU is also used for:

Returning the synthesized graphics layer to the frame layer;

The frame layer is also used to send the synthesized graphics layer to the HWC;

The HWC is also used to send the synthesized graphics layer to the display driver;

The Media HW is also used to send the processed video layer to the display driver.
The device according to any one of claims 22 to 29, wherein the frame layer comprises SurfaceFlinger, and the SurfaceFlinger is used for:

Creating the first thread and the second thread in the initialization phase;

The SurfaceFlinger is also used to,

When receiving the Graphic Buffer Graphic Buffer, notify the first thread to process the graphics layer data in the Graphic Buffer;

When the Video Buffer is received, notify the second thread to process the video layer data in the Video Buffer;

Wherein, a layer of graphics layer data is stored in the Graphic Buffer, and a layer of video layer data is stored in the Video Buffer.
The device according to any one of claims 22 to 30, wherein the Media HW is specifically used for:

Receiving the first frame of video layer data in the second thread;

Receiving information about the size of the video layer in the first thread;

The framework layer is specifically used for:

In the second thread, the Media HW is called to process the first frame of video layer data according to the information related to the size of the video layer.
A computer-readable storage medium in which instructions are stored, when the instructions are run on a computer or a processor, the computer or the processor is made to execute any one of claims 1 to 11 The method described.
A computer program product containing instructions, when it runs on a computer or processor, causes the computer or processor to execute the method according to any one of claims 1 to 11.