CN113050899A - Method and system for directly displaying video and UI drm based on Wayland protocol - Google Patents
Method and system for directly displaying video and UI drm based on Wayland protocol Download PDFInfo
- Publication number
- CN113050899A CN113050899A CN202110167640.1A CN202110167640A CN113050899A CN 113050899 A CN113050899 A CN 113050899A CN 202110167640 A CN202110167640 A CN 202110167640A CN 113050899 A CN113050899 A CN 113050899A
- Authority
- CN
- China
- Prior art keywords
- window
- video
- protocol
- layer
- weston
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
- G06F13/28—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal
Abstract
The invention discloses a direct display method of video and UI drm based on a Wayland protocol, which comprises the following steps: the video end applies for creating a first drawing surface of a video window to a Weston open source library through a standard desktop protocol and generates a video buffer zone corresponding to the first drawing surface; the UI end applies for creating a second drawing surface of the UI window from a Weston open source library through an yvideo-shell-protocol internal protocol and generates a UI buffer area corresponding to the second drawing surface; after receiving display requests of a video end and a UI end, the Weston open source library submits a video buffer area and a UI buffer area to a hardware layer unit of a direct rendering manager respectively according to the sequence of creating a first rendering surface and the sequence of creating a second rendering surface, and submits the display requests of DRM; the DRM combines the video buffer and the video buffer in a fixed hardware pipeline according to a display request to output a mixed video and UI. The invention realizes the mixed output of the video/UI under the scene of double-screen 4K resolution, the video can reach 60 frames, and the UI can reach 30 frames.
Description
Technical Field
The invention relates to the technical field of video and UI rendering management, in particular to a method and a system for drm direct display of videos and UIs based on a Wayland protocol and a computer-readable storage medium.
Background
In a video system, a common scene is that video and UI are overlapped and mixed to be output, and a video page is divided into layouts such as single-side full screen, 1+ N, equal division and the like, so that a video/UI mixed display module is required to realize corresponding functions. In a Linux system, a common display driver is divided into two modes, namely fbdev and drm (Direct Rendering Manager), and application access modes corresponding to the two display drivers are Framebuffer and libdrm respectively. Common solutions are as follows:
the FrameBuffer protocol: the video and UI are mixed using a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and the resultant buffer is then submitted to the driver for display. The scheme has the characteristics of better implementation, but for multi-screen high-resolution pictures, the performance of a CPU or a GPU is greatly consumed, and the phenomena of insufficient refresh frame rate, frame blocking, frame dropping and the like are easily caused.
gpu + drm protocol: the GPU is used for mixing the video and the UI, and then the synthesized buffer is submitted to the drive display, the defects of the scheme are the same as those of the FrameBuffer scheme, the performance consumption of the GPU is large, and the phenomena of pause and frame dropping are easy to occur.
drm direct visualization protocol: the video and the UI are in the same process, and respective buffers are directly displayed according to Plane submission. The scheme has the characteristics that the overall performance is good, the defects are that the overall structure is complex, the UI/video is not easy to be opened independently, and in addition, the access to the third party App is easy to be limited. Open source weston + drm scheme: the video and UI screens are composited using an open source weston graphics desktop and rendered for display by drm. The scheme has the characteristics of easy realization and capability of meeting the normal performance requirement of the video in a single video and single UI scene. The defects are that interface window management is complex, plane allocation of drm does not accord with a video priority principle, problems encountered by GPU synthesis easily occur in a multilayer video scene, surface management of a video or UI client is troublesome, most importantly, window levels of the video and the UI cannot be guaranteed, and the video is easily displayed on the UI.
Disclosure of Invention
The invention aims to provide a method and a system for directly displaying drm of videos and UIs based on the Wayland protocol and a computer readable storage medium, so as to solve the problems that when the videos and the UIs are displayed in a mixed mode, the performance consumption is high, the window levels of the videos and the UIs cannot be guaranteed, and the videos are easily displayed on the UIs.
In order to achieve the above object, an embodiment of the present invention provides a drm direct display method for videos and UIs based on the Wayland protocol, including:
a video end applies for creating a first drawing surface of a video window to a Weston open source library through a standard desktop protocol, and generates a video buffer zone corresponding to the first drawing surface; wherein the standard desktop protocol comprises xdg-shell or wl-shell standard protocol;
the UI end applies for creating a second drawing surface of a UI window from the Weston open source library through an yvideo-shell-protocol internal protocol and generates a UI buffer area corresponding to the second drawing surface; wherein the ylvideo-shell-protocol internal protocol is a proprietary protocol established based on the Wayland standard;
after the Weston open source library receives the display requests of the video end and the UI end, the Weston open source library submits the video buffer area and the UI buffer area to a hardware layer unit of a direct rendering manager according to the sequence of creating the first drawing surface and the second drawing surface, and submits the display requests of the direct rendering manager;
the direct rendering manager synthesizes the video buffer and the video buffer in a fixed hardware pipeline according to the display request to output a mixed video and UI.
In one embodiment, before the step of applying, by the video terminal, to the Weston open source library through a standard desktop protocol to create a first drawing surface of a video window, and generating a video buffer corresponding to the first drawing surface, the method further includes:
the Weston open source library completes registration of the ylvideo-shell-protocol internal protocol and registration of the standard desktop protocol api callback function, and places a UI window created through the ylvideo-shell-protocol internal protocol and a video window created through the standard desktop protocol into corresponding window level objects; the window level objects comprise video layer window objects and UI layer window objects, the video layer window objects are used for storing video layer windows, and the UI layer window objects are used for storing UI layer windows.
In one embodiment, the step of completing registration of the ylvideo-shell-protocol internal protocol and registration of the standard desktop protocol api callback function in the Weston open source library, and placing the UI window created by the ylvideo-shell-protocol internal protocol and the video window created by the standard desktop protocol into the corresponding window hierarchy object specifically includes:
creating a window management module object and a window hierarchy object of a Weston open source library;
setting the hierarchy attribute of the video LAYER window object as a WESTON _ LAYER _ POSITION _ BOTTOM _ UI with a first preset value, and setting the hierarchy attribute of the UI LAYER window object as a WESTON _ LAYER _ POSITION _ NORMAL with a second preset value;
creating a weston _ desktop object according to the structure body of the defined weston _ desktop _ api callback function, and creating a yyydeo-shell global resource object according to the structure body of the defined yydeo-shell-protocol callback function;
and monitoring the standard desktop protocol and the yvido-shell-protocol internal protocol, creating a video window into the list of the video layer window objects when receiving a video window creation request sent by the standard desktop protocol, and creating a UI window into the list of the UI layer window objects when receiving a UI window creation request sent by the yvido-shell-protocol internal protocol.
In one embodiment, after receiving the display requests of the video end and the UI end, the Weston open source library submits the video buffer and the UI buffer to a hardware layer unit of a direct rendering manager according to an order of creating the first drawing surface and creating the second drawing surface, and submits the display requests of the direct rendering manager, including:
the Weston open source library sequentially forms a new window queue with the Weston _ view window objects in the video layer window object and the UI layer window object according to the sequence of creating the first drawing surface and the second drawing surface;
traversing the window queue to obtain a window which has an effective buffer area and is not transparent in a full screen or a last window which has an effective buffer area, and taking the window as an output window corresponding to the main hardware layer;
in the window queue, from the output window corresponding to the main hardware layer, windows are sequentially distributed to the corresponding overlapped hardware layers from bottom to top;
and binding the video buffer area of the video window and the UI buffer area of the UI window to a frame buffer area of a corresponding hardware layer unit of the direct rendering manager in sequence, and submitting a display request of the direct rendering manager.
The embodiment of the invention provides a video and UI drm direct display system based on a Wayland protocol, which comprises: the system comprises a video end, a UI end, a Weston open source library and a direct rendering manager;
the video end is used for applying for creating a first drawing surface of a video window to a Weston open source library through a standard desktop protocol and generating a video buffer area corresponding to the first drawing surface; wherein the standard desktop protocol comprises xdg-shell or wl-shell standard protocol;
the UI end is used for applying for creating a second drawing surface of a UI window from the Weston open source library through an ylvideo-shell-protocol internal protocol and generating a UI buffer area corresponding to the second drawing surface; wherein the ylvideo-shell-protocol internal protocol is a proprietary protocol established based on the Wayland standard;
the Weston open source library comprises a window management module, wherein the window management module is used for submitting the video buffer area and the UI buffer area to a hardware layer unit of a direct rendering manager and submitting a display request of the direct rendering manager according to the sequence of creating the first drawing surface and the second drawing surface after receiving the display requests of the video end and the UI end;
the direct rendering manager synthesizes the video buffer and the video buffer in a fixed hardware pipeline according to the display request to output a mixed video and UI.
In one embodiment, the window management module is further configured to:
completing registration of the yvido-shell-protocol internal protocol and registration of the standard desktop protocol api callback function, and placing a UI window created through the yvido-shell-protocol internal protocol and a video window created through the standard desktop protocol into a corresponding window level object; the window level objects comprise video layer window objects and UI layer window objects, the video layer window objects are used for storing video layer windows, and the UI layer window objects are used for storing UI layer windows.
In one embodiment, the window management module is further configured to:
creating a window management module object and a window hierarchy object of a Weston open source library;
setting the hierarchy attribute of the video LAYER window object as a WESTON _ LAYER _ POSITION _ BOTTOM _ UI with a first preset value, and setting the hierarchy attribute of the UI LAYER window object as a WESTON _ LAYER _ POSITION _ NORMAL with a second preset value;
creating a weston _ desktop object according to the structure body of the defined weston _ desktop _ api callback function, and creating a yyydeo-shell global resource object according to the structure body of the defined yydeo-shell-protocol callback function;
and monitoring the standard desktop protocol and the yvido-shell-protocol internal protocol, creating a video window into the list of the video layer window objects when receiving a video window creation request sent by the standard desktop protocol, and creating a UI window into the list of the UI layer window objects when receiving a UI window creation request sent by the yvido-shell-protocol internal protocol.
In one embodiment, the window management module is further configured to:
sequentially forming a new window queue by the webton _ view window objects in the video layer window object and the UI layer window object according to the sequence of creating the first drawing surface and the second drawing surface;
traversing the window queue to obtain a window which has an effective buffer area and is not transparent in a full screen or a last window which has an effective buffer area, and taking the window as an output window corresponding to the main hardware layer;
in the window queue, from the output window corresponding to the main hardware layer, windows are sequentially distributed to the corresponding overlapped hardware layers from bottom to top;
and binding the video buffer area of the video window and the UI buffer area of the UI window to a frame buffer area of a corresponding hardware layer unit of the direct rendering manager in sequence, and submitting a display request of the direct rendering manager.
Embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the drm direct display method for videos and UIs based on the Wayland protocol in any of the above embodiments.
Compared with the prior art, in the method and the system for directly displaying the video and the drim of the UI based on the Wayland protocol and the computer-readable storage medium, the window management module is newly developed based on the Weston open source library of the Wayland protocol: and modifying a Plane (hardware layer) distribution rule of the optimized Drm-background, so that both buffers (Buffer areas) after video decoding and buffers (UIs) drawn are directly bound to a Plane (hardware layer unit) of the Drm (direct rendering manager), and finally, mixed output is realized through a fixed hardware pipeline Pipe. Therefore, video/UI mixed output under the scene of double-screen 4K resolution is realized, the video can reach 60 frames, and the UI can reach 30 frames.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a method for direct display of video and UI drm according to the Wayland protocol according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a video and UI drm direct display system based on the Wayland protocol according to an embodiment of the present invention;
fig. 3 is a schematic diagram illustrating a storage location of a weston _ layer of a video window and a storage location of a weston _ layer of a UI window according to an embodiment of the present invention;
fig. 4 is a schematic diagram illustrating an initialization flow of the yvideo-shell module according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a process of assigning a View window to a drm plane (direct rendering manager hardware layer) according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a computer terminal device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be understood that the step numbers used herein are for convenience of description only and are not intended as limitations on the order in which the steps are performed.
It is to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
The terms "comprises" and "comprising" indicate the presence of the described features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The term "and/or" refers to and includes any and all possible combinations of one or more of the associated listed items.
In the embodiment of the invention, the abbreviation, English and key term definition list is as follows:
1. CPU, Central Processing Unit
2. GPU, Graphics Processing Unit, Graphics processor
3. DRM, Direct Rendering Manager
4. DMA, Direct Memory Access
Referring to fig. 1, an embodiment of the present invention provides a method for drm direct display of videos and UIs based on the Wayland protocol, including the following steps:
s10, the video end applies for creating a first drawing surface of a video window to a Weston open source library through a standard desktop protocol, and generates a video buffer area corresponding to the first drawing surface; wherein the standard desktop protocol comprises xdg-shell or wl-shell standard protocol;
s20, the UI end applies for creating a second drawing surface of the UI window from the Weston open source library through the ylvideo-shell-protocol internal protocol, and generates a UI buffer area corresponding to the second drawing surface; wherein the ylvideo-shell-protocol internal protocol is a proprietary protocol established based on the Wayland standard;
s30, after the Weston open source library receives the display requests of the video end and the UI end, submitting the video buffer area and the UI buffer area to a hardware layer unit of a direct rendering manager according to the sequence of creating the first drawing surface and the second drawing surface, and submitting the display requests of the direct rendering manager;
s40, the direct rendering manager synthesizes the video buffer and the video buffer in a fixed hardware pipeline according to the display request to output a mixed video and UI.
Referring to fig. 2, an embodiment of the present invention further provides a drm direct display system for video and UI based on the Wayland protocol, including: the system comprises a video end, a UI end, a Weston open source library and a direct rendering manager;
the video end is used for applying for creating a first drawing surface of a video window to a Weston open source library through a standard desktop protocol and generating a video buffer area corresponding to the first drawing surface; wherein the standard desktop protocol comprises xdg-shell or wl-shell standard protocol;
the UI end is used for applying for creating a second drawing surface of a UI window from the Weston open source library through an ylvideo-shell-protocol internal protocol and generating a UI buffer area corresponding to the second drawing surface; wherein the ylvideo-shell-protocol internal protocol is a proprietary protocol established based on the Wayland standard;
the Weston open source library comprises a window management module, wherein the window management module is used for submitting the video buffer area and the UI buffer area to a hardware layer unit of a direct rendering manager and submitting a display request of the direct rendering manager according to the sequence of creating the first drawing surface and the second drawing surface after receiving the display requests of the video end and the UI end;
the direct rendering manager synthesizes the video buffer and the video buffer in a fixed hardware pipeline according to the display request to output a mixed video and UI.
Among them, Wayland is a set of communication protocols between display servers (Wayland composer) and clients, and Weston open source library is a reference implementation of Wayland composer.
From the internal architecture, the Weston open source library is mainly divided into window management (shell), synthesizer (synthesizer) and input management. It can be seen that, analogous to Android, the Weston open source library is functionally approximately equivalent to InputManagerService, WindowManagerService, and SurfaceFlinger. In general terms, the input management module of the Weston open source library accepts user input, and then on one hand, the shell performs corresponding window management operations (such as window stack changes, focus changes, etc.), and on the other hand, the input event is transmitted to the client which has registered the corresponding input event. After receiving the command, the client performs a corresponding action in the handler, such as adjusting the view and then redrawing. If redrawing occurs, after the new buffer is rendered, the client transmits the handle of the new buffer to the server, then the server generates a window list of z-order, then the composer is synthesized by the render, and finally the window list is output (such as to the framebuffer).
In the embodiment of the invention, the window management module is redeveloped based on the Weston open source library of the Wayland protocol: and modifying a Plane (hardware layer) distribution rule of the optimized Drm-background, so that both buffers (Buffer areas) after video decoding and buffers (UIs) drawn are directly bound to a Plane (hardware layer unit) of the Drm (direct rendering manager), and finally, mixed output is realized through a fixed hardware pipeline Pipe. The specific operation is as follows:
firstly, a video end applies for creating a surface of a video window to weston through a standard desktop protocol containing xdg-shell or wl-shell, and submits a decoded buffer to weston after decoding. Therefore, the third-party video App can be ensured to be accessed into the system according to the standard protocol.
Secondly, the UI end applies for the surface of the UI window to the weston through an internal protocol yvideo-shell-protocol established based on the Wayland standard, and submits the buffer drawn by the GPU to the weston.
And thirdly, after the web receives the display request of the video/UI, submitting the buffer corresponding to the surface to the Plane of the DRM according to the created surface sequence, and submitting the display request of the DRM.
Finally, the DRM synthesizes the UI and the buffer of the video in the fixed hardware pipeline Pipe (scaling may be performed before synthesis) according to the request, thereby completing the output.
In summary, compared with the prior art, in the drm direct display method and system for videos and UIs based on the Wayland protocol in the embodiment of the present invention, the Weston open source library based on the Wayland protocol redevelops the window management module: and modifying a Plane (hardware layer) distribution rule of the optimized Drm-background, so that both buffers (Buffer areas) after video decoding and buffers (UIs) drawn are directly bound to a Plane (hardware layer unit) of the Drm (direct rendering manager), and finally, mixed output is realized through a fixed hardware pipeline Pipe. Therefore, video/UI mixed output under the scene of double-screen 4K resolution is realized, the video can reach 60 frames, and the UI can reach 30 frames.
In one embodiment, before the video end applies for creating the first drawing surface of the video window to the Weston open source library through the standard desktop protocol in step S10, and generates the video buffer corresponding to the first drawing surface, the method further includes the following steps:
s50, the Weston open source library completes registration of the ylvideo-shell-protocol internal protocol and registration of the standard desktop protocol api callback function, and places the UI window created through the ylvideo-shell-protocol internal protocol and the video window created through the standard desktop protocol into corresponding window level objects; the window level objects comprise video layer window objects and UI layer window objects, the video layer window objects are used for storing video layer windows, and the UI layer window objects are used for storing UI layer windows.
Correspondingly, the window management module is further configured to execute the step S50, specifically: completing registration of the yvido-shell-protocol internal protocol and registration of the standard desktop protocol api callback function, and placing a UI window created through the yvido-shell-protocol internal protocol and a video window created through the standard desktop protocol into a corresponding window level object; the window level objects comprise video layer window objects and UI layer window objects, the video layer window objects are used for storing video layer windows, and the UI layer window objects are used for storing UI layer windows.
In the embodiment of the invention, the Weston open source library comprises a window management module: the windows management module ylvideo-shell mainly completes registration of a private protocol ylvideo-shell-protocol and registration of a standard desktop protocol api callback, and places windows created through the private protocol and the standard desktop protocol into corresponding window level weston _ layer objects.
Thus, the windows management module ylvideo-shell continues the windows hierarchy weston _ layer of weston to manage windows, defining two levels to store the video window list and the UI window list, respectively.
As shown in fig. 3, the weston _ layer of the video window is placed at the already defined position: WESTON _ LAYER _ POSITION _ BOTTOM _ UI, the WESTON _ LAYER of the UI window is placed in the already defined POSITION: WESTON _ LAYER _ POSITION _ NORMAL. Since weston organizes the corresponding window queues in the layer in order according to the weston _ layer _ position size, it can be guaranteed that the UI window can always be displayed on the video window.
In one embodiment, in step S50, the Weston open source library completes registration of the ylvideo-shell-protocol internal protocol and registration of the standard desktop protocol api callback function, and places the UI window created by the ylvideo-shell-protocol internal protocol and the video window created by the standard desktop protocol into a corresponding window hierarchy object, which specifically includes the following steps:
s51, creating a window management module object (yl _ desktop _ shell object) and a window hierarchy object (Weston _ layer object) of the Weston open source library;
s52, setting the hierarchy attribute of the video LAYER window object as WESTON _ LAYER _ POSITION _ BOTTOM _ UI with a first preset value, and setting the hierarchy attribute of the UI LAYER window object as WESTON _ LAYER _ POSITION _ NORMAL with a second preset value;
s53, creating a weston _ desktop object according to the structure body of the well-defined weston _ desktop _ api callback function, and creating a ylvideo-shell global resource object according to the structure body of the well-defined ylvideo-shell-protocol callback function;
s54, monitoring the standard desktop protocol and the yvido-shell-protocol internal protocol, creating a video window into the list of the video layer window object when receiving a request for creating the video window sent by the standard desktop protocol, and creating a UI window into the list of the UI layer window object when receiving a request for creating the UI window sent by the yvido-shell-protocol internal protocol.
Correspondingly, the window management module is further configured to execute the steps S51-S54, specifically:
creating a window management module object (yl _ desktop _ shell object) and a window hierarchy object (Weston _ layer object) of the Weston open source library;
setting the hierarchy attribute of the video LAYER window object as a WESTON _ LAYER _ POSITION _ BOTTOM _ UI with a first preset value, and setting the hierarchy attribute of the UI LAYER window object as a WESTON _ LAYER _ POSITION _ NORMAL with a second preset value;
creating a weston _ desktop object according to the structure body of the defined weston _ desktop _ api callback function, and creating a yyydeo-shell global resource object according to the structure body of the defined yydeo-shell-protocol callback function;
and monitoring the standard desktop protocol and the yvido-shell-protocol internal protocol, creating a video window into the list of the video layer window objects when receiving a video window creation request sent by the standard desktop protocol, and creating a UI window into the list of the UI layer window objects when receiving a UI window creation request sent by the yvido-shell-protocol internal protocol.
For better understanding of the embodiment, please refer to fig. 4, in the embodiment of the present invention, the initialization process of the yvideo-shell of the window management module is as follows:
firstly, a window management module object yl _ desktop _ shell object of a weston end, and window hierarchy weston _ layer objects, namely a video _ layer and a UI _ layer, are created, wherein the video _ layer is used for storing video layer windows, and the UI _ layer is used for storing UI layer windows.
Second, the level attribute POSITION of the video _ LAYER is set to store _ LAYER _ POSITION _ BOTTOM _ UI (value 0x30000000), and the level attribute POSITION of UI _ LAYER is set to store _ LAYER _ POSITION _ NORMAL (value 0x50000000), thereby ensuring that the UI window can always be displayed on top of the video window without affecting each other.
And then, creating a weston _ desktop object according to the structure body of the well-defined weston _ desktop _ api callback function, and creating a ylvideo-shell global resource object according to the structure body of the well-defined private protocol yl-video-shell processing function, so that the request received from the client can be executed into the corresponding processing function.
And finally, entering a default listening flow of weston to receive the message communication encapsulated by the xdg _ shell and yvido-shell protocols of the client. when the webston receives a window creation request sent by the xdg-shell protocol, the window is created into a list of video _ layer; upon receiving a create window request from the yvideo-shell proprietary protocol, the window is created into the list of ui _ layer.
In a certain embodiment, in step S30, after receiving the display requests of the video end and the UI end, the Weston source library submits the video buffer and the UI buffer to a hardware layer unit of a direct rendering manager according to the order of creating the first drawing surface and the order of creating the second drawing surface, and submits the display requests of the direct rendering manager, including the following steps:
s31, the Weston open source library sequentially forms a new window queue by the video layer window object and the Weston _ view window object in the UI layer window object according to the sequence of creating the first drawing surface and the second drawing surface;
s32, traversing the window queue, and acquiring a full-screen opaque window with an effective buffer area or the last window with an effective buffer area as an output window corresponding to the main hardware layer;
s33, in the window queue, sequentially distributing windows to corresponding overlapped hardware layers from bottom to top from the output window corresponding to the main hardware layer;
s34, binding the video buffer area of the video window and the UI buffer area of the UI window to the frame buffer area of the corresponding hardware layer unit of the direct rendering manager in sequence, and submitting the display request of the direct rendering manager.
Correspondingly, the window management module is further configured to execute the steps S31-S34, specifically:
sequentially forming a new window queue by the webton _ view window objects in the video layer window object and the UI layer window object according to the sequence of creating the first drawing surface and the second drawing surface;
traversing the window queue to obtain a window which has an effective buffer area and is not transparent in a full screen or a last window which has an effective buffer area, and taking the window as an output window corresponding to the main hardware layer;
in the window queue, from the output window corresponding to the main hardware layer, windows are sequentially distributed to the corresponding overlapped hardware layers from bottom to top;
and binding the video buffer area of the video window and the UI buffer area of the UI window to a frame buffer area of a corresponding hardware layer unit of the direct rendering manager in sequence, and submitting a display request of the direct rendering manager.
For better understanding of the embodiment, please refer to fig. 5, in the embodiment of the present invention, a hardware layer Plane allocation procedure for directly starting a drm-backup at a back end of a rendering manager is specifically as follows:
firstly, according to the sequence of layer queues in the weston, sequentially forming a new window queue view _ list by the weston _ view window objects in the ui _ layer object and the video _ layer object.
Secondly, traversing the window queue from top to bottom, filtering out windows of Direct display buffers (Direct Memory Access) which are not submitted with Direct Memory Access (DMA), and finding out non-transparent windows which can be fully paved with a full screen as Primary Plane (main hardware layer) output windows. And if the full-screen non-transparent window does not exist, default to the last window with the effective Buffer as a Primary Plane output window.
And thirdly, in the window queue, from the Primary Plane window, distributing windows to the corresponding Overlay Plane from bottom to top in sequence.
And finally, binding the buffers of the windows to the FrameBuffers of the corresponding Plane in sequence, and submitting a drm output request.
In summary, compared with the prior art, in the Linux system, the yvido-shell, which is a window management module of the video/UI system, is implemented based on the Weston framework, and the complete drm direct display system of the video/UI is implemented by modifying and optimizing the Plane hardware layer allocation rule and the display logic of the drm-backup (direct rendering manager back-end startup), so that the hardware resources are fully used and reasonably allocated, and the display scheme with optimized overall performance is achieved, specifically:
firstly, integrating yvideo-shell into windows of wl-shell and xdg-shell protocols through libweston-desktop to provide the windows for video use, facilitating access of third-party video applications, and defining a video layer to manage the video windows.
Secondly, the yvido-shell defines a protocol-yvido-shell-protocol used inside the UI based on a standard Wayland protocol framework so as to realize the functions of managing non-video windows such as the UI, the white board and the annotations, setting the resolution, sleeping and the like, and defining a UI layer to manage the UI window.
And thirdly, the yvideo-shell performs z-order management on the video layer and the UI layer, namely the UI layer is above the video layer, so as to ensure that the window in the UI layer is always displayed above the window in the video layer.
And finally, modifying the Plane distribution rule and the display logic of the optimized drm-backup to ensure that sufficient Plane is distributed to each window at any time, outputting the main video to the main Plane in the most efficient format by default, directly outputting the 4K main video after decoding, amplifying the 4K UI in the Plane by 1080P Buffer, and reducing the performance influence of the UI on the video.
Referring to fig. 6, an embodiment of the invention provides a computer terminal device, which includes one or more processors and a memory. The memory is coupled to the processor for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement a drm direct display method for video and UI based on Wayland protocol as in any of the above embodiments.
The processor is used for controlling the overall operation of the computer terminal equipment so as to complete all or part of the steps of the drm direct display method of the video and the UI based on the Wayland protocol. The memory is used to store various types of data to support the operation at the computer terminal device, which data may include, for example, instructions for any application or method operating on the computer terminal device, as well as application-related data. The Memory may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk, or optical disk.
In an exemplary embodiment, the computer terminal Device may be implemented by one or more Application Specific 1 integrated circuits (AS 1C), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and is configured to perform the above-mentioned method for direct display of video and UI data based on the wash protocol, and achieve the technical effects consistent with the above-mentioned method.
In another exemplary embodiment, there is also provided a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the method for drm direct display of videos and UIs based on the Wayland protocol in any of the above embodiments. For example, the computer readable storage medium may be the above-mentioned memory comprising program instructions executable by the processor of the computer terminal device to perform the above-mentioned method for drm direct display of videos and UIs based on the Wayland protocol, and to achieve the technical effects consistent with the above-mentioned method.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention.
Claims (9)
1. A drm direct display method of video and UI based on Wayland protocol, comprising:
a video end applies for creating a first drawing surface of a video window to a Weston open source library through a standard desktop protocol, and generates a video buffer zone corresponding to the first drawing surface; wherein the standard desktop protocol comprises xdg-shell or wl-shell standard protocol;
the UI end applies for creating a second drawing surface of a UI window from the Weston open source library through an yvideo-shell-protocol internal protocol and generates a UI buffer area corresponding to the second drawing surface; wherein the ylvideo-shell-protocol internal protocol is a proprietary protocol established based on the Wayland standard;
after the Weston open source library receives the display requests of the video end and the UI end, the Weston open source library submits the video buffer area and the UI buffer area to a hardware layer unit of a direct rendering manager according to the sequence of creating the first drawing surface and the second drawing surface, and submits the display requests of the direct rendering manager;
the direct rendering manager synthesizes the video buffer and the video buffer in a fixed hardware pipeline according to the display request to output a mixed video and UI.
2. The method for drm direct display of video and UI according to claim 1, wherein before the step of applying for creating the first drawing surface of the video window from Weston's open source library through standard desktop protocol at the video end and generating the video buffer corresponding to the first drawing surface, further comprising:
the Weston open source library completes registration of the ylvideo-shell-protocol internal protocol and registration of the standard desktop protocol api callback function, and places a UI window created through the ylvideo-shell-protocol internal protocol and a video window created through the standard desktop protocol into corresponding window level objects; the window level objects comprise video layer window objects and UI layer window objects, the video layer window objects are used for storing video layer windows, and the UI layer window objects are used for storing UI layer windows.
3. The method for drm direct display of video and UI based on Wayland protocol as claimed in claim 2, wherein the Weston open source library completes the registration of the ylvideo-shell-protocol internal protocol and the registration of the standard desktop protocol api callback function, and the step of placing the UI window created by the ylvideo-shell-protocol internal protocol and the video window created by the standard desktop protocol into the corresponding window level object specifically comprises:
creating a window management module object and a window hierarchy object of a Weston open source library;
setting the hierarchy attribute of the video LAYER window object as a WESTON _ LAYER _ POSITION _ BOTTOM _ UI with a first preset value, and setting the hierarchy attribute of the UI LAYER window object as a WESTON _ LAYER _ POSITION _ NORMAL with a second preset value;
creating a weston _ desktop object according to the structure body of the defined weston _ desktop _ api callback function, and creating a yyydeo-shell global resource object according to the structure body of the defined yydeo-shell-protocol callback function;
and monitoring the standard desktop protocol and the yvido-shell-protocol internal protocol, creating a video window into the list of the video layer window objects when receiving a video window creation request sent by the standard desktop protocol, and creating a UI window into the list of the UI layer window objects when receiving a UI window creation request sent by the yvido-shell-protocol internal protocol.
4. The method for drm direct display of video and UI according to Wayland protocol as claimed in claim 3, wherein the Weston open source library submits the video buffer and the UI buffer to the hardware layer unit of the direct rendering manager according to the order of creating the first drawing surface and the second drawing surface after receiving the display request of the video end and the UI end, and submits the display request of the direct rendering manager, comprising:
the Weston open source library sequentially forms a new window queue with the Weston _ view window objects in the video layer window object and the UI layer window object according to the sequence of creating the first drawing surface and the second drawing surface;
traversing the window queue to obtain a window which has an effective buffer area and is not transparent in a full screen or a last window which has an effective buffer area, and taking the window as an output window corresponding to the main hardware layer;
in the window queue, from the output window corresponding to the main hardware layer, windows are sequentially distributed to the corresponding overlapped hardware layers from bottom to top;
and binding the video buffer area of the video window and the UI buffer area of the UI window to a frame buffer area of a corresponding hardware layer unit of the direct rendering manager in sequence, and submitting a display request of the direct rendering manager.
5. A drm direct display system for video and UI based on Wayland protocol, comprising: the system comprises a video end, a UI end, a Weston open source library and a direct rendering manager;
the video end is used for applying for creating a first drawing surface of a video window to a Weston open source library through a standard desktop protocol and generating a video buffer area corresponding to the first drawing surface; wherein the standard desktop protocol comprises xdg-shell or wl-shell standard protocol;
the UI end is used for applying for creating a second drawing surface of a UI window from the Weston open source library through an ylvideo-shell-protocol internal protocol and generating a UI buffer area corresponding to the second drawing surface; wherein the ylvideo-shell-protocol internal protocol is a proprietary protocol established based on the Wayland standard;
the Weston open source library comprises a window management module, wherein the window management module is used for submitting the video buffer area and the UI buffer area to a hardware layer unit of a direct rendering manager and submitting a display request of the direct rendering manager according to the sequence of creating the first drawing surface and the second drawing surface after receiving the display requests of the video end and the UI end;
the direct rendering manager synthesizes the video buffer and the video buffer in a fixed hardware pipeline according to the display request to output a mixed video and UI.
6. The Wayland protocol based video and UI drm direct display system of claim 5, wherein the window management module is further configured to:
completing registration of the yvido-shell-protocol internal protocol and registration of the standard desktop protocol api callback function, and placing a UI window created through the yvido-shell-protocol internal protocol and a video window created through the standard desktop protocol into a corresponding window level object; the window level objects comprise video layer window objects and UI layer window objects, the video layer window objects are used for storing video layer windows, and the UI layer window objects are used for storing UI layer windows.
7. The Wayland protocol based video and UI drm direct display system of claim 6, wherein the window management module is further configured to:
creating a window management module object and a window hierarchy object of a Weston open source library;
setting the hierarchy attribute of the video LAYER window object as a WESTON _ LAYER _ POSITION _ BOTTOM _ UI with a first preset value, and setting the hierarchy attribute of the UI LAYER window object as a WESTON _ LAYER _ POSITION _ NORMAL with a second preset value;
creating a weston _ desktop object according to the structure body of the defined weston _ desktop _ api callback function, and creating a yyydeo-shell global resource object according to the structure body of the defined yydeo-shell-protocol callback function;
and monitoring the standard desktop protocol and the yvido-shell-protocol internal protocol, creating a video window into the list of the video layer window objects when receiving a video window creation request sent by the standard desktop protocol, and creating a UI window into the list of the UI layer window objects when receiving a UI window creation request sent by the yvido-shell-protocol internal protocol.
8. The Wayland protocol based video and UI drm direct display system of claim 7, wherein the window management module is further configured to:
sequentially forming a new window queue by the webton _ view window objects in the video layer window object and the UI layer window object according to the sequence of creating the first drawing surface and the second drawing surface;
traversing the window queue to obtain a window which has an effective buffer area and is not transparent in a full screen or a last window which has an effective buffer area, and taking the window as an output window corresponding to the main hardware layer;
in the window queue, from the output window corresponding to the main hardware layer, windows are sequentially distributed to the corresponding overlapped hardware layers from bottom to top;
and binding the video buffer area of the video window and the UI buffer area of the UI window to a frame buffer area of a corresponding hardware layer unit of the direct rendering manager in sequence, and submitting a display request of the direct rendering manager.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the drm direct display method of videos and UIs based on the Wayland protocol according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110167640.1A CN113050899B (en) | 2021-02-07 | 2021-02-07 | Method and system for directly displaying video and UI drm based on Wayland protocol |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110167640.1A CN113050899B (en) | 2021-02-07 | 2021-02-07 | Method and system for directly displaying video and UI drm based on Wayland protocol |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113050899A true CN113050899A (en) | 2021-06-29 |
| CN113050899B CN113050899B (en) | 2022-09-27 |
Family
ID=76508740
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110167640.1A Active CN113050899B (en) | 2021-02-07 | 2021-02-07 | Method and system for directly displaying video and UI drm based on Wayland protocol |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113050899B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672321A (en) * | 2021-07-26 | 2021-11-19 | 中标软件有限公司 | Implementation method of lightweight wayland synthesizer supporting client window coordinates |
| CN114896002A (en) * | 2022-04-13 | 2022-08-12 | 北京智象信息技术有限公司 | Browser application window level switching method and device based on Wayland protocol |
| CN117278796A (en) * | 2023-11-23 | 2023-12-22 | 深圳市泛联信息科技有限公司 | Multi-area image data display method, device, playing equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150161754A1 (en) * | 2013-12-10 | 2015-06-11 | Joel Solomon Isaacson | System and method for remote graphics using non-pixel rendering interfaces |
| CN105701037A (en) * | 2015-12-30 | 2016-06-22 | 惠州市德赛西威汽车电子股份有限公司 | System and method for simultaneously accessing to display controller by weston and DRM in vehicle-mounted entertainment system |
| CN107168715A (en) * | 2017-05-25 | 2017-09-15 | 深圳市视维科技股份有限公司 | A kind of intelligent terminal menu subject GUI engine implementation method |
| CN107291319A (en) * | 2016-03-31 | 2017-10-24 | 阿里巴巴集团控股有限公司 | A kind of window display method, information interacting method and system |
| CN108093292A (en) * | 2016-11-21 | 2018-05-29 | 阿里巴巴集团控股有限公司 | For managing the method, apparatus and system of caching |
| CN110166792A (en) * | 2018-02-14 | 2019-08-23 | 阿里巴巴集团控股有限公司 | Image display method, device, system and storage medium |
-
2021
- 2021-02-07 CN CN202110167640.1A patent/CN113050899B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150161754A1 (en) * | 2013-12-10 | 2015-06-11 | Joel Solomon Isaacson | System and method for remote graphics using non-pixel rendering interfaces |
| CN105701037A (en) * | 2015-12-30 | 2016-06-22 | 惠州市德赛西威汽车电子股份有限公司 | System and method for simultaneously accessing to display controller by weston and DRM in vehicle-mounted entertainment system |
| CN107291319A (en) * | 2016-03-31 | 2017-10-24 | 阿里巴巴集团控股有限公司 | A kind of window display method, information interacting method and system |
| CN108093292A (en) * | 2016-11-21 | 2018-05-29 | 阿里巴巴集团控股有限公司 | For managing the method, apparatus and system of caching |
| CN107168715A (en) * | 2017-05-25 | 2017-09-15 | 深圳市视维科技股份有限公司 | A kind of intelligent terminal menu subject GUI engine implementation method |
| CN110166792A (en) * | 2018-02-14 | 2019-08-23 | 阿里巴巴集团控股有限公司 | Image display method, device, system and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| JACKIEGEMINI: "Wayland与Weston简介", 《HTTPS://BLOG.CSDN.NET/FYH2003/ARTICLE/DETAILS/49253713》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672321A (en) * | 2021-07-26 | 2021-11-19 | 中标软件有限公司 | Implementation method of lightweight wayland synthesizer supporting client window coordinates |
| CN113672321B (en) * | 2021-07-26 | 2023-08-22 | 中标软件有限公司 | Implementation method of lightweight wayside synthesizer supporting client window coordinates |
| CN114896002A (en) * | 2022-04-13 | 2022-08-12 | 北京智象信息技术有限公司 | Browser application window level switching method and device based on Wayland protocol |
| CN117278796A (en) * | 2023-11-23 | 2023-12-22 | 深圳市泛联信息科技有限公司 | Multi-area image data display method, device, playing equipment and storage medium |
| CN117278796B (en) * | 2023-11-23 | 2024-04-19 | 深圳市泛联信息科技有限公司 | Multi-area image data display method, device, playing equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113050899B (en) | 2022-09-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113050899B (en) | Method and system for directly displaying video and UI drm based on Wayland protocol | |
| US10031712B2 (en) | System and method for display mirroring | |
| US10356467B2 (en) | Virtual user interface including playback control provided over computer network for client device playing media from another source | |
| EP2962191B1 (en) | System and method for virtual displays | |
| US7911409B1 (en) | Independent views generated for multiple display devices by a software application | |
| US20110292060A1 (en) | Frame buffer sizing to optimize the performance of on screen graphics in a digital electronic device | |
| RU2463661C2 (en) | Animated desktop | |
| KR101232428B1 (en) | Display data management techniques | |
| JP2019512749A (en) | Application program processing method and terminal device | |
| TW201539294A (en) | Cross-platform rendering engine | |
| JP4742051B2 (en) | Spatial and temporal motion blur effect generation method | |
| KR20150016527A (en) | Virtual surface allocation | |
| CN116136784A (en) | Data processing method, device, storage medium and program product | |
| US20030011534A1 (en) | Display privacy for enhanced presentations with real-time updates | |
| CN113655975A (en) | Image display method, image display device, electronic apparatus, and medium | |
| JP5229727B2 (en) | Multi-image display system, image processing method and program | |
| CN114020375A (en) | Display method and device | |
| US6888550B2 (en) | Selecting between double buffered stereo and single buffered stereo in a windowing system | |
| CN111813404B (en) | Application method, medium and client based on mixed graphic display | |
| TW202245478A (en) | Method and apparatus for screen displaying | |
| CN117156186A (en) | Fusion display system and image processing method | |
| CN114327722A (en) | Mobile terminal performance optimization method, device, equipment and readable storage medium | |
| Martin et al. | Roadmap to a GL-based composited desktop for Linux | |
| TW201734946A (en) | Apparatus and method for combining video frame and graphics frame |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |