CN112614202B - GUI rendering display method, terminal, server, electronic equipment and storage medium - Google Patents

GUI rendering display method, terminal, server, electronic equipment and storage medium Download PDF

Info

Publication number
CN112614202B
CN112614202B CN202011551948.8A CN202011551948A CN112614202B CN 112614202 B CN112614202 B CN 112614202B CN 202011551948 A CN202011551948 A CN 202011551948A CN 112614202 B CN112614202 B CN 112614202B
Authority
CN
China
Prior art keywords
frame image
rendered
rendering
rendering instruction
terminal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011551948.8A
Other languages
Chinese (zh)
Other versions
CN112614202A (en
Inventor
奚智
姜哲
邹仕洪
张广伟
黄浩东
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yuanxin Technology
Original Assignee
Yuanxin Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yuanxin Technology filed Critical Yuanxin Technology
Priority to CN202011551948.8A priority Critical patent/CN112614202B/en
Publication of CN112614202A publication Critical patent/CN112614202A/en
Application granted granted Critical
Publication of CN112614202B publication Critical patent/CN112614202B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/50—Controlling the output signals based on the game progress
    • A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/60—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/40—Filling a planar surface by adding surface attributes, e.g. colour or texture
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The embodiment of the application provides a method, a terminal, a server, electronic equipment and a storage medium for rendering and displaying a GUI, and relates to the technical field of software. The method comprises the following steps: the method comprises the steps of receiving interactive operation information sent by a terminal, updating a rendering instruction of each frame of frame image and a timestamp corresponding to a screen updating period according to the interactive operation information, rendering the frame images to be rendered by a server and the terminal respectively and independently according to the rendering instruction to obtain rendered frame images, sending the rendered frame images to the terminal by the server, receiving the rendered frame images sent by the server by the terminal, and displaying a system GUI by combining the rendered frame images of the terminal. According to the method and the device for processing the frame images, the terminal and the server can be isolated from each other, the respective capabilities are fully exerted, the time sequence errors are avoided according to the time stamp information for synchronization, the image processing rate can be improved, the hardware resources of the terminal and the server can be fully utilized, the overall efficiency is improved, and the user experience is improved.

Description

GUI rendering display method, terminal, server, electronic equipment and storage medium
Technical Field
The application relates to the technical field of software, in particular to a method, a terminal, a server, electronic equipment and a storage medium for rendering and displaying a GUI.
Background
With the development of cloud technology, virtualization and internet technology, virtualized desktops represented by cloud desktops are widely used. Cloud desktop (virtual desktop) infrastructure (VDI, virtual Desktop Infrastructure) typically virtualizes a user's desktop by running an operating system through servers in a data center. Users connect to virtual desktops through a client computing protocol from a client device (either a client or a home PC), and users access their desktops as they would a traditional locally installed desktop.
With the development of hardware technology and the wide application of VR, AR cloud rendering and cloud games, the hardware performance of a user terminal is generally improved, how a terminal participates in the collaborative rendering of a terminal and a cloud GUI (computer operation user interface displayed in a graphic manner), the hardware performance of the cloud and the terminal is fully exerted, and the collaborative rendering display capability of the high-performance terminal and the cloud GUI is provided, so that the problem to be solved is solved.
Currently, methods mainly adopted to provide high-performance end-to-end, cloud GUI collaborative rendering display are as follows: rendering by the cloud, directly mapping the result to the virtual machine, and then directly displaying by the client; the cloud end is required to adopt a high-performance processing device, the cloud load is easy to aggravate, the processing speed is influenced, the other mode adopts the end and cloud to participate in the whole GUI rendering process in a distributed mode, and then the client end is used for displaying, and the mode mainly enables the end and the cloud to render and process the same image together, so that the image processing speed is improved, the end and the cloud are easy to influence each other when in interaction, the hardware capacity of the end and the cloud cannot be fully exerted, and the user experience is also influenced.
Disclosure of Invention
Embodiments of the present invention provide a GUI rendering display method, terminal, server, electronic device, and storage medium that overcome or at least partially solve the above-described problems.
In a first aspect, a method for rendering a display of a GUI is provided, applied to a server, the method comprising:
receiving interactive operation information sent by a terminal;
updating a rendering instruction and a time stamp corresponding to each frame of frame image to be rendered according to the interactive operation information, and forming a rendering instruction sequence; the time stamp is used for representing the initiation time of the screen refresh period corresponding to the frame image;
the method comprises the steps of obtaining graphic processing capacity data of a terminal and a server for processing frame images, and determining a first rendering instruction and a second rendering instruction from a rendering instruction sequence, wherein the first rendering instruction is a rendering instruction executed by the server, and the second rendering instruction is a rendering instruction executed by the terminal;
the second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction and the timestamp corresponding to the second rendering instruction are sent to the terminal, so that the terminal renders the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction, and a rendered second frame image is obtained;
And rendering the frame image corresponding to the first rendering instruction according to the first rendering instruction, obtaining a rendered first frame image, and sending the rendered first frame image and the corresponding timestamp to the terminal, so that the terminal determines a GUI rendering result to be displayed according to the rendered first frame image and the second frame image and the timestamp corresponding to the rendered first frame image and the rendered second frame image.
Further, updating the rendering instruction and the time stamp corresponding to each frame of frame image to be rendered according to the interactive operation information, including:
updating and processing the frame image to be rendered and rendering nodes in the frame image to be rendered according to the interactive operation information sent by the terminal;
the method comprises the steps of determining a rendering instruction corresponding to a frame image to be rendered according to the frame image to be rendered and a rendering node in the frame image to be rendered, and taking the starting time of a screen update period corresponding to the frame image to be rendered as a time stamp corresponding to the frame image to be rendered.
Further, sending the rendered first frame image and the corresponding timestamp to the terminal, and then further including:
and identifying the rendered first frame image according to the timestamp corresponding to the rendered first frame image, and sending the identified first frame image to the terminal.
In a second aspect, a method for rendering and displaying a GUI is provided, and applied to a terminal, the method includes:
the interactive operation information is sent to a server, so that the server updates a rendering instruction and a time stamp corresponding to each frame of frame image to be rendered and a first frame of image rendered by the server according to the interactive operation information, and sends a second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction and the time stamp corresponding to the second rendering instruction to a terminal;
receiving a second rendering instruction, a frame image to be rendered corresponding to the second rendering instruction and a time stamp corresponding to the second rendering instruction, which are sent by a server, and rendering the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction to obtain a rendered second frame image;
and receiving the rendered first frame image and the corresponding time stamp sent by the server, and determining a GUI rendering result to be displayed by combining the rendered first frame image and the second frame image and the time stamp corresponding to the rendered first frame image and the second frame image.
Further, the frame image to be rendered corresponding to the second rendering instruction is rendered according to the second rendering instruction, so as to obtain a rendered second frame image, and then the method further comprises the following steps:
And identifying the rendered second frame image according to the timestamp corresponding to the rendered second frame image, and obtaining the identified second frame image.
Further, determining a GUI rendering result to be displayed in combination with the rendered first frame image and the second frame image and the time stamps corresponding to the rendered first frame image and the rendered second frame image, includes:
determining frame images to be displayed in the first frame image and the second frame image;
acquiring the current time, and determining the time interval of the frame images to be displayed according to the time stamp corresponding to the frame images to be displayed and the current time;
if the time interval of the frame images to be displayed is smaller than or equal to a preset threshold value, reserving the frame images to be displayed, and taking the frame images to be displayed as a GUI rendering result to be displayed;
and if the time interval of the frame images to be displayed is greater than a preset threshold value, the frame images to be displayed are cleared.
In a third aspect, a server is provided, comprising:
the first receiving module is used for receiving the interactive operation information sent by the terminal;
the first processing module is used for updating a rendering instruction and a time stamp corresponding to each frame of frame image to be rendered according to the interactive operation information, and forming a rendering instruction sequence; the time stamp is used for representing the initiation time of a screen update period corresponding to a rendering instruction of the frame image;
The second processing module is used for acquiring graphic processing capacity data of the frame images processed by the terminal and the server so as to determine a first rendering instruction and a second rendering instruction from the rendering instruction sequence, wherein the first rendering instruction is a rendering instruction executed by the server, and the second rendering instruction is a rendering instruction pointed by the terminal;
the second sending module is used for sending a second rendering instruction, a frame image to be rendered corresponding to the second rendering instruction and a timestamp corresponding to the second rendering instruction to the terminal, so that the terminal renders the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction, and a rendered second frame image is obtained;
the third processing module is used for rendering the frame image corresponding to the first rendering instruction according to the first rendering instruction, obtaining a rendered first frame image, and sending the rendered first frame image and the corresponding timestamp to the terminal, so that the terminal determines a GUI rendering result to be displayed according to the rendered first frame image and the second frame image and the timestamp corresponding to the rendered first frame image and the rendered second frame image.
In a fourth aspect, there is provided a terminal comprising:
the first sending module is used for sending the interactive operation information to the server, so that the server updates a rendering instruction and a time stamp corresponding to each frame of frame image to be rendered and the first frame image rendered by the server according to the interactive operation information, and sends a second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction and the time stamp corresponding to the second rendering instruction to the terminal;
The second receiving module is used for receiving a second rendering instruction, a frame image to be rendered corresponding to the second rendering instruction and a time stamp corresponding to the second rendering instruction sent by the server, and rendering the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction to obtain a rendered second frame image;
the third receiving module is used for receiving the rendered first frame image and the corresponding time stamp sent by the server, and determining a GUI rendering result to be displayed by combining the rendered first frame image and the second frame image and the time stamp corresponding to the rendered first frame image and the rendered second frame image.
In a fifth aspect, there is provided an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method as provided in the first or second aspect when the program is executed.
In a sixth aspect, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method as provided in the first or second aspect.
According to the method, the terminal, the server, the electronic equipment and the storage medium for rendering and displaying the GUI, the terminal receives the interactive operation information sent by the terminal, the rendering instruction of each frame of frame image and the time stamp corresponding to the screen updating period are updated according to the interactive operation information, the server and the terminal render the frame image to be rendered according to the rendering instruction respectively and independently to obtain the rendered frame image, the server sends the rendered frame image to the terminal, and the terminal receives the rendered frame image sent by the server and is used for displaying the GUI of the system together in combination with the rendered frame image of the terminal. The scheme isolates the capability of processing the frame images of the terminal and the server, does not affect each other, gives full play to respective capability, synchronizes according to the timestamp information, avoids time sequence errors, can improve the image processing rate, can fully utilize hardware resources of the terminal and the server, improves the overall efficiency, and improves the user experience.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that are required to be used in the description of the embodiments of the present application will be briefly described below.
Fig. 1 is a schematic diagram of an application environment of a GUI rendering display method according to an embodiment of the present application;
FIG. 2 is a flowchart of a method for rendering a display of a GUI according to an embodiment of the present application;
fig. 3 is a schematic diagram of frame image rendering according to an embodiment of the present application;
fig. 4 is a schematic diagram of frame image combination according to an embodiment of the present application;
fig. 5 is an interaction schematic diagram of GUI rendering display performed by a terminal and a server according to an embodiment of the present application;
FIG. 6 is a flow chart of another method for rendering a display of a GUI according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of a terminal according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are exemplary only for the purpose of illustrating the present application and are not to be construed as limiting the invention.
As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein includes all or any element and all combination of one or more of the associated listed items.
The application provides a GUI rendering display method, a device, electronic equipment and a storage medium, which aim to solve the technical problems in the prior art.
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
Before describing the various embodiments shown in this application, several concepts related to this application will be described first:
1) GUI (Graphical User Interface, graphical user interface, also known as graphical user interface) refers to a computer-operated user interface that is displayed in a graphical manner, an interface display format in which a person communicates with a computer, allowing a user to manipulate icons or menu options on a screen using an input device such as a mouse to select commands, invoke files, launch programs, or perform some other daily task.
2) The GPU (Graphics Processing Unit, graphics processor), also known as a display core, a vision processor, a display chip, is a microprocessor that performs image and graphics related operations on personal computers, workstations, game consoles, and some mobile devices (e.g., tablet computers, smartphones, etc.).
3) The CPU (Central Processing Unit ) is used as the operation and control core of the computer system and is the final execution unit for information processing and program running.
First, the embodiment of the application can be applied to application scenes displayed in a GUI rendering, for example: cloud desktops, VR, AR cloud rendering, cloud gaming, etc., typically run an operating system through a server of a data center, virtualize a user's desktop, connect with virtual desktops through a client computing protocol from a client device (client or home PC), and access their desktops as if they were a traditional locally installed desktop.
It should be understood that the method for rendering and displaying a GUI provided in the present application may be applied to any system including a computer with a GUI graphics system and a server, and referring to fig. 1, fig. 1 is a schematic diagram of an application environment of a method for rendering and displaying a GUI provided in an embodiment of the present application, and an example is a method that a terminal and a server jointly execute GUI rendering and displaying. The method for rendering and displaying the GUI provided by the embodiment of the application comprises a terminal 10, a server 11 and the like; the terminal 10 may include a mobile phone, a tablet computer, a notebook computer, a personal computer (PC, personal Computer), or the like, and the server 11 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server that provides cloud computing services. The terminal 10 and the server 11 are connected via a network, for example, a wired or wireless network connection. According to the scheme, a time stamp and a frame buffer pool technology are adopted, a frame buffer technology is respectively adopted at two sides of a terminal 10 and a server 11, the terminal 10 and the server 11 cooperatively render graphic drawing instructions, complete GUI drawing work is completed, frame images are generated and stored in local frame buffer pools of the terminal 10 and the server 11, and a time stamp information mark is added, so that the server 11 sends the latest rendered frame images in the server 11 to the terminal 10 according to the time stamp; the terminal 10 receives the latest frame image transmitted from the server 11, compares the latest frame image with the local latest frame image according to the time stamp, and selects and displays the latest frame image on the terminal 10. The method and the device isolate the capability of processing the frame images of the terminal and the server, are not influenced by each other, give full play to respective capability, synchronize according to the timestamp information, avoid errors, improve the image processing rate, fully utilize the hardware resources of the terminal and the server, improve the overall efficiency and improve the user experience.
The following describes the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
In order to solve the above problems, embodiments of the present application provide a method for rendering a display of a GUI. The method for rendering and displaying the GUI provided in the embodiment of the present application will be described in detail with reference to the accompanying drawings by using specific embodiments and other application scenarios, and fig. 2 is a schematic flow chart of a method for rendering and displaying the GUI provided in the embodiment, as shown in fig. 2, where the method includes:
s202, receiving the interactive operation information sent by the terminal.
Specifically, the interactive operation information sent by the terminal is received, and the interactive operation information comprises movement of a mouse, input of a keyboard, execution logic of an application program, menu instructions, switching of windows and the like.
The server of the embodiment of the application receives the traffic operation information sent by the terminal and lays a foundation for realizing the subsequent frame image rendering and selecting the frame image meeting the requirements.
And S203, updating a rendering instruction and a time stamp corresponding to each frame of frame image to be rendered according to the interactive operation information, and forming a rendering instruction sequence.
After receiving the interactive operation information sent by the terminal, the embodiment of the application needs to determine a rendering instruction of each frame image according to the interactive operation information, for example: when the end user performs interactive operations (such as moving a mouse, inputting a keyboard, etc.), the operating system of the server or related applications of the server determine the rendering instruction of each frame of image through the function by using the function similar to the response event, wherein the rendering instruction is generated by each function in the system according to the program logic in response to the external input. The method comprises the steps that a terminal does not input, a user does not perform interactive operation (does not perform operations such as a keyboard and a mouse) on the terminal, a server also needs to update each frame of frame image to be rendered, update the rendering instruction and the time stamp corresponding to each frame of frame image to be rendered, render the frame image corresponding to the first rendering instruction according to the first rendering instruction, and obtain the rendered first frame image.
The server of the embodiment of the application lays a foundation for realizing the subsequent frame image rendering and selecting the frame image meeting the requirements by updating the rendering instruction and the time stamp corresponding to each frame image to be rendered.
S204, acquiring graphic processing capability data of the frame image processed by the terminal and the server to determine a first rendering instruction and a second rendering instruction from the rendering instruction sequence, wherein the first rendering instruction is a rendering instruction executed by the server, and the second rendering instruction is a rendering instruction executed by the terminal.
According to the method and the device, graphics processing capability data of frame images processed by the terminal and the server are required to be acquired, rendering tasks distributed to the terminal and the server are determined according to data analysis, the first rendering instruction is the rendering instruction executed by the server, the second rendering instruction is the rendering instruction executed by the terminal, it is understood that the server is required to acquire the capability of the terminal for processing the frame images, such as the number and the model of the owned GPUs and performance evaluation obtained by scoring by test software, the rendering tasks are distributed according to the performance correspondence between the terminal and the server, namely, a part of frame images rendered by the server in a frame image sequence are distributed correspondingly, such as 10 processing tasks are distributed correspondingly, the hardware capability of the server is better, 7 tasks can be processed, and the rest 3 tasks are distributed to the terminal for processing, so that the processing speed can be improved, and the hardware resources of the terminal and the server can be fully utilized.
According to the method and the device for processing the frame images, the performance data of the server and the terminal are obtained, the frame images to be rendered are distributed according to the performance data, the terminal and the server can independently process the frame images, the frame image processing speed is improved, and the hardware resources of the terminal and the server can be fully utilized.
S205, sending the second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction and the timestamp corresponding to the second rendering instruction to the terminal, so that the terminal renders the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction, and the rendered second frame image is obtained.
In the embodiment of the invention, the server processes the second part task according to the rendering task distributed by the image processing performance data of the terminal and the server, and sends the rendering task of the terminal to the terminal, namely, the second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction and the corresponding timestamp, so that the terminal can render the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction, obtain the rendered second frame image, and ensure that the required frame image can be rendered.
According to the method and the device for rendering the frame images, the second rendering instruction, the frame images to be rendered corresponding to the second rendering instruction and the corresponding time stamp are sent to the terminal, so that the terminal can render the frame images, hardware resources of the terminal can be fully utilized, and overall efficiency is improved.
S206, rendering the frame image corresponding to the first rendering instruction according to the first rendering instruction, obtaining a rendered first frame image, and sending the rendered first frame image and the corresponding timestamp to the terminal, so that the terminal determines a GUI rendering result to be displayed according to the rendered first frame image and the second frame image and the timestamp corresponding to the rendered first frame image and the rendered second frame image.
In the embodiment of the application, a first rendering task, namely a first rendering instruction, a frame image to be rendered corresponding to the first rendering instruction and a timestamp corresponding to the frame image to be rendered, is processed by a server according to rendering tasks distributed by the image processing performance data of the terminal and the server, the server renders the frame image corresponding to the first rendering instruction according to the first rendering instruction to obtain a rendered first frame image, then the rendered first frame image and a rendered second frame image and the timestamp corresponding to the rendered first frame image and the rendered second frame image are determined, and a GUI rendering result to be displayed is determined.
According to the GUI rendering display method, through receiving the interactive operation information sent by the terminal, the rendering instruction of each frame of frame image and the timestamp corresponding to the screen updating period are updated according to the interactive operation information, the server and the terminal render the frame images according to the rendering instruction respectively and independently to obtain the rendered frame images, the server sends the rendered frame images to the terminal, the terminal receives the frame images sent by the server, and the terminal is combined with the rendered frame images, and selects the optimal frame images to display at the terminal according to the timestamp.
On the basis of the foregoing embodiments, as an optional embodiment, updating, according to the interaction information, a rendering instruction and a timestamp corresponding to each frame of frame image to be rendered, includes:
updating and processing the frame image to be rendered and rendering nodes in the frame image to be rendered according to the interactive operation information sent by the terminal;
the method comprises the steps of determining a rendering instruction corresponding to a frame image to be rendered according to the frame image to be rendered and a rendering node in the frame image to be rendered, and taking the initiating time of a screen updating period corresponding to the frame image to be rendered as a time stamp corresponding to the frame image to be rendered.
After receiving exchange operation information sent by a terminal, the server generates all frame images of complete actions according to the exchange operation information, determines a region where each frame image changes according to the change of the actions, determines a place to be rendered according to the texture, coordinates, transparency and color of the transformed region, generates a rendering instruction, and records the time when the transformed region of the frame image is converted into various rendering parameters, wherein the time is the initiation time of a response screen update period of the frame image to be rendered, and the time is used as a timestamp of the frame image, such as: when the end user performs interactive operation (moving a mouse, inputting a keyboard, etc.), the remote desktop is changed, and then an operating system of the server or related applications of the server determine a rendering instruction of each frame of image through a function by using a function similar to a response event, wherein the rendering instruction is generated by each function in the system according to the program logic in response to the external input. The remote desktop typically includes a plurality of display interfaces (windows); each window has a corresponding plurality of surfaces (window frame, title bar, menu bar, status bar, content view); each surface may in turn include rendering nodes (characters, primitives, textures); the rendering instructions are all corresponding to the rendering nodes, the area to be rendered is determined by detecting the area which changes each time according to the position and the attribute of the window (whether other windows are covered or not and whether the window has transparency or not), the area to be rendered is processed by the CPU, and the rendering instructions are obtained, wherein the rendering instructions comprise specific rendering parameters (texture, transparency, color and the like).
Fig. 3 is a schematic diagram of frame image rendering provided in this embodiment of the present application, as shown in fig. 3, in which part 1 of the drawing indicates an area where a frame image is not transformed, and part 2 indicates an area where a frame image is transformed, after receiving interactive operation information sent by a terminal, in this embodiment of the present application, a rendering instruction and a timestamp of each frame image need to be updated, first, according to an arrangement position of each frame image, a time sequence of each frame image may be determined, that is, where a frame image starts from, where a frame image ends, and then, each frame image is compared, a transformed area of each frame image may be determined, for example, a movement of a mouse is established at a lower left corner of each frame image, a position coordinate of the mouse in each frame image may be found to be changed, according to a change in a coordinate position, the transformed area in each frame image may be determined, and the transformed area of each frame image may be converted into parameters of each frame image by a CPU, including a color, a texture, a transparency, and a frame image may be rendered according to the transformed parameters of the transformed position of the mouse. In order to save resources and improve the rendering rate, only the transformed areas in each frame of frame image are converted into rendering parameters, and the areas which are not transformed in each frame of frame image are reserved, so that when the transformed areas are rendered subsequently, the areas which are not transformed are directly multiplexed, and the time is saved. And simultaneously recording the moment when the conversion area of each frame image is converted into a rendering instruction, and taking the moment as a time stamp of each frame image, wherein the time stamp is a part of the time interval which is used for determining each frame image.
After receiving the interactive operation information sent by the standby terminal, the embodiment of the application determines the rendering instruction and the time stamp corresponding to each frame of frame image to be rendered, wherein the time stamp is set to ensure the feasibility of subsequent steps and the time sequence of the frame images, and avoid disorder.
After the rendering instruction of each frame image is determined, the frame image to be rendered corresponding to the rendering instruction is rendered according to the rendering instruction, and a rendered frame image is obtained, which comprises the following steps:
rendering any frame image in a part of frame images, and performing rendering processing on rendering parameters corresponding to any frame image to obtain a rendered transformation area of any frame image;
and combining the rendered transformation area of any frame of frame image with the area which is not transformed in any frame of frame image to obtain the rendered any frame of frame image.
Fig. 4 is a schematic diagram of frame image combination provided in the embodiment of the present application, as shown in fig. 4, in which part 1 of the diagram indicates an area where no transformation occurs in a frame image, part 2 indicates an area where a frame image is rendered, and two parts 1 and 2 are combined into a rendered frame image. And rendering a frame image according to the mouse position coordinate parameters, wherein the frame image only has a part, namely a mouse position transformed region, and combining the part of transformed region and the region which is not transformed to form a complete frame image.
According to the real-time instance, the corresponding frame images to be rendered are rendered according to the rendering instruction, the transformed areas of each frame image are rendered, and the rendered transformed areas and the areas which are not transformed are combined to form the rendered frame image, so that the rendering time can be saved, the system resources can be reduced, and the rendering speed can be improved.
On the basis of the foregoing embodiments, as an optional embodiment, the sending, to the terminal, the rendered first frame image and the corresponding timestamp further includes:
and identifying the rendered first frame image according to the timestamp corresponding to the rendered first frame image, and sending the identified first frame image to the terminal.
After the frame image rendering is completed, the rendered frame image is required to be identified by the corresponding timestamp of the frame image, the sequence of the rendered frame image can be clearly known through the timestamp, the display time of the frame image can be conveniently determined according to the timestamp, the optimal frame image is selected, and a foundation is laid for the subsequent GUI rendering display.
Fig. 5 is an interaction schematic diagram of GUI rendering display performed by a terminal and a server according to an embodiment of the present application; as shown in fig. 5, the terminal sends the interactive operation information to the server, the server receives the interactive operation information, determines a rendering instruction and a timestamp corresponding to each frame image to be rendered, and forms a rendering instruction sequence, acquires graphics processing capability data of the frame images processed by the terminal and the server, determines a first rendering instruction and a second rendering instruction from the rendering instruction sequence, sends the second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction and the timestamp corresponding to the frame image to be rendered to the terminal, then the server renders the frame image corresponding to the first rendering instruction according to the first rendering instruction, obtains a first frame image after being rendered, and sends the first frame image to be rendered and the corresponding timestamp to the terminal, the terminal receives the second rendering instruction sent by the server, the frame image to be rendered corresponding to the second rendering instruction and the timestamp of the second rendering instruction, obtains a second frame image after being rendered, and receives the first frame image after being rendered and the second frame image after being rendered, and the timestamp corresponding to the first frame image after being rendered and the second frame image after being rendered are received, and the GUI image after being rendered is determined.
The application also provides a GUI rendering display method, which is applied to a terminal, and fig. 6 is a flowchart of another GUI rendering display method provided in this embodiment, as shown in fig. 6, where the method includes:
s201, the interactive operation information is sent to a server, so that the server updates a rendering instruction and a time stamp corresponding to each frame of frame image to be rendered and a first frame of image rendered by the server according to the interactive operation information, and sends a second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction and the time stamp corresponding to the second rendering instruction to the terminal.
After the terminal sends the interactive operation information to the server, specifically, the terminal sends the specific interactive operation information of the user to the server according to the operation of the user on the interface, so that the server updates a rendering instruction and a time stamp corresponding to each frame of frame image to be rendered according to the interactive operation information, renders the frame image corresponding to the first rendering instruction according to the first rendering instruction to obtain a rendered first frame image, for example, when the user moves a mouse at the terminal, the terminal sends the information of the movement of the mouse to the server, the server determines to generate a rendering instruction of each frame of frame image corresponding to the movement of the mouse according to the whole process of the movement of the mouse, renders the frame image according to the rendering instruction to obtain the rendered frame image, and sends the rendered frame image to the terminal for display by the terminal. If the user does not perform the interactive operation at the terminal, the terminal does not send the interactive operation information, but the GUI is refreshed at the moment, the rendering instruction and the time stamp corresponding to each frame of frame image to be rendered are updated, and then the frame image corresponding to the first rendering instruction is rendered according to the first rendering instruction, so that the rendered first frame image is obtained.
According to the method, the terminal records the interactive operation of the user, the interactive operation information is sent to the server, and the server generates the rendering instruction and the time stamp of each frame of frame image, so that a foundation is laid for realizing the rendering display of the GUI subsequently.
S207, receiving a second rendering instruction sent by the server, a frame image to be rendered corresponding to the second rendering instruction and a time stamp corresponding to the second rendering instruction, and rendering the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction to obtain a rendered second frame image.
According to the method, a server distributes a second rendering instruction, a frame image to be rendered corresponding to the second rendering instruction and a timestamp of the second rendering instruction to the terminal according to the performance of the terminal and the server for processing the image, renders the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction sent by the server, the frame image to be rendered corresponding to the second rendering instruction and the timestamp of the second rendering instruction, and the specific rendering method comprises the steps of rendering the frame image to be rendered through the second rendering instruction, obtaining a region transformed by the frame image to be rendered, combining the region transformed by the frame image and the region not transformed by the frame image into a complete rendered frame image, storing the frame image into a preset buffer pool, and caching the frame image to avoid losing the frame image.
S208, receiving the rendered first frame image and the corresponding time stamp sent by the server, and determining a GUI rendering result to be displayed by combining the rendered first frame image and the second frame image and the time stamp corresponding to the rendered first frame image and the second frame image.
According to the method and the device for displaying the GUI, the rendered first frame image and the corresponding timestamp sent by the server are combined with the rendered first frame image and the rendered second frame image and the timestamp corresponding to the rendered first frame image and the rendered second frame image, a GUI rendering result to be displayed needs to be determined according to the timestamp, specifically, when the frame image is displayed, the current moment needs to be acquired, the interval time of the frame image is obtained according to the current moment and the timestamp of the frame image, the frame image meeting the time requirement is determined according to the size of the interval time, the frame image which does not meet the time requirement needs to be deleted, and the frame images meeting the timestamp are displayed according to the sequence of the timestamps.
According to the GUI rendering display method, the server updates the rendering instruction and the corresponding time stamp of each frame of frame image according to the interactive operation information by sending the interactive operation information, the server and the terminal render the frame images to be rendered according to the rendering instruction respectively and independently to obtain the rendered frame images, the server sends the rendered frame images to the terminal, the terminal receives the rendered frame images sent by the server, and the terminal is used for displaying the system GUI in combination with the rendered frame images.
On the basis of the foregoing embodiments, as an optional embodiment, the rendering the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction, to obtain a rendered second frame image, and then further includes:
and identifying the rendered second frame image according to the timestamp corresponding to the rendered second frame image, and obtaining the identified second frame image.
After the frame image rendering is completed, the rendered frame image is required to be identified by the corresponding timestamp of the frame image, the sequence of the rendered frame image can be clearly known through the timestamp, the display time of the frame image can be conveniently determined according to the timestamp, the optimal frame image is selected, and a foundation is laid for the subsequent GUI rendering display.
On the basis of the foregoing embodiments, as an optional embodiment, determining, in combination with the rendered first frame image and the second frame image and the time stamps corresponding to the rendered first frame image and the rendered second frame image, a GUI rendering result to be displayed includes:
determining frame images to be displayed in the first frame image and the second frame image;
acquiring the current time, and determining the time interval of the frame images to be displayed according to the time stamp corresponding to the frame images to be displayed and the current time;
If the time interval of the frame images to be displayed is smaller than or equal to a preset threshold value, reserving the frame images to be displayed, and taking the frame images to be displayed as a GUI rendering result to be displayed;
and if the time interval of the frame images to be displayed is greater than a preset threshold value, the frame images to be displayed are cleared.
After receiving the first and second frame images, the embodiment of the application needs to determine the frame images to be displayed, acquire the current time, determine the time interval of any frame image according to the current time and the timestamp of the frame images to be displayed, compare the time interval of any frame image with the preset time interval, if the time interval is smaller than or equal to the preset interval, the any frame image meets the requirements, and send the frame image to the terminal, if the time interval is greater than the preset interval, delete the any frame image, clear the buffer Chi Huancun, avoid occupying resources and affecting other frame images, for example: there are 3 available frame images, X1, X2, X3 in time sequence, and the time intervals of the three frame images are T1, T2, T3 respectively, if the time intervals are too long, the image display is delayed, and is not coherent, so that experience is affected, if the image display is 60 frames/second, each frame time interval should be about 16.6666ms, and if a delay of 2 frames can be tolerated between the frame images, then frame images near the time interval of 49ms can be considered to be selected as frame images meeting the requirements. Assuming that the interval t1=52 ms of X1, the interval t2=46 ms of X2, and the interval t3=16 ms of X3, in order to display more frames in a unit time, X3 may be used as an optimal frame image for displaying smoother frames, X3 is reserved, where X2 also accords with a preset time interval, X2 is reserved as a standby frame image, in order to avoid that no frame image may be displayed when X3 is problematic or lost, a dead space occurs, the time interval of X1 is greater than the preset interval, and is not satisfactory, X1 is cleared, memory resources occupied by X1 in a buffer pool are released, and other frame images may be provided at the same time, so that resources are reasonably utilized.
Specifically, the server completes the frame image 1 and the frame image 2, sends the frame images to the terminal, and writes the frame images into a waiting available frame image queue; at this time, possibly due to network reasons, only the sequence that the frame images 1 and 2 can be inserted into the available frame queue of the terminal (in the case of transmitting by only one network card) can be guaranteed, at this time, the situation of the frame image 3 at the terminal cannot be determined, and if rendering is not completed, only the frame images 1 and 2 are in the available frame queue of the terminal; otherwise there are a number of possible frame queue orders that may be available, the sequence may be: frame image 3 (network busy, no transmission of server frame images 1, 2) or frame image 1, frame image 3 (no transmission of server frame image 2 to terminal) or frame image 3, frame image 1 or frame image 1, frame image 2, frame image 3 or frame image 1, frame image 3, frame image 2 or frame image 3, frame image 2, frame image 1, etc., at this time, the terminal needs to take out an optimal frame image according to the timestamp of the frame image to display, if 3 is selected to refresh the screen, 1, 2 will be abandoned;
yet another possibility is that the GPU and network capabilities of the server are too good, the server frame image 4 has arrived at the terminal, then it is possible that the frame image 3 task being rendered by the terminal (the terminal GPU capabilities are too weak), the screen is refreshed directly with the frame image 4,
According to the method and the device for displaying the frame images, the frame images meeting the preset requirements are determined according to the time stamp and the current time of any frame image to be displayed, the display quality of the frame images can be improved, the smoothness of the whole display is ensured, and the overall working efficiency is improved.
The embodiment of the application provides a server, as shown in fig. 7, the server may include: a first receiving module 302, a first processing module 303, a second processing module 304, a second sending module 305, and a third processing module 306, specifically:
the first receiving module 302 is configured to receive the interaction information sent by the terminal.
The first processing module 303 is configured to update a rendering instruction and a timestamp corresponding to each frame of frame image to be rendered according to the interaction operation information, and form a rendering instruction sequence; the timestamp is used for indicating the initiation time of a screen update period corresponding to a rendering instruction of the frame image.
The second processing module 304 is configured to obtain graphics processing capability data of the terminal and the server for processing the frame image, so as to determine a first rendering instruction and a second rendering instruction from the rendering instruction sequence, where the first rendering instruction is a rendering instruction executed by the server, and the second rendering instruction is a rendering instruction pointed by the terminal.
The second sending module 305 is configured to send the second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction, and the timestamp corresponding to the second rendering instruction to the terminal, so that the terminal renders the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction, and obtains a rendered second frame image.
The third processing module 306 is configured to render the frame image corresponding to the first rendering instruction according to the first rendering instruction, obtain a rendered first frame image, and send the rendered first frame image and a corresponding timestamp to the terminal, so that the terminal determines a GUI rendering result to be displayed according to the rendered first frame image and the second frame image and the timestamp corresponding to the rendered first frame image and the rendered second frame image.
The server provided in this embodiment of the present application, specifically executes the process of the foregoing method embodiment on the server side, please see in detail the content of the embodiment of the foregoing GUI rendering display method on the server side, which is not described herein, updates the rendering instruction of each frame image and the timestamp corresponding to the screen update period according to the interaction operation information sent by the terminal by receiving the interaction operation information, where the server and the terminal render the frame image to be rendered according to the rendering instruction separately to obtain the rendered frame image, and sends the rendered frame image to the terminal, where the terminal receives the rendered frame image sent by the server, and combines the rendered frame image of the terminal to display the GUI together.
On the basis of the above embodiments, as an alternative embodiment, the first processing module 303 includes:
the determining module is used for updating and processing the frame image to be rendered and the rendering node in the frame image to be rendered according to the interactive operation information sent by the terminal;
the method comprises the steps of determining a rendering instruction corresponding to a frame image to be rendered according to the frame image to be rendered and a rendering node in the frame image to be rendered, and taking the starting time of a screen update period corresponding to the frame image to be rendered as a time stamp corresponding to the frame image to be rendered.
Based on the foregoing embodiments, as an alternative embodiment, the third processing module 306 includes:
the first identification module is used for identifying the rendered first frame image according to the timestamp corresponding to the rendered first frame image and sending the identified first frame image to the terminal.
The embodiment of the application provides a terminal, as shown in fig. 8, the terminal may include: the first transmitting module 301, the second receiving module 307, and the third receiving module 308, specifically,
the first sending module 301 is configured to send the interaction information to the server, so that the server updates a rendering instruction and a timestamp corresponding to each frame of frame image to be rendered and a first frame image rendered by the server according to the interaction information, and sends a second rendering instruction, a frame image to be rendered corresponding to the second rendering instruction, and a timestamp of the second rendering instruction to the terminal;
The second receiving module 307 is configured to receive a second rendering instruction sent by the server, a frame image to be rendered corresponding to the second rendering instruction, and a timestamp corresponding to the second rendering instruction, and render the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction, so as to obtain a rendered second frame image;
the third receiving module 308 is configured to receive the rendered first frame image and the corresponding timestamp sent by the server, and determine a GUI rendering result to be displayed in combination with the rendered first frame image and the second frame image and the timestamps corresponding to the rendered first frame image and the rendered second frame image.
The terminal provided in this embodiment of the present application, specifically executes the process of the foregoing method embodiment on the terminal side, specifically please see details of the foregoing GUI rendering display method on the content of the embodiment on the terminal side, which is not described herein, makes the server update the rendering instruction of each frame image and the timestamp corresponding to the screen update period according to the interactive operation information by sending the interactive operation information, the server and the terminal render the frame image to be rendered according to the rendering instruction separately, so as to obtain the rendered frame image, the server sends the rendered frame image to the terminal, the terminal receives the rendered frame image sent by the server, compares the rendered frame image with the locally latest frame image according to the timestamp, and selects the latest frame image to display on the terminal.
On the basis of the above embodiments, as an alternative embodiment, the second receiving module 307 includes:
the second identification module is used for identifying the rendered second frame image according to the timestamp corresponding to the rendered second frame image to obtain the identified second frame image.
On the basis of the above embodiments, as an alternative embodiment, the third receiving module 308 includes:
the screening module is used for determining frame images to be displayed in the first frame image and the second frame image;
acquiring the current time, and determining the time interval of the frame images to be displayed according to the time stamp corresponding to the frame images to be displayed and the current time;
if the time interval of the frame images to be displayed is smaller than or equal to a preset threshold value, reserving the frame images to be displayed, and taking the frame images to be displayed as a GUI rendering result to be displayed;
and if the time interval of the frame images to be displayed is greater than a preset threshold value, the frame images to be displayed are cleared.
An embodiment of the present application provides an electronic device, including: a memory and a processor; at least one program stored in the memory for execution by the processor, which, when executed by the processor, performs: according to the scheme, through receiving the interactive operation information sent by the terminal, the rendering instruction of each frame image and the time stamp corresponding to the screen updating period are updated according to the interactive operation information, the server and the terminal render the frame images to be rendered according to the rendering instruction respectively and independently to obtain the rendered frame images, the server sends the rendered frame images to the terminal, the terminal receives the rendered frame images sent by the server, and the terminal is combined with the rendered frame images, so that the terminal and the server are jointly used for displaying the frame images of the system GUI.
In an alternative embodiment, there is provided an electronic device, as shown in fig. 9, the electronic device 4000 shown in fig. 9 includes: a processor 4001 and a memory 4003. Wherein the processor 4001 is coupled to the memory 4003, such as via a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004. It should be noted that, in practical applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 is not limited to the embodiment of the present application.
The processor 4001 may be a CPU (Central Processing Unit ), general purpose processor, DSP (Digital Signal Processor, data signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic device, transistor logic device, hardware components, or any combination thereof. Which may implement or perform the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. The processor 4001 may also be a combination that implements computing functionality, e.g., comprising one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
Bus 4002 may include a path to transfer information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture ) bus, or the like. The bus 4002 can be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in fig. 9, but not only one bus or one type of bus.
Memory 4003 may be, but is not limited to, ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, RAM (Random Access Memory ) or other type of dynamic storage device that can store information and instructions, EEPROM (Electrically Erasable Programmable Read Only Memory ), CD-ROM (Compact Disc Read Only Memory, compact disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
The memory 4003 is used for storing application program codes for executing the present application, and execution is controlled by the processor 4001. The processor 4001 is configured to execute application program codes stored in the memory 4003 to realize what is shown in the foregoing method embodiment.
The present application provides a computer readable storage medium having a computer program stored thereon, which when run on a computer, causes the computer to perform the corresponding method embodiments described above. Compared with the prior art, the method and the device have the advantages that through receiving the interactive operation information sent by the terminal, the rendering instruction of each frame image and the time stamp corresponding to the screen updating period are updated according to the interactive operation information, the server and the terminal render the frame images to be rendered according to the rendering instruction respectively and independently to obtain the rendered frame images, the server sends the rendered frame images to the terminal, the terminal receives the rendered frame images sent by the server, and the terminal is combined with the rendered frame images, so that the terminal and the server are jointly used for displaying the frame images of the system GUI to isolate the capability of processing the frame images, the capability of each other is not affected, the respective capability is fully exerted, the time stamp information is used for synchronization, time sequence errors are avoided, the image processing rate can be improved, the hardware resources of the terminal and the server can be fully utilized, the overall efficiency is improved, and the user experience is improved.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.
The foregoing is only a partial embodiment of the present invention, and it should be noted that it will be apparent to those skilled in the art that modifications and adaptations can be made without departing from the principles of the present invention, and such modifications and adaptations should and are intended to be comprehended within the scope of the present invention.

Claims (10)

1. A method for rendering a display of a GUI applied to a server, comprising:
receiving interactive operation information sent by a terminal;
Updating a rendering instruction and a time stamp corresponding to each frame of frame image to be rendered according to the interactive operation information, and forming a rendering instruction sequence; the time stamp is used for representing the initiation time of a screen update period corresponding to a rendering instruction of the frame image;
acquiring graphic processing capability data of a terminal and a server for processing frame images, so as to determine a first rendering instruction and a second rendering instruction from the rendering instruction sequence, wherein the first rendering instruction is a rendering instruction executed by the server, and the second rendering instruction is a rendering instruction executed by the terminal;
the second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction and the timestamp corresponding to the second rendering instruction are sent to the terminal, so that the terminal renders the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction, and a rendered second frame image is obtained;
and rendering the frame image corresponding to the first rendering instruction according to the first rendering instruction, obtaining a rendered first frame image, and sending the rendered first frame image and a corresponding timestamp to the terminal, so that the terminal determines a GUI rendering result to be displayed according to the rendered first frame image and the second frame image and the timestamps corresponding to the rendered first frame image and the rendered second frame image.
2. The GUI rendering display method according to claim 1, wherein updating the rendering instruction and the time stamp corresponding to each frame of frame image to be rendered according to the interactive operation information comprises:
updating and processing a frame image to be rendered and a rendering node in the frame image to be rendered according to the interactive operation information sent by the terminal;
determining a rendering instruction corresponding to the frame image to be rendered according to the frame image to be rendered and a rendering node in the frame image to be rendered, and taking the initiating time of a screen updating period corresponding to the frame image to be rendered as a time stamp corresponding to the frame image to be rendered.
3. The method of GUI rendering a display according to claim 1, wherein the sending the rendered first frame image and the corresponding timestamp to the terminal further comprises:
and identifying the rendered first frame image according to the timestamp corresponding to the rendered first frame image, and sending the identified first frame image to the terminal.
4. A method for rendering and displaying a GUI applied to a terminal, comprising:
The method comprises the steps of sending interactive operation information to a server, so that the server updates a rendering instruction corresponding to each frame of frame image to be rendered and a time stamp according to the interactive operation information, and forms a rendering instruction sequence, wherein the time stamp is used for representing the initiation time of a screen update period corresponding to the rendering instruction of the frame image; sending graphic processing capability data of a terminal for processing a frame image to the server, so that the server determines a first rendering instruction and a second rendering instruction from the rendering instruction sequence according to the graphic processing capability data of the terminal and the graphic processing capability data of the server for processing the frame image, wherein the first rendering instruction is executed by the server, and the second rendering instruction is executed by the terminal;
receiving the second rendering instruction, a frame image to be rendered corresponding to the second rendering instruction and a time stamp corresponding to the second rendering instruction sent by a server, and rendering the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction to obtain a rendered second frame image;
and receiving the rendered first frame image and the corresponding time stamp sent by the server, and determining a GUI rendering result to be displayed by combining the rendered first frame image and the rendered second frame image and the time stamp corresponding to the rendered first frame image and the rendered second frame image.
5. The method for rendering and displaying a GUI according to claim 4, wherein the rendering the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction, to obtain a rendered second frame image, further comprises:
and identifying the rendered second frame image according to the timestamp corresponding to the rendered second frame image, so as to obtain the identified second frame image.
6. The method of GUI rendering according to claim 4, wherein the determining the GUI rendering result to be displayed by combining the rendered first and second frame images and the time stamps corresponding to the rendered first and second frame images includes:
determining frame images to be displayed in the first frame image and the second frame image;
acquiring the current time, and determining the time interval of the frame image to be displayed according to the time stamp corresponding to the frame image to be displayed and the current time;
if the time interval of the frame images to be displayed is smaller than or equal to a preset threshold value, reserving the frame images to be displayed, and taking the frame images to be displayed as GUI rendering results to be displayed;
And if the time interval of the frame images to be displayed is larger than a preset threshold value, the frame images to be displayed are cleared.
7. A server, comprising:
the first receiving module is used for receiving the interactive operation information sent by the terminal;
the first processing module is used for updating a rendering instruction and a time stamp corresponding to each frame of frame image to be rendered according to the interactive operation information, and forming a rendering instruction sequence; the time stamp is used for representing the initiation time of a screen update period corresponding to a rendering instruction of the frame image;
the second processing module is used for acquiring graphic processing capability data of a terminal and a server for processing frame images so as to determine a first rendering instruction and a second rendering instruction from the rendering instruction sequence, wherein the first rendering instruction is a rendering instruction executed by the server, and the second rendering instruction is a rendering instruction pointed by the terminal;
the second sending module is used for sending the second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction and the timestamp corresponding to the second rendering instruction to the terminal, so that the terminal renders the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction, and a rendered second frame image is obtained;
And the third processing module is used for rendering the frame image corresponding to the first rendering instruction according to the first rendering instruction, obtaining a rendered first frame image, and sending the rendered first frame image and the corresponding timestamp to the terminal, so that the terminal determines a GUI rendering result to be displayed according to the rendered first frame image and the second frame image and the timestamp corresponding to the rendered first frame image and the rendered second frame image.
8. A terminal, comprising:
the first sending module is used for sending the interactive operation information to the server, so that the server updates a rendering instruction and a time stamp corresponding to each frame of frame image to be rendered according to the interactive operation information, and forms a rendering instruction sequence, wherein the time stamp is used for representing the initiation time of a screen update period corresponding to the rendering instruction of the frame image; sending graphic processing capability data of a terminal for processing a frame image to the server, so that the server determines a first rendering instruction and a second rendering instruction from the rendering instruction sequence according to the graphic processing capability data of the terminal and the graphic processing capability data of the server for processing the frame image, wherein the first rendering instruction is executed by the server, and the second rendering instruction is executed by the terminal;
The second receiving module is used for receiving the second rendering instruction, the frame image to be rendered corresponding to the second rendering instruction and the timestamp corresponding to the second rendering instruction sent by the server, and rendering the frame image to be rendered corresponding to the second rendering instruction according to the second rendering instruction to obtain a rendered second frame image;
and the third receiving module is used for receiving the rendered first frame image and the corresponding time stamp sent by the server, and determining a GUI rendering result to be displayed by combining the rendered first frame image and the second frame image and the time stamp corresponding to the rendered first frame image and the second frame image.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor performs the steps of the method of rendering a display of a GUI as claimed in any one of claims 1 to 6 when the program is executed.
10. A computer readable storage medium storing computer instructions for causing a computer to perform the steps of the method of GUI rendering display according to any of claims 1 to 6.
CN202011551948.8A 2020-12-24 2020-12-24 GUI rendering display method, terminal, server, electronic equipment and storage medium Active CN112614202B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011551948.8A CN112614202B (en) 2020-12-24 2020-12-24 GUI rendering display method, terminal, server, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011551948.8A CN112614202B (en) 2020-12-24 2020-12-24 GUI rendering display method, terminal, server, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112614202A CN112614202A (en) 2021-04-06
CN112614202B true CN112614202B (en) 2023-07-14

Family

ID=75245487

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011551948.8A Active CN112614202B (en) 2020-12-24 2020-12-24 GUI rendering display method, terminal, server, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112614202B (en)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113360150B (en) * 2021-05-25 2024-04-26 广东海启星海洋科技有限公司 Multi-module data linkage display method and device
CN113296865B (en) * 2021-05-28 2022-09-06 贵州海誉科技股份有限公司 Cloud desktop mouse position sensing and mouse operation event management method
CN113244614B (en) * 2021-06-07 2021-10-26 腾讯科技（深圳）有限公司 Image picture display method, device, equipment and storage medium
CN113342527A (en) * 2021-06-09 2021-09-03 炫我信息技术(北京)有限公司 Method, apparatus, electronic device, and computer-readable storage medium for rendering
CN114115688B (en) * 2021-11-23 2023-08-01 北京字节跳动网络技术有限公司 Method and device for inhibiting display jitter, electronic equipment and storage medium
CN114077508B (en) * 2022-01-19 2022-10-11 维塔科技(北京)有限公司 Remote image rendering method and device, electronic equipment and medium
CN114501062B (en) * 2022-01-27 2023-02-21 腾讯科技（深圳）有限公司 Video rendering coordination method, device, equipment and storage medium
CN114205359A (en) * 2022-01-27 2022-03-18 腾讯科技（深圳）有限公司 Video rendering coordination method, device and equipment
CN114513512B (en) * 2022-02-08 2023-01-24 腾讯科技（深圳）有限公司 Interface rendering method and device
CN114820272A (en) * 2022-03-18 2022-07-29 维塔科技(北京)有限公司 Data interaction method and device, storage medium and electronic equipment
CN116450074B (en) * 2023-06-15 2023-09-05 摩尔线程智能科技(北京)有限责任公司 Image display method and device

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108765531A (en) * 2018-03-27 2018-11-06 广东欧珀移动通信有限公司 Image rendering method, device, storage medium and intelligent terminal
CN111654720A (en) * 2020-07-17 2020-09-11 腾讯科技（深圳）有限公司 Video coding method, device, equipment and readable storage medium
CN111818120A (en) * 2020-05-20 2020-10-23 北京元心科技有限公司 End cloud user interaction method and system, corresponding equipment and storage medium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190114733A1 (en) * 2017-10-12 2019-04-18 Red Hat, Inc. Display content currentness validation

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108765531A (en) * 2018-03-27 2018-11-06 广东欧珀移动通信有限公司 Image rendering method, device, storage medium and intelligent terminal
CN111818120A (en) * 2020-05-20 2020-10-23 北京元心科技有限公司 End cloud user interaction method and system, corresponding equipment and storage medium
CN111654720A (en) * 2020-07-17 2020-09-11 腾讯科技（深圳）有限公司 Video coding method, device, equipment and readable storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
多通道三维场景渲染的同步策略研究;徐莉;《北京信息科技大学学报(自然科学版)》;第30卷(第4期);27-30、45 *

Also Published As

Publication number Publication date
CN112614202A (en) 2021-04-06

Similar Documents

Publication Publication Date Title
CN112614202B (en) GUI rendering display method, terminal, server, electronic equipment and storage medium
US11909984B2 (en) Video encoding and decoding for cloud gaming
CN113661471B (en) Hybrid rendering
CN113244614B (en) Image picture display method, device, equipment and storage medium
CN111818120A (en) End cloud user interaction method and system, corresponding equipment and storage medium
CN112596843B (en) Image processing method, device, electronic equipment and computer readable storage medium
CN112843676B (en) Data processing method, device, terminal, server and storage medium
CN111491208B (en) Video processing method and device, electronic equipment and computer readable medium
WO2023173516A1 (en) Data exchange method and apparatus, and storage medium and electronic device
KR20190138798A (en) Live Ink Presence for Real-Time Collaboration
JPWO2011077550A1 (en) Screen relay device
WO2006001679A1 (en) Method and system for renewing screen
WO2017129105A1 (en) Graphical interface updating method and device
CN114570020A (en) Data processing method and system
CN112565883A (en) Video rendering processing system and computer equipment for virtual reality scene
CN110089076B (en) Method and device for realizing information interaction
CN116958487A (en) Gesture prediction method, device, apparatus, storage medium, and computer program product
CN116966546A (en) Image processing method, apparatus, medium, device, and program product
CN116843802A (en) Virtual image processing method and related product
CN111243069B (en) Scene switching method and system of Unity3D engine
CN115705668A (en) View drawing method and device and storage medium
JPH10198823A (en) Video generating device
US20060170706A1 (en) Systems and methods for rendering three-dimensional graphics in a multi-node rendering system
CN113034653A (en) Animation rendering method and device
US20230122666A1 (en) Cloud xr-based program virtualizing method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant