WO2022242487A1

WO2022242487A1 - Display method and related device

Info

Publication number: WO2022242487A1
Application number: PCT/CN2022/091634
Authority: WO
Inventors: 朱伟; 孙兴阳
Original assignee: 华为技术有限公司
Priority date: 2021-05-19
Filing date: 2022-05-09
Publication date: 2022-11-24
Also published as: CN115373777A

Abstract

A display method and a related device. The display method comprises: in response to an instruction sent by a window management module, an application program layer determining, on the basis of the interface layout of a first user interface in a first display picture, the position and size 1 of an image layer of the first user interface in a user interface image having resolution 1, and the position and size of a first dynamic interface element in the image layer of the first user interface, so as to draw and render an image layer of size 1 corresponding to a dynamic part of the first user interface (S101); sending image layer data of the image layer corresponding to the dynamic part of each user interface in the first display picture (S102); synthesizing a user interface image 1 having resolution 1 on the basis of image layer data of at least one image layer sent by the application program layer (S103); sending image cache data of the user interface image 1 having resolution 1 (S104); amplifying the user interface image having resolution 1 to a user interface image 1 having resolution 2, resolution 2 being greater than resolution 1 (S105); sending image cache data of the user interface image 1 having resolution 2 (S106); in response to an instruction 1 sent by the window management module, the application program layer further determining that a base image of the first display picture is base image 3, and acquiring storage data of the base image 3 having resolution 2 from a base image storage module (S107); sending the storage data of the base image 3 (S108); performing decoding on the basis of the storage data to obtain the base image 3 having resolution 2 (S109); sending image cache data of the base image 3 having resolution 2 (S110); and generating a composite image on the basis of the user interface image 1 having resolution 2 and the base image 3 having resolution 2, and sending the composite image to a display screen 194 for display, the user interface image 1 being located above the base image 3 (S111). By implementing the method, smooth and clear high-resolution display can be achieved, the power consumption of high-resolution display is reduced, and the user experience is effectively improved.

Description

Display method and related device

This application claims the priority of a Chinese patent application with application number 202110547689.X and application title "A Display Method and Related Device" filed with the China Patent Office on May 19, 2021, the entire contents of which are hereby incorporated by reference Applying. This application claims the priority of a Chinese patent application with application number 202110742507.4 and application title "Display Method and Related Devices" filed with the China Patent Office on June 30, 2021, the entire contents of which are incorporated herein by reference.

technical field

The present application relates to the field of electronic technology, in particular to a display method and related devices.

Background technique

With the popularization and development of smart terminal devices, the display screen resolution required by smart terminal devices is getting higher and higher, gradually developing towards 2K resolution, 3K resolution, 4K resolution, and 8K resolution. The higher the display resolution of smart terminal equipment, the finer the picture quality displayed, and the more realistic the visual effects of applications such as video and games.

At present, the high-resolution interface display has problems such as picture freezing, blurred image details, and high power consumption.

Contents of the invention

The present application provides a display method and a related device, which realize smooth and clear high-resolution display, reduce power consumption of high-resolution display, and effectively improve user experience.

In a first aspect, the present application provides a display method, which is applied to a terminal device, and the terminal device includes a display screen, including: drawing a first user interface image of a first resolution corresponding to a first display screen in response to a first instruction, And acquire the first base image of the second resolution of the first display image; wherein, the first instruction is used to instruct the terminal device to draw the first display image of the second resolution, and the first user interface image includes the first display image in the first display image Dynamic interface elements, the first base map includes static interface elements in the first display screen, the terminal device stores at least one base map of the display screen that has been drawn, and the second resolution is greater than the first resolution; based on the first resolution The first user interface image and the first base image of the second resolution are synthesized into the first display picture of the second resolution; and the first display picture is displayed through the display screen.

Implement the embodiment of the present application, divide the display screen into the base map of the static part and the user interface image of the dynamic part, the base map includes the static interface elements in the display screen, and the user interface image of the dynamic part only includes all the user interfaces in the display screen The dynamic interface element in the terminal device stores a pre-drawn high-resolution basemap; in response to the first instruction, the terminal device only draws a low-resolution user interface image containing the dynamic interface element, and then converts the low-resolution The composite image of the user interface image and the high-resolution basemap is sent to the display screen for display. Implementing the embodiment of the present application can realize high-resolution interface display and effectively improve user experience. In addition, the load and performance requirements on hardware such as processor and/or memory are greatly reduced, and the power consumption of real-time interface display is effectively reduced.

In an implementation manner, the first display screen includes at least one user interface, and the drawing of the first user interface image with the first resolution corresponding to the first display screen includes: drawing the dynamic interface element corresponding to the first user interface The first layer, the first user interface is any user interface in the at least one user interface, the resolution of the first layer is less than or equal to the first resolution; the dynamics of each user interface in the first display screen The layers corresponding to the interface elements are synthesized into the first user interface image with the first resolution.

In an implementation manner, the drawing of the first layer corresponding to the dynamic interface elements of the first user interface includes: determining the first layer in the first user interface image with the first resolution based on the interface layout of the first user interface. The size of the layer is the first size, the position of the first layer is the first position, the position of the first dynamic interface element in the first layer of the first size is the second position and the size of the first dynamic interface element is The second size, the first dynamic interface element is any dynamic interface element of the first user interface in the first layer; based on the second position and second size of the first dynamic interface element, draw on the canvas of the first size The first dynamic interface element generates a first layer of a first size corresponding to the dynamic interface element of the first user interface.

In an implementation manner, the above synthesizing the layers corresponding to the dynamic interface elements of each user interface in the first display screen into the first user interface image with the first resolution includes: based on the first layer in the first The first position and the first size in the first user interface image of the resolution, and the layers corresponding to the dynamic interface elements of each user interface are synthesized into the first user interface image of the first resolution.

In an implementation manner, the synthesis of the first display image of the second resolution based on the first user interface image of the first resolution and the first base image of the second resolution includes: combining the first user interface image of the first resolution Enlarging a user interface image into a first user interface image with a second resolution; synthesizing the first user interface image with the second resolution and the first base image with the second resolution into a first display image with the second resolution.

In an implementation manner, the drawing of the first layer corresponding to the dynamic interface elements of the first user interface includes: determining the first layer in the first user interface image of the second resolution based on the interface layout of the first user interface The position of the first dynamic interface element in a layer is the third position and the size of the first dynamic interface element is the third size; based on the third size of the first dynamic interface element, draw the first dynamic interface on the canvas of the fourth size The interface elements generate the first layer, there is no overlap between the dynamic interface elements in the first layer, and the position of the first dynamic interface element in the first layer is the fourth position.

In an implementation manner, the dynamic interface elements of the first user interface on the first layer are sequentially arranged in a first preset order on the first layer.

In an implementation manner, the above synthesizing the layers corresponding to the dynamic interface elements of each user interface in the first display screen into the first user interface image with the first resolution includes: based on the first The third size and the fourth position of the dynamic interface element determine that the position of the first dynamic interface element in the first user interface image of the first resolution is the fifth position; based on the first dynamic interface element in the first layer at the A third size and a fifth position in a user interface image are synthesized into a first user interface image with a first resolution.

In an implementation manner, the dynamic interface elements of multiple layers in the first user interface image are sequentially arranged in a second preset order in the first user interface image.

In an implementation manner, synthesizing the first user interface image with the first resolution based on the third size and the fifth position of the first dynamic interface element in the first layer in the first user interface image includes: obtaining The image data of the first area where the first dynamic interface element is located in the first layer, the size of the first area is the third size, and the position of the first area is the fourth position; determine the first dynamic interface in the first user interface image The image data of the second area where the element is located is the image data of the first area, the size of the second area is the third size, and the position of the second area is the fifth position.

In an implementation manner, synthesizing the first display picture with the second resolution based on the first user interface image with the first resolution and the first base map with the second resolution includes: obtaining the first user interface image in the first user interface image The image data of the second area where the first dynamic interface element is located, the size of the second area is the third size, and the position of the second area is the fifth position; obtain the image data of the third area in the first base image, the third area The size of is the third size, and the position of the third area is the third position of the first dynamic interface element; the image data of the second area and the image data of the third area are synthesized into the image data of the third area of the first display screen , to generate the first display screen; the size of the third area is the third size, and the position of the third area is the third position.

In one implementation, when the terminal device runs the first display screen, the interface elements whose display content is not changeable in the first display screen are static interface elements, and the interface elements whose display content is variable are dynamic interface elements; or, the terminal device runs When the first display screen is displayed, interface elements whose display content change probability is lower than a preset value in the first display screen are static interface elements, and interface elements whose display content change probability is greater than or equal to a preset value are dynamic interface elements.

In an implementation manner, the drawing of the first user interface image of the first resolution corresponding to the first display screen and obtaining the first base image of the second resolution of the first display screen include: using the first frame rate Draw the first user interface image of the first resolution corresponding to the first display picture, and acquire the first base map of the second resolution of the first display picture at the second frame rate; the first frame rate is greater than the second frame rate; The above synthesizing the first display image with the second resolution includes: synthesizing the first display image with the second resolution at the first frame rate.

In one implementation manner, the fourth size is smaller than the first size, which is the size of the first layer in the first user interface image at the first resolution.

Compared with traditional layer drawing, in the embodiment of the present application, by reducing the size of the drawn layer, performance requirements and power consumption for drawing user interface images can be further reduced.

In an implementation manner, the fourth size is greater than the sum of sizes of all dynamic interface elements of the first user interface in the first user interface image of the second resolution.

In an implementation manner, the fourth size is determined by the terminal device based on the sum of the foregoing sizes.

In one implementation manner, the first dynamic interface element is a single dynamic interface element or a composite dynamic interface element composed of multiple dynamic interface elements.

In this embodiment of the present application, drawing and compositing can be performed at the granularity of composite dynamic interface elements. In this way, the unit granularity of drawing and compositing is larger, which can improve the rendering and compositing efficiency of the display picture.

In a second aspect, the present application provides a terminal device, including one or more processors and one or more memories. The one or more memories are coupled with one or more processors, the one or more memories are used to store computer program codes, the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the terminal device executes A display method in any possible implementation manner of any one of the above aspects.

In a third aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are run on a terminal device, the terminal device is made to execute the display method in any possible implementation of any one of the above aspects.

In a fourth aspect, an embodiment of the present application provides a computer program product, which, when the computer program product is run on a computer, causes the computer to execute the display method in any possible implementation manner of any one of the above aspects.

Description of drawings

FIG. 1A to FIG. 1C are schematic diagrams of a user interface of an application scenario provided by an embodiment of the present application;

FIG. 2A is a schematic structural diagram of a terminal device provided in an embodiment of the present application;

FIG. 2B is a schematic diagram of a base map storage provided by the embodiment of the present application;

FIG. 3 is a schematic diagram of a software system architecture provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of the internal implementation of a display method provided by the embodiment of the present application;

FIG. 5A is a schematic diagram of the internal implementation of another display method provided by the embodiment of the present application;

FIG. 5B is a schematic flowchart of another display method provided by the embodiment of the present application;

FIG. 6A is a schematic diagram of a layer corresponding to a dynamic part of a user interface provided in an embodiment of the present application;

FIG. 6B is a schematic diagram of layers corresponding to the dynamic part of another user interface provided by the embodiment of the present application;

FIG. 6C is an enlarged schematic diagram of a layer provided in the embodiment of the present application;

FIG. 6D is a composite schematic diagram of a layer and a base map provided by the embodiment of the present application;

FIG. 7A is a schematic diagram of the internal implementation of another display method provided by the embodiment of the present application;

FIG. 7B is a schematic flowchart of another display method provided by the embodiment of the present application;

FIG. 8A is a schematic diagram of layers corresponding to the dynamic part of another user interface provided by the embodiment of the present application;

FIG. 8B is a schematic diagram of layers corresponding to the dynamic part of another user interface provided by the embodiment of the present application;

Fig. 8C is a composite schematic diagram of another layer and base map provided by the embodiment of the present application;

Fig. 9A is a schematic diagram of a composite dynamic interface element provided by the embodiment of the present application;

FIG. 9B is a schematic diagram of layers corresponding to the dynamic part of another user interface provided by the embodiment of the present application;

FIG. 9C is a schematic diagram of layers corresponding to the dynamic part of another user interface provided by the embodiment of the present application;

FIG. 9D is a schematic diagram of another compound dynamic interface element provided by the embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. Among them, in the description of the embodiments of this application, unless otherwise specified, "/" means or means, for example, A/B can mean A or B; "and/or" in the text is only a description of associated objects The association relationship indicates that there may be three kinds of relationships, for example, A and/or B, which may indicate: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiment of the present application , "plurality" means two or more than two.

Hereinafter, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as implying or implying relative importance or implicitly specifying the quantity of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present application, unless otherwise specified, the "multiple" The meaning is two or more.

First, the technical concepts involved in the embodiments of the present application are introduced.

Display resolution: refers to the physical resolution of the display itself, usually expressed in the form of "horizontal pixels X vertical pixels" on the display panel in the display, for example, the resolution of 800×600 refers to the Each row can display 800 pixels horizontally, and each column can display 600 pixels vertically.

Image Resolution: Refers to the amount of information stored in an image. There are many measurement methods for image resolution. In measurement method 1, image resolution is measured by the number of pixels in the picture, expressed in the form of "number of horizontal pixels X number of vertical pixels" in the picture. In measurement method 2, the image resolution is measured by the number of pixels per inch (PPI, pixel per inch) or the number of pixels per centimeter (PPC, pixel per centimeter). In the embodiment of the present application, measurement method 1 is taken as an example for description. It can be understood that in measurement method 1, the image resolution of an image may also correspond to the size of the image, that is, the number of horizontal pixels is the width of the image, and the number of vertical pixels is the height of the image.

It should be noted that nK resolutions such as 2K resolution, 3K resolution, 4K resolution, and 8K resolution involved in this application refer to the number of pixels in each row in the horizontal direction being close to n times 1024, or the total number of pixels is close to nK resolution The preset pixel amount of the image. For example, a resolution whose total number of pixels is close to 8,847,360 pixels of a 4K image may also be referred to as 4K resolution. Therefore, the nK resolution can have various specifications. For example, the existing 4K resolution includes 4096×2160, 3656*2664, 3840*2160 and other resolutions. The 4K resolution involved in the subsequent embodiments takes 4096×2560 resolution as an example, and the 2K resolution takes 2560×1440 resolution as an example, which is not limited in this application.

User interface (user interface, UI), the user interface is the medium interface for interaction and information exchange between the application program or operating system and the user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is the source code written in a specific computer language such as java and extensible markup language (XML). The source code of the interface is parsed and rendered on the electronic device, and finally presented as content that can be recognized by the user.

A control is a basic interface element of a user interface. A control can encapsulate data and methods, and a control can have its own properties and methods. For example, the types of controls may include, but are not limited to: user interface controls (controls used to develop and build user interfaces), chart controls (controls used to develop charts, which can realize data visualization, etc.), report controls (used to develop reports) Controls, which realize the functions of browsing, designing, editing, printing, etc. of reports), table controls (controls used to develop tables (cells), and realize the functions of data processing and operation in grids), composite controls formed by multiple controls, etc. .

In the embodiment of the present application, controls may include but not limited to: window, scroll bar, table view, button, menu bar, text box, navigation Bars, toolbars (toolbar), images (lmage), static text (tatictext), components (Widget) and other visual interface elements.

Optionally, in response to the user input that the user acts on the control, the terminal device may execute the response event corresponding to the control. For example, the main interface of the terminal device displays the icon 1 of the video application, and in response to the user input that the user acts on the icon 1 , the terminal device switches from the current user interface to the user interface of the video application.

The application program window: the application program window may be an application program window in the Android system, an application program window in the IOS system, or an application program window in other operating systems, which is not specifically limited here. In this embodiment of the present application, the application program window may be referred to as a window for short. An application includes multiple application windows, and one application window corresponds to one or more user interfaces.

Layers: Generally, there are mainly three types of layers displayed by the terminal device 100 , namely, a status bar, a navigation bar, and a user interface corresponding to an application window. In the embodiment of the present application, the status bar and the navigation bar can be respectively used as a user interface of the desktop application, and the terminal device 100 can draw and render at least one layer corresponding to the application window running in the foreground, and perform the above-mentioned at least one layer After synthesis, it is sent to the display screen.

Layer size: refers to the size of the area occupied by the layer when it is displayed on the display screen, and is usually characterized by two parameters: the width (such as the number of horizontal pixels) and the height (such as the number of vertical pixels) of the layer.

Layer position: refers to the display position of the layer on the display screen, usually represented by the display coordinates of the upper left corner or lower left corner of the layer on the display screen. The above display coordinates are usually pixel coordinates, for example, the pixel coordinates of the upper left corner of the layer are (50,100). It can be understood that based on the size and position of the layer, the display area of the layer on the display screen can be determined.

Transparency of a layer: refers to the degree of transparency of a layer, which affects the visual effect of overlapping display of this layer with other layers, usually represented by the parameter Alpha. In one implementation, the value of Alpha ranges from 0 to 100%. When the value of Alpha is 0, the layer is completely transparent (that is, the layer is invisible visually), and when the value of Alpha is At 100%, the layer is opaque.

Layer order: The order of layers is usually characterized by Z-order. The Z axis refers to the reference axis perpendicular to the plane direction of the display screen. Different layers determine the front and rear order of display according to the coordinates on the Z axis. This order is called Z-order. In an implementation manner, the smaller the Z-order of the layer, the higher the layer is displayed. In the application scenario where multiple windows are displayed at the same time, the Z-order of the layer corresponding to window 1 is smaller than the Z-order of the layer corresponding to window 2. When the display areas of window 1 and window 2 overlap and display, the layer of window 1 Located on the upper layer of the window 2 layer, if the window 1 layer is opaque, the window 1 layer will block the window 2 layer. In another implementation manner, the larger the Z-order of the layer, the higher the layer is displayed.

In the embodiment of the present application, the size, position and transparency of the user interface elements can refer to the related description of the size, position and transparency of the layer, and will not be repeated here.

Frame rate: the number of frame images that can be rendered per second depends on the rendering capability of the hardware of the terminal device 100 . The screen refresh rate refers to the number of times the display can be refreshed per second. When the frame rate of the terminal device 100 is higher than the screen refresh rate, it may cause an abnormal picture, and the vertical synchronization technology can make the actual frame rate not exceed the screen refresh rate. Specifically, after the vertical synchronization (Vertical Synchronization, VSYNC) signal arrives, the CPU and/or GPU draws, synthesizes and renders the display content of the user interface, and submits the rendering result to the frame buffer. When the VSYNC signal arrives next time, the terminal device 100 reads the data in the frame buffer line by line according to the VSYNC signal, and transmits the data to the display screen for display.

The application scenarios of the display method provided by the present application will be described as examples below with reference to FIG. 1A and FIG. 1B .

Exemplarily, the resolution of the display screen of the terminal device 100 shown in FIG. 1A and FIG. 1B is 4K resolution.

As shown in FIG. 1A, it is an exemplary display screen 11 provided by the embodiment of the present application. The display screen 11 may include a status bar 101 and a user interface 1 of a desktop application. The user interface 1 includes a background image 102 and a wired input source icon. 103 , a screen casting icon 104 and a setting icon 105 . in:

The status bar 101 may include: a screen projection device text 101A, a screen projection device name 101B, and one or more signal strength indicators 101C for a wireless fidelity (Wi-Fi) signal.

The wired input source icon 103 may receive an input operation (such as a touch operation, a selection instruction sent by a remote controller), and in response to the detected input operation, the terminal device 100 may display multiple selection controls for wired input sources.

The screen projection icon 104 may receive an input operation (such as a touch operation, a selection instruction sent by a remote controller), and in response to the detected input operation, the terminal device 100 may display a tutorial of the wireless screen projection function.

The setting icon 105 may receive an input operation (such as a touch operation, a selection instruction sent by a remote controller), and in response to the detected input operation, the terminal device 100 may display a screen projection setting interface.

Exemplarily, when the terminal device 100 detects an input operation acting on the display area of the wired input source icon 103, in response to the above input operation, the terminal device 100 displays on the display screen 11 the selection of the wired input source as shown in FIG. 1B Indication box 107 and switching input source interface 12, switching input source interface 12 may include switching input source text 108, input source 1 icon 109, input source 2 icon 110, input source 2 selection indicator box 111, input source 3 icon 112 and background 113.

Wherein, the selected indication box 111 is used to indicate that the input source 2 icon 110 is selected. The input source 2 icon 110 is selected, which means that the terminal device selects a USB-C connected device (for example, "HUAWEI P40") as the screen projection input. Exemplarily, the selected indication box 111 of the input source 2 may receive a user's confirmation operation (such as a touch operation, a confirmation instruction sent by a remote control), and in response to the detected confirmation operation, the terminal device 100 may display the The content of the screen projection of the device connected by USB-C (that is, the user interface 2 shown in Figure 1C), and modify the content of the screen projection device name 101B to "HUAWEI P40". It is not limited to the user interface 2, and the above screen projection content may also be other user interfaces, which are not specifically limited here.

It should be noted that FIG. 1A and FIG. 1B are exemplary user interfaces provided in the embodiments of the present application, and do not limit the embodiments of the present application. The user interfaces shown in FIG. 1A and FIG. 1B may include fewer or more controls . Optionally, the user interface shown in FIG. 1A and FIG. 1B may not include a status bar, and the screen projection device text 101A, the screen projection device name 101B and the signal strength indicator 101C are controls in the user interface 1 .

Referring to FIG. 1A and FIG. 1B, if the terminal device 100 renders the image to be displayed at 4K resolution and displays it, there will be high power consumption, the picture displayed by the terminal device 100 freezes, and it cannot quickly respond to the user's input operation to display the image shown in FIG. 1B. If the performance of the processor and/or memory of the terminal device 100 is not enough to support the smooth display of the 4K resolution interface, the above problems will seriously affect the display effect of the terminal device 100. If the terminal device 100 utilizes the existing hardware magnification scheme, that is, first renders the picture to be displayed at a low resolution (for example, 2K resolution), and then enlarges the 2K resolution image picture to 4K resolution before displaying, it will cause Visually, the image details are blurred and the display effect is not good.

However, the display method provided by the embodiment of the present application reduces the hardware performance requirements and power consumption of the processor (such as CPU and GPU), and improves the fineness of the interface display while realizing high-resolution interface display. The frame rate required for smooth display (for example, 60 frames per second (fps)) is supported, and the display interface is updated in response to the user's input operation quickly, thereby effectively improving the user's visual experience and user experience.

The specific implementation manner of the embodiment of the present application will be described in detail below.

Exemplarily, FIG. 2A is a schematic structural diagram of a terminal device 100 provided in an embodiment of the present application.

The terminal device 100 may be a terminal device equipped with iOS, Android, Microsoft or other operating systems. Exemplarily, the terminal device 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, and a projection display device , cellular phone, personal digital assistant (personal digital assistant, PDA), augmented reality (augmented reality, AR) device, virtual reality (virtual reality, VR) device, artificial intelligence (artificial intelligence, AI) device, wearable device, Vehicle-mounted devices, smart home devices (such as smart TVs), and/or smart city devices, the embodiment of the present application does not specifically limit the specific type of the electronic devices.

The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.

Wherein, some components shown in the figure (for example, the processor 110 and the internal memory 121 ) may be integrated into a system-on-chip (System-On-Chip, SOC).

It can be understood that, the structure shown in the embodiment of the present invention does not constitute a specific limitation on the terminal device 100 . In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a central processing unit (central processing unit, CPU), a graphics processor ( graphics processing unit (GPU), image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through the wireless charging coil of the terminal device 100 . While the charging management module 140 is charging the battery 142 , it can also provide power for electronic devices through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be disposed in the processor 110 . In some other embodiments, the power management module 141 and the charging management module 140 may also be set in the same device.

The wireless communication function of the terminal device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device 100 can be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the terminal device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou navigation satellite system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The terminal device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

The terminal device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.

The ISP is used for processing the data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 193 .

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The terminal device 100 may support one or more video codecs. In this way, the terminal device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the terminal device 100 can be realized through the NPU, such as: image recognition, face recognition, speech recognition, text understanding, etc.

The internal memory 121 may include one or more random access memories (random access memory, RAM) and one or more non-volatile memories (non-volatile memory, NVM).

Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (synchronous dynamic random access memory, SDRAM), double data rate synchronous Dynamic random access memory (double data rate synchronous dynamic random access memory, DDR SDRAM, such as the fifth generation DDR SDRAM is generally called DDR5SDRAM), etc.; non-volatile memory can include disk storage devices, flash memory (flash memory).

According to the operating principle, flash memory can include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. According to the potential order of storage cells, it can include single-level storage cells (single-level cell, SLC), multi-level storage cells (multi-level cell, MLC), triple-level cell (TLC), quad-level cell (QLC), etc., can include universal flash storage (English: universal flash storage, UFS) according to storage specifications , embedded multimedia memory card (embedded multi media Card, eMMC), etc.

The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of an operating system or other running programs, and can also be used to store data of users and application programs.

The non-volatile memory can also store executable programs and data of users and application programs, etc., and can be loaded into the random access memory in advance for the processor 110 to directly read and write.

The external memory interface 120 may be used to connect an external non-volatile memory, so as to expand the storage capacity of the terminal device 100 . The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and video are stored in an external non-volatile memory.

The terminal device 100 may implement an audio function through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, and an application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal.

Speaker 170A, also referred to as a "horn", is used to convert audio electrical signals into sound signals.

Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button.

The motor 191 can generate a vibrating reminder. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

In some embodiments, the internal memory 121 (such as a non-volatile memory) of the terminal device 100 is also used to store a drawn picture with a preset resolution corresponding to at least one display screen of the terminal device 100 (for ease of description, this In the embodiment of the application, this picture is referred to as the base picture for short). The display screen includes at least one user interface. When the display screen includes at least two user interfaces, any two user interfaces in the at least one user interface may have no overlap or partial overlap. The terminal device 100 divides the interface elements in the user interface into static and dynamic, and the interface elements displayed in the base map include the static interface elements in the visible area of at least one user interface. In addition, the aforementioned preset resolution is smaller than or equal to the resolution of the display screen of the terminal device 100 .

In this embodiment of the present application, static interface elements refer to interface elements whose display content is less likely to change (or immutable) when the user interface is running, such as background (background 102, background 113 as shown in FIG. 1A ), fixed text ( Text 101A) as shown in Figure 1A), icon etc. (icon 103 as shown in Figure 1A, the icon 109 shown in Figure 1B); Dynamic interface element refers to that the display content change possibility is higher when user interface runs (or can be variable) interface elements, such as a selected indicator box (selected indicator box 107 and selected indicator box 111 as shown in Figure 1B), a control with variable display icons (indicator 101C as shown in Figure 1A), variable input text (the screen projection device name 101B as shown in FIG. 1B ) and so on.

Exemplarily, as shown in FIG. 2B , in the application scenarios shown in FIG. 1A and FIG. 1B , the internal memory 121 of the terminal device 100 stores pre-drawn basemap 1 and basemap 2 with a resolution of 4K, and the basemap 1 includes visible static interface elements in the display screen 11 shown in FIG. 1A , and base map 2 includes visible static interface elements in the display screen shown in FIG. 1B . Optionally, as shown in FIG. 2B , the internal memory 121 also stores pictures of pre-drawn dynamic interface elements (such as an indicator 101C and a selected indicator box 107 ). Through the display method provided by the embodiment of the present application, before the terminal device 100 displays the display screen 11 shown in FIG. rate display screen 11 and display. In response to the input operation on the display area of the wired input source icon 103, the terminal device 100 acquires the base map 2 with 4K resolution from the internal memory 121, and then synthesizes the base map 2 with 4K resolution as shown in FIG. 1B based on the base map 2 with 4K resolution. display screen.

The software system architecture of the terminal device 100 involved in the embodiment of the present application is illustrated below by way of example.

The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is taken as an example to illustrate the software structure of the terminal device 100 .

Referring to FIG. 3 , FIG. 3 exemplarily shows a software structural block diagram of the terminal device 100 provided by the embodiment of the present application.

As shown in Figure 3, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system can be divided into an application program layer, an application program framework layer, a hardware abstraction layer (hardware abstraction layer, HAL) layer and a kernel layer (kernel) from top to bottom. in:

The application layer consists of a series of application packages. As shown in FIG. 3 , the application package may include applications such as desktop applications, video applications, and screen-casting applications. It can also include applications such as camera, gallery, call, music, etc., which will not be described here.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. The application framework layer can include window management module, application management module, input management module, media framework and graphics framework.

The application management module can be used to obtain application windows running in the foreground and running in the background. In some embodiments, the application management module includes Activity Manager Service (ActivityManagerService, AMS), and AMS is a system service of the Android (android) system, which is used to control the reasonable scheduling and operation of different application programs in the system. The stack is managed. The main responsibilities of AMS include uniformly scheduling the activities of various applications, managing application processes, managing memory, scheduling services, scheduling tasks, querying the current running status of the system, and so on. Activity is an interface for interaction between users and applications. Each Activity component is associated with a Window object, which is used to describe a specific application window. The application management module can determine the application windows running in the foreground and the background by calling the AMS.

The input management module is used to manage the input events and event distribution of the whole system. In some embodiments, the input management module includes an input management service (InputManagerService, IMS), and the IMS is used for input event management, that is, to monitor all input device nodes, and when the input device node has data, process the data and find A suitable application window distributes input events to the window through the window management module. The IMS can determine the activity running in the foreground by calling the AMS, and then combine the interface layout of the application window corresponding to the activity running in the foreground to determine the specified application window of the specified application corresponding to the above input event.

The window management module is used to manage application windows. The window management module can obtain the resolution of the display screen, judge whether there is a status bar, and so on. The above-mentioned application program window may be an application program window in the Android system, an application program window in the IOS system, or an application program window in other operating systems, which is not specifically limited here. In some embodiments, the window management module may include a window management service (Window Manager Service, WMS), and WMS is a system service of the android system, which controls the creation, display and hiding of all windows. WMS assigns layers (Surface) to all windows, manages the display order, size, position, etc. of layers, and controls window animation. It is an important transfer station for the input system.

After the window management module receives the input event distributed by the input management module, it sends an instruction to the designated application corresponding to the input event to instruct the application to draw the user interface of the designated application window corresponding to the input event. The application layer can draw and render the layer corresponding to the above user interface according to the instruction sent by the window management module, and call the system process to draw and render the layer corresponding to the status bar and the navigation bar.

The media framework includes a media player, video decoder, image player and image decoder, and the graphic framework includes an interface composition module.

After the application layer obtains the video data, it can call the video decoder through the media player to decode the above video data. After the application layer obtains the image data, it can call the image decoder through the media player to decode the above image data. It should be noted that the video data and image data obtained by the application layer are usually compressed data.

The interface composition module is used for synthesizing at least one layer into a user interface image. In one implementation, the above-mentioned interface synthesis module is SurfaceFlinger, and SurfaceFlinger determines the display area and display content of each layer in the composite image based on parameters such as the size, position, Z-order, and transparency of each layer, and then SurfaceFlinger calls the hardware compositor (Hardware Composer, HWC) to generate the display buffer data (Buffer) of the composite image based on the display area and display content of each layer in the composite image. It is not limited to realizing the composition of the user interface image by calling a hardware compose (Hardware Compose, HWC). In the embodiment of the present application, the composition of the user interface image may also be realized by calling the GPU.

Android Runtime includes core library and virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

The core library consists of two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

Hardware Abstraction Layer (Hardware Abstraction Layer, HAL) is an interface layer between the kernel and the hardware circuit, providing a virtual hardware platform for the software system shown in Figure 3. The HAL layer can include: video processing module (Video Process, VP) HAL, HWC HAL and frame buffer (FrameBuffer, FB) HAL.

The kernel layer is the layer between hardware and software. The kernel layer can include FB driver, video object (Video Output, VO) driver and input system. It can also include sensor driver, touch IC driver and audio driver, etc. The HAL layer and the kernel layer (kernel) can perform corresponding operations in response to functions called by the application framework layer.

In the embodiment of this application, the FB driver can send the display buffer data of the user interface image to the VO driver, and the VO driver can call the image amplifier and/or synthesizer to synthesize the above-mentioned user interface image and the image output by the media framework to generate a display screen, and the screen to be displayed can be sent to the display screen 194 for display.

In some embodiments, the kernel layer includes display drivers. The image output by the media framework can be sent to the graphics framework, and the interface synthesis module of the graphics framework synthesizes the image output by the media framework and the layer output by the application layer into a picture to be displayed; then, the interface synthesis module calls the display driver through the hardware abstraction layer Send the user interface image to the display for display.

In some embodiments, the application framework layer may also include a basemap storage module. The terminal device 100 draws the base map of each display screen in advance, and stores it in the base map storage module. Exemplarily, the base map storage module stores the pre-drawn base map 1 and base map 2 shown in FIG. 2B, the base map of the display screen of the terminal device 100 shown in FIG. The bottom image of the display screen of the terminal device 100 is the bottom image 2 .

In some embodiments, the application layer draws and renders the user interface in response to the instruction sent by the window management module, and only needs to draw and render the dynamic interface elements in the user interface when generating the layer corresponding to the user interface. The application layer can determine the base map 3 of the picture to be displayed according to the instruction sent by the window management module, and then obtain the stored data of the base map 3 from the base map storage module and send it to the image processor; the image processor calls the image decoder based on The above stored data decodes the base image 3, and sends the base image 3 to the VO driver through the VO HAL. The VO driver can invoke the synthesizer to synthesize the user interface image containing only dynamic interface elements and the base map 3 to generate a picture to be displayed. For ease of description, in subsequent embodiments, the above-mentioned to-be-displayed screen may be referred to as the first display screen for short.

In the embodiment of the present application, the image format of the base map stored by the base map storage module may be a compressed image format (such as JPEG format). The image decoder can decode the basemap into YUV format. YUV format refers to the pixel format in which brightness parameters and chrominance parameters are expressed separately. "Y" means brightness (Luminance or Luma), that is, gray value, "U" and "V" means chroma (Chrominance or Chroma). Storing images in YUV format generally takes up less storage space than RGBA format.

In one implementation, the user interface image drawn by the terminal device 100 is in RGBA format, and the synthesizer provided in the embodiment of the present application can support the synthesis of the user interface image in RGBA format output by the software system and the video image in YUV format, and The composite image is sent to the display screen 194 for display.

Combining the above hardware structure and software system architecture, the specific implementation of the display method provided by the embodiment of the present application will be introduced below.

FIG. 4 is a schematic diagram of an implementation of a display method provided by an embodiment of the present application.

As shown in Figure 4, in response to the instructions sent by the window management module, the application layer draws and renders the layers corresponding to the user interface running in the foreground, and sends the layer data of the above layers to the interface synthesis module of the graphics framework; the interface The synthesis module synthesizes the user interface image of resolution 1 based on the layer data of at least one layer sent by the application layer, and sends the image cache data of the user interface image of resolution 1 to the image amplifier; the image amplifier converts the resolution 1 The UI image is upscaled to a resolution 2 UI image, and the resolution 2 UI image is sent to the compositor. In response to the instruction sent by the window management module, the application layer also obtains the video source with resolution 2, and sends each frame of video image in the video source with resolution 2 to the video decoder of the media framework; the video decoder decodes based on the above video data The dynamic video image of resolution 2 is decoded, and the dynamic video image of resolution 2 is sent to the synthesizer. The synthesizer generates a synthesized image based on the user interface image with resolution 2 and the dynamic video image with resolution 2, and sends the synthesized image to the display screen 194 for display.

It can be understood that the resolution 2 is greater than the resolution 1, for example, the resolution 1 is a 2K resolution, and the resolution 2 is a 4K resolution.

The display method provided by the embodiment of the present application supports the independent output of user interface and video. The terminal device 100 draws a low-resolution (such as resolution 1) user interface image and enlarges it to a high-resolution (such as resolution 2) The user interface image is displayed after decoding the high-resolution (eg resolution 2) video source into a high-resolution video image and then displayed. In this way, for a terminal device 100 with a high-resolution display screen, insufficient processor and/or memory performance, and strong video decoding capabilities, the display method provided by the embodiment of the present application can realize high-resolution screen display, Solve the problem of insufficient performance of hardware (processor and/or memory, etc.) in high-resolution video playback scenarios in the current industry.

However, in the above display method, the high-resolution user interface image displayed by the terminal device 100 is obtained by enlarging the low-resolution user interface image, and there is a problem that the details of the user interface part in the display screen are blurred and the visual effect is not good.

The embodiment of the present application also provides another display method. In the method, the terminal device 100 can divide the display screen into a base map of the static part and a user interface image of the dynamic part. Static interface elements, the dynamic part of the user interface image only includes dynamic interface elements in all user interfaces in the display screen, the terminal device 100 stores a pre-drawn high-resolution base map; the terminal device 100 draws a low-resolution base map that only contains The user interface image of the dynamic interface element is then sent to the display screen 194 to display the composite image of the low-resolution user interface image and the high-resolution base map. Implementing the display method provided by the embodiment of the present application can realize high-resolution interface display. Since the embodiments of the present application have relatively low hardware performance requirements for processors (such as GPU, CPU) and/or memory, the embodiments of the present application can support the frame rate required for smooth display while realizing high-resolution interface display. And update the display interface quickly in response to the user's input operation. In addition, the use of pre-drawn basemaps also effectively reduces the power consumption of high-resolution interface displays. Regarding the above display method, the embodiment of the present application provides two implementation schemes (ie, the following scheme 1 and scheme 2), and the above two implementation schemes are described in detail below.

Exemplarily, FIG. 5A is a schematic diagram of internal implementation of solution 1 provided in the embodiment of the present application, and FIG. 5B is a schematic flow diagram of the method of solution 1 provided in the embodiment of the present application. The method includes but not limited to steps S101 to S111.

S101. In response to the instruction sent by the window management module, based on the interface layout of the first user interface in the first display screen, the application layer determines the position and size of the layer of the first user interface in the user interface image with resolution 1. , and the position and size of the first dynamic interface element in the layer of the first user interface, and then draw and render a layer of size 1 corresponding to the dynamic part of the first user interface.

Wherein, the first user interface is any user interface in the above user interface images, and the first dynamic interface element is any dynamic interface element in the first user interface.

It should be noted that the terminal device 100 stores the interface layout of the first user interface, and the interface layout information of the first user interface includes the layer size of the first user interface in the user interface image with resolution 1 (for ease of description, Referred to as size 1), position (referred to as position 1 for convenience of description), transparency and z-order, and the position of the first dynamic interface element of the first user interface in the above layers (referred to as position for convenience of description 2), size (for convenience of description, referred to as size 2 for short), transparency and other information. When the terminal device 100 draws the layer of the first user interface, based on the interface layout information of the user interface, the position and size of each dynamic interface element of the first user interface in the layer can be determined, and then the dynamic interface elements of the first user interface can be drawn on the canvas of size 1. For each dynamic interface element of the first user interface, a layer of size 1 corresponding to the dynamic part of the first user interface is generated.

Exemplarily, the following uses the application scenarios shown in FIG. 1A and FIG. 1B for exemplary description.

The display screen 194 of the terminal device 100 displays the display screen 11 shown in FIG. Touch time and other information) is sent to the input system, and the input system processes the above-mentioned input operation 1 into an original input event, and the original input event is stored in the kernel layer. The input management module obtains the original input event from the kernel layer, and the input management module calls the application management module to determine the user interface of the application program window running in the foreground and the interface layout information of the user interface (such as the user interface 1 of the desktop application), combined with the above interface layout information It is determined that the above original input event is a click event acting on the display area of the wired input source icon 103 . The event management module sends the input event to the window management module. The window management module sends an instruction 1 to the application layer, instructing the application layer to update the display screen to the display screen shown in FIG. 1B . The application layer determines that the base map of the updated display screen is base map 2, and the user interface image corresponding to the dynamic interface elements in the updated display screen, and calls the graphics framework to draw the above user interface image. The display screen shown in FIG. 1B includes a status bar 101 , a desktop application user interface 1 and an input source switching interface 12 .

Optionally, the static interface elements of the status bar 101 may include text 101A, and the dynamic interface elements of the status bar 101 may include a screen projection device name 101B and an indicator 101C. The static interface elements of the user interface 1 may include a background 102, a wired input source icon 103, a screen projection icon 104, and a setting icon 105, and the dynamic interface elements of the user interface 1 may include a selection indicator box 107 for a wired input source, and an unselected screen projection icon. The indication box 201 and the unselected indication box 202 of the setting. The static interface elements of the switching input source interface 12 may include text 108, the input source 1 icon 109, the input source 2 icon 110 and the input source 3 icon 112, and the dynamic interface elements of the switching input source interface 12 may include an unselected indication of the input source 1 Box 203 , the selected indication box 111 of input source 1 and the unselected indication box 204 of input source 1 . It should be noted that, in one implementation, the selected indicator box 107 and the selected indicator box 111 are visible, while the unselected indicator box 201, the unselected indicator box 202, the unselected indicator box 203, and the unselected indicator box 204 are visible. Invisible.

In response to the instruction 1 sent by the window management module, the application layer calls the system process to draw the layer 1 corresponding to the dynamic part of the status bar, and the screen projection application in the application layer responds to the above instruction 1, and draws the user interface 1 of the screen projection application The layer 2 corresponding to the dynamic part of , and the layer 3 corresponding to the dynamic part of the switching input source interface 12 .

In an implementation manner, the size and position of layer 1, layer 2 and layer 3 are the same, and the size is equal to the size corresponding to the resolution of the user interface image to be synthesized in the interface synthesis module. Exemplarily, the resolution of the user interface image to be synthesized is resolution 1, the sizes of layer 1, layer 2 and layer 3 shown in FIG. 6A are all equal to the size corresponding to resolution 1, and layers 1, The positions of layer 2 and layer 3 in the user interface image to be synthesized (for example, the position of the upper left corner) are both (0, 0). In another implementation manner, the sizes and positions of layer 1, layer 2 and layer 3 may be different. Exemplarily, the resolution of the user interface image to be synthesized is resolution 1, and the sizes of layer 1, layer 2, and layer 3 shown in FIG. 6B are different, and they are all smaller than or equal to the size corresponding to resolution 1. Wherein, the position of layer 2 in the user interface image to be combined is (0, 0), and the positions of layer 1 and layer 3 in the user interface image to be combined are not (0, 0).

As shown in FIG. 6A and FIG. 6B, the dotted interface elements (for example, the unselected indicator box 201 and the unselected indicator box 203) in layer 2 and layer 3 represent that the interface element is invisible in the screen displayed by the terminal device 100. , when the terminal device 100 draws the layer 2 and the layer 3, it may not draw the dotted line interface element, or it may draw the interface element, but set the transparency of the interface element to 100%.

In an implementation manner, when the terminal device displays the display screen 1 shown in FIG. 1A , layer 1 and layer 2 have already been drawn, and image cache data of layer 1 and layer 2 have been stored. In response to the user's input operation, when the terminal device 100 superimposes and displays the switching input source interface 12 on the display screen shown in FIG. 1A , in response to the above instruction 1, the application layer only needs to draw the layer 3 corresponding to the switching input source interface 12 , and can directly obtain the image cache data of layer 1 and layer 2 that have been drawn.

S102. The application layer sends the layer data of the layers corresponding to the dynamic parts of the user interfaces in the first display screen to the interface synthesis module of the graphics framework.

S103. The interface composition module synthesizes the user interface image 1 with resolution 1 based on the layer data of at least one layer sent by the application layer.

It can be understood that the above user interface image 1 only includes dynamic interface elements.

In some embodiments, the layer data of a layer may include information such as the size, position, transparency, and Z-order of the layer. When the compositor receives the layer data of at least one layer, the compositor will composite the above-mentioned at least two layers on the canvas with resolution 1 based on the size, position, transparency and Z-order of the above-mentioned at least one layer UI image 1 for resolution 1.

Exemplarily, the application layer sends the layer data of layer 1, layer 2 and layer 3 to the interface synthesis module of the graphics framework. As shown in Figure 6A and Figure 6B, the interface composition module can determine the display area of each layer on the canvas with resolution 1 based on the size and position of the above-mentioned layer 1, layer 2, and layer 3, and The Z-order of the layer Composite the above layer 1, layer 2 and layer 3 on the canvas of resolution 1 in Z-order order to form user interface image 1 of resolution 1.

In the embodiment of the present application, the transparency of areas other than the effective display area of the dynamic interface elements in the layer corresponding to the dynamic part of each user interface is set to be completely transparent or a preset transparency.

S104. The interface synthesis module sends the image cache data of the user interface image 1 with resolution 1 to the image amplifier.

S105 , the image amplifier enlarges the user interface image 1 with resolution 1 into user interface image 1 with resolution 2, where resolution 2 is greater than resolution 1.

Wherein, the ratio of the number of horizontal pixels between the resolution 2 and the resolution 1 is A, and the ratio of the number of vertical pixels between the resolution 2 and the resolution 1 is B. The resolution 1 is smaller than the resolution of the display screen 194, and the resolution 2 may be equal to or smaller than the resolution of the display screen 194, which is not specifically limited here.

In one implementation manner, A is equal to B, and the image amplifier scales up the user interface image 1 with resolution 1 to the user interface image 1 with resolution 2 in equal proportion. Exemplarily, as shown in FIG. 6C , the user interface image 1 with resolution 1 is scaled up to the user interface image 1 with resolution 2 in equal proportions.

S106. The image amplifier sends the image cache data of the user interface image 1 with resolution 2 to the synthesizer.

S107. In response to the instruction 1 sent by the window management module, the application layer further determines that the base image of the first display screen is the base image 3, and acquires the stored data of the base image 3 with a resolution of 2 from the base image storage module.

S108. The application layer sends the stored data of the base image 3 to the image decoder of the media framework.

It should be noted that, in the embodiment of the present application, the implementation order of steps S108 and S101 is not specifically limited.

S109. The image decoder decodes the base image 3 with resolution 2 based on the above stored data.

S110. The image decoder sends the image buffer data of the base map 3 with resolution 2 to the compositor.

S111. The synthesizer generates a composite image based on the above-mentioned user interface image 1 with resolution 2 and the above-mentioned base image 3 with resolution 2, and sends the above-mentioned composite image to the display screen 194. The user interface image 1 is located on the upper layer of the base image 3.

Specifically, the synthesizer determines that the original position and target position of the base image 3 are both (0,0) when the above-mentioned user interface image 1 with resolution 2 is synthesized, and the original size and target size of the synthesized base image 3 are both corresponding to resolution 1. Size; the compositor composites the UI image 1 on top of the basemap 3 based on the original position, target position, original size, and target size above.

It should be noted that when the compositor synthesizes images, the user interface image 1 is located on the upper layer of the base image 3, since the transparency of the areas in the user interface image other than the effective display area of the dynamic interface elements are set to be completely transparent or preset transparency , in the image synthesized by the compositor, the area other than the effective display area of the dynamic interface element can display the display content of the base map.

Exemplarily, as shown in FIG. 6D , the synthesizer synthesizes the user interface image 1 of resolution 2 and the base image 3 of resolution 2 into the display picture shown in FIG. 1B , and sends the display picture to the display screen 194 for further processing. show. Referring to FIG. 2B , the base map 3 may be the base map 2 shown in FIG. 2B stored in the internal memory.

Exemplarily, FIG. 7A is a schematic diagram of internal implementation of solution 2 provided in the embodiment of the present application, and FIG. 7B is a schematic flow diagram of the method of solution 2 provided in the embodiment of the present application. The method includes but not limited to steps S201 to S209.

S201. In response to the instruction 1 sent by the window management module, the application layer determines, based on the interface layout of the first user interface in the first display screen, in the user interface image with resolution 2, the first user interface in the layer. Position 3 and size 3 of a dynamic interface element, and then draw and render the layer corresponding to the dynamic part of the first user interface based on the size 3 of the first dynamic interface element, position 3 is the first dynamic interface element between the user interface image and the second -Displays the target position in the screen.

Wherein, the first user interface is any user interface in the above user interface images, and the first dynamic interface element is any dynamic interface element in the first user interface. The layer size of the first user interface is smaller than the size corresponding to resolution 2.

Different from the display method shown in FIG. 5B , in the embodiment of the present application, when drawing the layer corresponding to the dynamic part of the first user interface, the terminal device 100 first determines the layer at resolution 2 based on the interface layout information of the first user interface. The size of each dynamic interface element of the first user interface in the user interface image. For ease of description, in this embodiment of the present application, the position of the first dynamic interface element of the first user interface in the user interface image with resolution 2 may be referred to as position 3 for short, and the position of the first dynamic interface element of the first user interface in the resolution The dimensions in UI images of rate 2 are simply referred to as dimension 3. Then, the application layer can draw the first dynamic interface element of size 3 on the canvas with the specified resolution (or specified size), so that the layer of the specified resolution (or specified size) of the first user interface can be obtained. Different from scheme 1 shown in FIG. 5B , scheme 2 does not specifically limit the positions of the dynamic interface elements in the layer, as long as the dynamic interface elements do not overlap in the layer. For ease of description, in this embodiment of the present application, the actual position of the first dynamic interface element of the first user interface in the layer with the specified resolution may be referred to as position 4 for short.

In an implementation manner, the dynamic interface elements are arranged in the upper left corner of the layer in order from left to right without overlapping.

It should be noted that, in the embodiment of the present application, the target display resolution of the terminal device 100 is resolution 2, therefore, the position 3 of the first dynamic interface element of the first user interface in the resolution 2 layer is the second A target position of a dynamic interface element in the first display frame.

In an implementation manner, the specified resolutions of the layers corresponding to the dynamic parts of the user interfaces in the first display screen are the resolutions of the user interface images to be synthesized, that is, resolution 3. Resolution 3 may be a resolution preset by the terminal device 100 that is smaller than resolution 2; or, resolution 3 may be based on the size of all dynamic interface elements in the first display screen of the terminal device 100 (for example, the size of the first dynamic interface element 3) It is determined that the layer corresponding to each user interface can accommodate all the dynamic interface elements of the user interface. In this way, the terminal device 100 can draw the first dynamic interface element of size 3 corresponding to the high-resolution (for example, resolution 2) user interface image on the low-resolution (for example, resolution 3) canvas.

In response to the instruction 1 sent by the window management module, the application layer determines the size of all dynamic interface elements (such as the size 3 of the first dynamic interface element) in the status bar, user interface 1 and switching input source interface 12, and then based on all dynamic interface The size of the element determines the resolution 3; then call the system process to draw the dynamic interface elements of the status bar on the canvas of the resolution 3, and generate the layer 4 of the corresponding resolution 3 of the status bar; similarly, the projection in the application layer In response to the above instruction 1, the screen application draws a layer 5 of resolution 3 corresponding to the dynamic part of the user interface 1 of the screencasting application, and a layer 6 of resolution 3 corresponding to the dynamic part of the switching input source interface 12 .

Exemplarily, the user interface image to be synthesized has a resolution of 3, and the resolutions of layer 4, layer 5, and layer 6 shown in FIG. 8A are all resolution 3. Taking layer 4 and the aforementioned layer 1 as an example, comparing Figure 8A and Figure 6A, we can see that the size of the dynamic interface element in layer 4 (for example, the size 3 of the first dynamic interface element) is larger than the corresponding dynamic interface in layer 1 The size of the element (such as the size 2 of the first dynamic interface element), the element position of each dynamic interface element in layer 4 is not specifically limited (for example, each dynamic interface element in layer 4 can be pressed in the upper left corner of the layer There is no overlapping distribution from left to right), and the position of each dynamic interface element in layer 1 is determined based on the interface layout information of the first user interface.

In an implementation manner, the resolutions of the layers corresponding to the dynamic parts of the user interfaces in the first display screen may be different, and the layers corresponding to the dynamic parts of the user interfaces can accommodate all the dynamic interface elements in the user interface (such as The first dynamic interface element of size 3) is enough. For example, the resolution of the layers of the dynamic part of the first user interface may be preset by the terminal device 100, or may be determined by the terminal device 100 based on the size of the dynamic interface elements in the first user interface (such as the size of the first dynamic interface element). 3) OK. Exemplarily, the user interface image to be synthesized has a resolution of 3, and the resolutions of layer 4, layer 5, and layer 6 shown in FIG. 8B are all smaller than or equal to resolution 3.

Similar to solution 1, the dotted interface elements in layers 5 and 6 indicate that the interface elements are invisible in the screen displayed by the terminal device 100, and the terminal device 100 may not draw the above-mentioned interface elements when drawing layers 5 and 6. The dotted interface element can also be drawn, but the transparency of the interface element is set to 100%.

S202. The application layer sends the layer data of the layers corresponding to the dynamic parts of the user interfaces in the first display screen to the interface composition module of the graphics framework, and the layer data corresponding to the first user interface includes the first dynamic interface elements in Position 4 and size 3 in the layer, and the target position of the first dynamic interface element (that is, position 3).

Exemplarily, the application layer sends the layer data of layer 4, layer 5 and layer 6 to the interface synthesis module of the graphics framework.

S203. After receiving the layer data of at least one layer sent by the application layer, the interface synthesis module synthesizes the at least one layer into a user interface image based on the position 4 and size 3 of the first dynamic interface element in the layer 2. The actual position of each dynamic interface element in the above user interface image 2 is position 3, and each dynamic interface element in the above user interface image 2 has no overlap.

In some embodiments, after receiving the layer data of at least one layer sent by the application program layer, the interface composition module based on the size 3 and position of the first dynamic interface element in the layer of the first user interface 4. Determine the position 5 of the first dynamic interface element in the user interface image 2; then based on the size and position of each dynamic interface element in the user interface image 2 (such as the size 3 and position 5 of the first dynamic interface element), the All the dynamic interface elements of each layer are synthesized into the canvas with resolution 3 to generate user interface image 2 with resolution 3. In the embodiment of the present application, the position of each dynamic interface element in the user interface image 2 is not specifically limited, as long as the dynamic interface elements do not overlap.

In one implementation, the dynamic interface elements in each layer are distributed from left to right in the canvas with a resolution of 3, and then the dynamic interface elements of each layer are arranged in units of layers at a resolution of 3 3 in the canvas from top to bottom.

Exemplarily, as shown in FIG. 8A and FIG. 8B , the interface synthesis module synthesizes a user interface image 2 with a resolution of 3 based on the layer data of the above-mentioned layers 3 , 4 , and 5 . The dynamic interface elements of layer 4 (or layer 5, layer 6) in user interface image 2 are distributed in order from left to right in user interface image 2, and the dynamic interface elements of each layer in user interface image 2 are distributed in the user interface image 2. The interface image 2 is distributed in the order of layer 4, layer 5 and layer 6 from top to bottom.

S204. The interface synthesis module sends the image buffer data of the user interface image 2 with a resolution of 3 to the synthesizer. The image buffer data includes the size 3 and position 5 of the first dynamic interface element in the user interface image 2, and the first dynamic interface element The target position of the interface element (ie position 3).

S205. In response to the instruction 1 sent by the window management module, the application layer further determines that the base image of the first display screen is base image 3, and acquires the stored data of base image 3 with resolution 2 from the base image storage module.

It should be noted that, in the embodiment of the present application, the implementation order of steps S205 and S201 is not specifically limited.

S206. The application layer sends the stored data of the base image 3 to the image decoder of the media framework.

S207. The image decoder decodes the base image 3 with resolution 2 based on the above stored data.

S208. The image decoder sends the image buffer data of the base image 3 with the resolution 2 to the compositor.

S209. The synthesizer synthesizes the image of the display area of each dynamic interface element in the user interface image 2 with a resolution of 3 and the image of the target area of the interface element in the base map 3 into a new image of the target area, thereby generating a resolution of 2 The synthesized image (that is, the first display picture) of the above-mentioned synthesized image is sent to the display screen 194 for display.

Specifically, the synthesizer determines the display area of the first dynamic interface element in the user interface image 2 based on the position 5 and size 3 of the first dynamic interface element in the user interface image 2 with a resolution of 3, and obtains the display area of the display area. Image cache data 1; the synthesizer also determines the target area of the first dynamic interface element in the base image 3 based on the target position (ie position 3) and size 3 of the first dynamic interface element in the base image, and obtains the target area The image buffer data 2 of the above image buffer data 2; then the above image buffer data 1 and image buffer data 2 are synthesized into the new image buffer data of the target area. In this way, after the base image 3 is synthesized with all the dynamic interface elements in the user interface image 2, a synthesized image to be displayed is generated.

Exemplarily, as shown in FIG. 8C , the synthesizer synthesizes the user interface image 2 of resolution 3 and the base image 3 of resolution 2 into the display picture shown in FIG. 1B , and sends the display picture to the display screen 194 for further processing. show. Specifically, taking the selected indicator box 111 of the input source 2 as an example, the synthesizer determines the area 1 corresponding to the selected indicator box 111 based on the position 5 and size 3 of the button 111, and obtains the image cache data of the selected indicator box 111 in area 1; The combiner is also based on the position 3 and size 3 of the selected indicator box 111, determines that the target area of the selected indicator box in the base map is area 2, and obtains the image cache data of area 2, area 2 includes the input source 2 icon 110; and then The image buffer data of area 1 and area 2 are synthesized into the image buffer data of the combined image of area 2 , and the combined image of area 2 includes the input source 2 icon 110 and the selected indication box 111 . In this way, the display screen 194 can display the input source 2 icon 110 and the selected indication box 111 in the area 2 according to the synthesized image buffer data in the area 2 .

In some embodiments, in solution 2, the dynamic interface elements of the first user interface can also be formed into a composite control, which is processed as a whole dynamic interface element. For the convenience of description, they are collectively referred to as composite dynamic interface elements hereinafter. In an implementation manner, the terminal device 100 may store interface layout information related to the composite dynamic interface element, and the terminal device 100 may directly acquire the position and size of the composite dynamic interface element. In one implementation, the terminal device 100 does not store the interface layout information about the compound dynamic interface element, and the terminal device 100 may indirectly acquire the position of the compound dynamic interface element based on the position and size of each dynamic interface element included in the compound dynamic interface element and size.

The following takes the switching of the input source interface 12 as an example for specific description.

Exemplarily, with reference to FIG. 9A , the dynamic interface elements in the switching input source interface 12 (that is, the unselected indicator box 203, the selected indicator box 111 and the unselected indicator box 204) form a composite dynamic interface element 301, and the switching input source interface 12 The interface layout information does not include the position and size of the compound dynamic interface element 301, and the terminal device 100 can determine that the position of the compound dynamic interface element 301 is the upper left corner position of the upper left corner interface element in the compound dynamic interface element 301, that is, the indicator box 203 is not selected The upper left corner position (a1, b1); and based on the upper left corner position (a1, b1) and the lower right corner position of the lower right corner interface element in the composite dynamic interface element 301 (that is, the lower right corner position (a2, b2)) Determine the size of the compound dynamic interface element 301, that is, (a2-a1)×(b2-b1). It can be understood that the relative positional relationship of each dynamic interface element in the composite dynamic interface element 301 is determined based on the interface layout information of the switching input source interface 12 .

Specifically, in step 201, when the terminal device 100 draws the layer 3 corresponding to the dynamic part of the switching input source interface 12, first, based on the interface layout information of the switching input source interface 12, it is determined that the composite dynamic part in the user interface image with resolution 2 Position 3 and size 3 of interface element 301 . Then, the application layer can draw the composite dynamic interface element 301 of size 3 on the canvas with the specified resolution, so that the layer 3 with the specified resolution of the switching input source interface 12 can be obtained. The actual position of the compound dynamic interface element 301 in layer 3 is the position 4 of the compound dynamic interface element 301 .

Exemplarily, referring to FIG. 9B , layers 1 , 2 , and 3 have the same resolution, and the dynamic part of the switching input source interface 12 is drawn on the upper left corner of layer 3 as a composite dynamic interface element 301 .

Exemplarily, referring to FIG. 9C , the resolutions of layer 1, layer 2, and layer 3 can be different, and the dynamic part of the switching input source interface 12 is drawn in layer 3 as a composite dynamic interface element 301, and the dynamic part of layer 3 The size may be equal to the size of the composite dynamic interface element 301 .

In step 202, after the interface composition module receives the layer data of layer 3 sent by the application program layer, based on the position 4 and the size 3 of the composite dynamic interface element 301 in layer 3, it is determined that the composite dynamic interface element 301 is in the user interface. Position 5 in image 2; furthermore, based on the position 5 and size 3 of the compound dynamic interface element 301 in the user interface image 2, the above compound dynamic interface element 301 is integrated into the canvas with resolution 3 to generate a user interface with resolution 3 image 2. For example, reference may be made to the synthesis of user interface images shown in FIG. 9B and FIG. 9C .

Referring to Fig. 9B and Fig. 9C, compared with the implementation of user interface image synthesis provided by the relevant embodiments in Fig. 8A and Fig. 8B, the composition granularity of this embodiment can be larger, and the layer 3 shown in Fig. Dynamic interface elements are synthesized at a granularity, while layer 3 shown in FIG. 9B and FIG. 9C is synthesized at a granularity of composite dynamic interface elements.

It should be noted that, Fig. 9A to Fig. 9C are illustrated by switching the input source interface 12 as an example, which does not constitute a limitation to the embodiment of this application, and other user interfaces in the screen to be displayed can also draw layers with the granularity of composite dynamic interface elements and compositing user interface images.

In some embodiments, in the application scenarios shown in FIG. 1A and FIG. 1B , when the terminal device 100 detects an input operation acting on the display area of the wired input source icon 103, in response to the above input operation, the terminal device 100 displays the 11 displays a wired input source button 107 and an input source switching interface 12 as shown in FIG. 1B . In some embodiments, in response to the above-mentioned input operation, the base map in the display screen remains unchanged, that is, the base map in the display screen in FIG. 1B is the same as the base map in the display screen in FIG. All the interface elements of the newly added switching input source interface 12 in the display screen are combined into a compound control, that is, the switching input source interface 12 as a whole is processed as a composite dynamic interface element 302 . FIG. 9D exemplarily shows the composite dynamic interface element 302. In addition to the button 203, the button 111 and the button 204, the composite dynamic interface element 302 also includes the text 108 for switching the input source, the input source 1 icon 109, the input source 2 icon 110, Input source 3 icon 112 and background 113 . For the drawing and composition of the layer 3 corresponding to the switching input source interface 12, reference may be made to the descriptions of the related embodiments in FIG. 9A to FIG. 9C , which will not be repeated here.

In the scheme 1 and scheme 2 proposed in the embodiment of this application, the terminal device 100 converts the static part of the user interface into a base map, draws the high-resolution base map in advance and stores it, and the terminal device 100 only needs to draw the low-resolution user interface The dynamic part of the interface. In this way, the load and performance requirements on hardware such as CPU, GPU and memory for high-resolution interface refresh are greatly reduced.

In the embodiment of this application, in scheme 1 and scheme 2, in response to the above instruction 1, the application layer can draw the low-resolution layer corresponding to the dynamic part of the user interface based on the preset frame rate 1, and based on the preset The frame rate of 2 obtains the base map required for the image to be displayed from the base image storage module; then the compositor can synthesize a high-resolution display image based on frame rate 1. Wherein, frame rate 2 is smaller than frame rate 1. In this way, the application layer obtains the basemap from the basemap storage module at a low frame rate, which can avoid the power consumption and load caused by frequent image decoding, and in turn can further increase the screen refresh rate; on this basis, the application layer high frame rate The layer corresponding to the dynamic part is drawn efficiently, and the combiner synthesizes the display picture at a high frame rate, which can ensure that the terminal device 100 can quickly respond to the user's interactive operation to update the interface.

The embodiment of the present application provides the following display method, the display method is applied to the terminal device 100, and the display method includes but not limited to steps S301 to S303. in,

S301. In response to the first instruction, draw the first user interface image of the first resolution corresponding to the first display screen, and acquire the first base image of the second resolution of the first display screen; wherein, the first instruction is used to Instructing the terminal device to draw a first display screen with a second resolution, the first user interface image includes dynamic interface elements in the first display screen, the first base image includes static interface elements in the first display screen, and the terminal device stores the The base map of at least one display screen is drawn, and the second resolution is greater than the first resolution;

S302. Based on the first user interface image of the first resolution and the first base image of the second resolution, synthesize the first display image of the second resolution;

S303. Display the first display image through the display screen.

In some embodiments, the above-mentioned first display screen includes at least one user interface, and the above-mentioned drawing of the first user interface image with the first resolution corresponding to the first display screen includes: drawing the first user interface image corresponding to the dynamic interface elements of the first user interface A layer, the first user interface is any user interface in the at least one user interface, the resolution of the first layer is less than or equal to the first resolution; the dynamic interface of each user interface in the first display screen The layers corresponding to the elements are synthesized into the first user interface image with the first resolution.

In this embodiment of the present application, for example, the first instruction may be the aforementioned instruction 1. Referring to FIG. 1A , the first instruction may be an input operation acting on the wired input source icon 103 shown in FIG. 1A . The first resolution may be the aforementioned resolution 1, the second resolution may be the aforementioned resolution 2, the first display screen may be the display screen of resolution 2 shown in FIG. For the user interface image 1 of resolution 1 shown in FIG. 6C , the first base image may be base image 3 of resolution 2 shown in FIG. 6D .

In the embodiment of the present application, for example, the first user interface may be the status bar 101, the user interface 1 or the switching input source interface 12 described in FIG. 1B and FIG. 6D, and the first layer may be the dynamic interface of the status bar 101 The layer 1 corresponding to the element, the first layer may be the layer 2 corresponding to the dynamic interface element of the user interface 1 , and the first layer may be the layer 3 corresponding to the dynamic interface element of the switching input source interface 12 . Referring to FIG. 6A and FIG. 6B , the resolutions of layer 1 , layer 2 and layer 3 are smaller than or equal to resolution 1 .

In some embodiments, drawing the first layer corresponding to the dynamic interface elements of the first user interface includes: determining the first image in the first user interface image with the first resolution based on the interface layout of the first user interface The size of the layer is the first size, the position of the first layer is the first position, the position of the first dynamic interface element in the first layer of the first size is the second position and the size of the first dynamic interface element is the second position Two dimensions, the first dynamic interface element is any dynamic interface element of the first user interface in the first layer; based on the second position and the second size of the first dynamic interface element, draw the second dynamic interface element on the canvas of the first size A dynamic interface element, generating a first layer of a first size corresponding to the dynamic interface element of the first user interface.

In this embodiment of the present application, for example, the first layer may be the layer of the aforementioned first user interface, the first position of the first layer may be the aforementioned position 1, and the first size of the first layer may be the aforementioned Size 1. The second position of the first dynamic interface element may be the aforementioned position 2, and the second size of the first dynamic interface element may be the aforementioned size 2.

In some embodiments, the above synthesizing the layers corresponding to the dynamic interface elements of each user interface in the first display screen into the first user interface image with the first resolution includes: based on the first layer at the first resolution The layer corresponding to the dynamic interface element of each user interface is synthesized into the first user interface image of the first resolution.

In some embodiments, synthesizing the first display image of the second resolution based on the first user interface image of the first resolution and the first base image of the second resolution includes: combining the first user interface image of the first resolution The user interface image is enlarged to a first user interface image with a second resolution; the first user interface image with a second resolution and the first base image with a second resolution are synthesized into a first display image with a second resolution.

In this embodiment of the present application, for example, referring to FIG. 6C and FIG. 6D , the terminal device 100 enlarges the user interface image 1 with resolution 1 to the user interface image 1 with resolution 2, and then enlarges the user interface image 1 with resolution 2 and the bottom image 3 with resolution 2 to form the display screen shown in FIG. 6D .

In some embodiments, drawing the first layer corresponding to the dynamic interface elements of the first user interface includes: determining the first layer in the first user interface image at the second resolution based on the interface layout of the first user interface. The position of the first dynamic interface element in the layer is the third position and the size of the first dynamic interface element is the third size; based on the third size of the first dynamic interface element, draw the first dynamic interface on the canvas of the fourth size element, generate the first layer, there is no overlap between dynamic interface elements in the first layer, and the position of the first dynamic interface element in the first layer is the fourth position.

In this embodiment of the present application, for example, the first instruction may be the aforementioned instruction 1. Referring to FIG. 1A , the first instruction may be an input operation acting on the wired input source icon 103 shown in FIG. 1A . The first resolution may be the aforementioned resolution 3, the second resolution may be the aforementioned resolution 2, the first display screen may be the display screen of resolution 2 shown in FIG. For user interface image 2 with resolution 3 shown in FIG. 8C , the first base image may be base image 3 with resolution 2 shown in FIG. 8C .

In the embodiment of this application, for example, the first user interface may be the status bar 101, user interface 1 and switching input source interface 12 described in FIG. 1B and FIG. 8C, and the first layer may be the dynamic interface of the status bar 101 The layer 4 corresponding to the element, the first layer may be the layer 5 corresponding to the dynamic interface element of the user interface 1 , and the first layer may be the layer 6 corresponding to the dynamic interface element of the switching input source interface 12 . Referring to FIG. 8A and FIG. 8B , the resolutions of layer 1 , layer 2 and layer 3 are less than or equal to resolution 3 .

In this embodiment of the application, the third position of the first dynamic interface element may be the aforementioned position 3, the third size of the first dynamic interface element may be the aforementioned size 3, and the actual position of the first dynamic interface element in the first layer It may be the aforementioned fourth position. The fourth size may be the size of the layer (layer 4, layer 5 or layer 6) shown in FIG. 8A or FIG. 8B, that is, the size corresponding to the aforementioned specified size or specified resolution.

In some embodiments, the dynamic interface elements of the first user interface in the first layer are sequentially arranged in a first preset order in the first layer.

Exemplarily, referring to FIG. 8A and FIG. 8B , the dynamic interface elements in the layer are distributed in the order from left to right in the upper left corner of the layer without overlapping.

In some embodiments, the above synthesizing the layer corresponding to the dynamic interface element of each user interface in the first display screen into the first user interface image with the first resolution includes: based on the first dynamic interface in the first layer The third size and the fourth position of the interface element determine that the position of the first dynamic interface element in the first user interface image of the first resolution is the fifth position; based on the position of the first dynamic interface element in the first layer in the first The third size and the fifth position in the user interface image are combined into the first user interface image with the first resolution.

Exemplarily, the fifth position of the first dynamic interface element may be the aforementioned position 5 . Referring to FIG. 8A and FIG. 8B , the dynamic interface elements of each layer are synthesized into the user interface image 2 .

In some embodiments, the dynamic interface elements of multiple layers in the first user interface image are sequentially arranged in a second preset order in the first user interface image.

Exemplarily, referring to FIG. 8A and FIG. 8B , the dynamic interface elements of each layer are distributed in the user interface image 2 in the order of layers from top to bottom.

In some embodiments, synthesizing the first user interface image with the first resolution based on the third size and the fifth position of the first dynamic interface element in the first layer in the first user interface image includes: obtaining the first user interface image The image data of the first area where the first dynamic interface element is located in a layer, the size of the first area is the third size, and the position of the first area is the fourth position; determine the first dynamic interface element in the first user interface image The image data of the second area is the image data of the first area, the size of the second area is the third size, and the position of the second area is the fifth position.

In some embodiments, synthesizing the first display picture with the second resolution based on the first user interface image with the first resolution and the first base image with the second resolution includes: obtaining the first user interface image in the first user interface image The image data of the second area where a dynamic interface element is located, the size of the second area is the third size, and the position of the second area is the fifth position; the image data of the third area in the first base image is obtained, and the image data of the third area is The size is the third size, and the position of the third area is the third position of the first dynamic interface element; the image data of the second area and the image data of the third area are synthesized into the image data of the third area of the first display screen, A first display image is generated; the size of the third area is a third size, and the position of the third area is a third position.

Exemplarily, the second area may be the area 1 in the aforementioned user interface image 2, and the third area may be the area 2 in the aforementioned base image.

In some embodiments, when the terminal device runs the first display screen, the interface elements whose display content cannot be changed in the first display screen are static interface elements, and the interface elements whose display content is variable are dynamic interface elements; or, the terminal device runs the first display screen. When a screen is displayed, interface elements whose display content change probability is lower than a preset value in the first display screen are static interface elements, and interface elements whose display content change probability is greater than or equal to a preset value are dynamic interface elements.

In some embodiments, the drawing of the first user interface image of the first resolution corresponding to the first display picture, and obtaining the first base image of the second resolution of the first display picture include: drawing at the first frame rate The first user interface image of the first resolution corresponding to the first display screen, and obtain the first base map of the second resolution of the first display screen at a second frame rate; the first frame rate is greater than the second frame rate; the above Synthesizing the first display image with the second resolution includes: synthesizing the first display image with the second resolution at the first frame rate.

Exemplarily, the first frame rate may be the aforementioned frame rate 1, and the second frame rate may be the aforementioned frame rate 2.

In some embodiments, the fourth size is smaller than the first size, which is the size of the first layer in the first user interface image at the first resolution. Exemplarily, comparing FIG. 8A with FIG. 6A (or comparing FIG. 8B with FIG. 6B ), it can be seen that the fourth size (for example, the size of layer 4 corresponding to status bar 101) is smaller than the first size (for example, the size of layer 4 corresponding to status bar 101 1 size).

In some embodiments, the fourth size is greater than the sum of sizes of all dynamic interface elements of the first user interface in the first user interface image at the second resolution.

In some embodiments, the fourth size is determined by the terminal device based on the sum of the above sizes.

In some embodiments, the first dynamic interface element is a single dynamic interface element or a composite dynamic interface element composed of multiple dynamic interface elements.

Various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the present application will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, DSL) or wireless (eg, infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (solid state disk, SSD)), etc.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments are realized. The processes can be completed by computer programs to instruct related hardware. The programs can be stored in computer-readable storage media. When the programs are executed , may include the processes of the foregoing method embodiments. The aforementioned storage medium includes: ROM or random access memory RAM, magnetic disk or optical disk, and other various media that can store program codes.

In a word, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. All modifications, equivalent replacements, improvements, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

A display method, characterized in that it is applied to a terminal device, the terminal device includes a display screen, and the method includes:

In response to the first instruction, draw the first user interface image of the first resolution corresponding to the first display screen, and acquire the first base image of the second resolution of the first display screen; wherein, the first instruction is used to instruct the terminal device to draw the first display screen with the second resolution, the first user interface image includes dynamic interface elements in the first display screen, and the first base image includes the The static interface elements in the first display screen, the terminal device stores a drawn base map of at least one display screen, and the second resolution is greater than the first resolution;

synthesizing the first display image of the second resolution based on the first user interface image of the first resolution and the first base image of the second resolution;

The first display image is displayed through the display screen.
The method according to claim 1, wherein the first display screen includes at least one user interface, and drawing the first user interface image of the first resolution corresponding to the first display screen includes:

Draw a first layer corresponding to the dynamic interface elements of the first user interface, the first user interface is any user interface in the at least one user interface, and the resolution of the first layer is less than or equal to the first layer a resolution;

and synthesizing the layers corresponding to the dynamic interface elements of each user interface in the first display frame into the first user interface image with the first resolution.
The method according to claim 2, wherein said drawing the first layer corresponding to the dynamic interface elements of the first user interface comprises:

Based on the interface layout of the first user interface, it is determined that the size of the first layer in the first user interface image of the first resolution is a first size, and the position of the first layer is The first position, the position of the first dynamic interface element in the first layer of the first size is the second position and the size of the first dynamic interface element is the second size, the first dynamic interface The element is any dynamic interface element of the first user interface in the first layer;

Based on the second position and the second size of the first dynamic interface element, the first dynamic interface element is drawn on the canvas of the first size, and the dynamic interface element corresponding to the first user interface is generated. The first layer of the first size.
The method according to claim 3, wherein the layers corresponding to the dynamic interface elements of each user interface in the first display screen are synthesized into the first user interface with the first resolution Interface images, including:

Based on the first position and the first size of the first layer in the first user interface image with the first resolution, the layers corresponding to the dynamic interface elements of each user interface are synthesized into the The first user interface image at the first resolution.
The method according to claim 4, wherein the second base map is synthesized based on the first user interface image of the first resolution and the first base map of the second resolution. resolutions of the first display screen, including:

upscaling the first user interface image at the first resolution to a first user interface image at the second resolution;

Combining the first user interface image with the second resolution and the first base image with the second resolution into the first display image with the second resolution.
The method according to claim 2, wherein said drawing the first layer corresponding to the dynamic interface elements of the first user interface comprises:

Based on the interface layout of the first user interface, determining that the position of the first dynamic interface element in the first layer in the first user interface image of the second resolution is the third position and the The size of the first dynamic interface element is a third size; based on the third size of the first dynamic interface element, drawing the first dynamic interface element on a canvas of a fourth size to generate the first layer , there is no overlap between the dynamic interface elements in the first layer, and the position of the first dynamic interface element in the first layer is the fourth position.
The method according to claim 6, wherein the dynamic interface elements of the first user interface in the first layer are sequentially arranged in a first preset order in the first layer.
The method according to claim 6, wherein the layers corresponding to the dynamic interface elements of each user interface in the first display screen are synthesized into the first user interface with the first resolution Interface images, including:

Based on the third size and the fourth position of the first dynamic interface element in the first layer, determine that the first dynamic interface element is in the first user interface image with the first resolution The position of is the fifth position;

Synthesizing the first user interface image with the first resolution based on a third size and a fifth position of the first dynamic interface element in the first layer in the first user interface image.
The method according to claim 8, wherein the dynamic interface elements of multiple layers in the first user interface image are sequentially arranged in a second preset order in the first user interface image.
The method according to claim 8, wherein the first resolution is synthesized based on the third size and the fifth position of the first dynamic interface element in the first layer in the first user interface image. Rate the first UI image, including:

Acquiring the image data of the first area where the first dynamic interface element is located in the first layer, the size of the first area is the third size, and the position of the first area is the fourth Location;

Determining that the image data of the second area where the first dynamic interface element is located in the first user interface image is the image data of the first area, and the size of the second area is the third size, so The position of the second area is the fifth position.
The method according to any one of claims 8 to 10, wherein the first user interface image based on the first resolution and the first base image of the second resolution are The figure, synthesizing the first display frame of the second resolution, comprises:

Acquiring image data of a second area where the first dynamic interface element is located in the first user interface image, the size of the second area is the third size, and the position of the second area is the fifth position ;

Acquiring image data of a third area in the first base image, where the size of the third area is the third size, and the position of the third area is the third position of the first dynamic interface element ;

Combining the image data of the second area and the image data of the third area into the image data of the third area of the first display screen to generate the first display screen; the size of the third area is the size of the third area size, the position of the third area is the third position.
The method according to any one of claims 1 to 11, wherein when the terminal device runs the first display screen, the interface element whose display content is not changeable in the first display screen is the The static interface element, the interface element whose display content is variable is the dynamic interface element; or, when the terminal device runs the first display screen, the change probability of the display content in the first display screen is lower than a preset value The interface elements are static interface elements, and the interface elements whose display content change probability is greater than or equal to a preset value are dynamic interface elements.
The method according to any one of claims 1 to 11, characterized in that drawing the first user interface image of the first resolution corresponding to the first display screen, and obtaining the first user interface image of the first display screen The first basemap of the second resolution, including:

Draw the first user interface image of the first resolution corresponding to the first display picture at a first frame rate, and acquire the image of the second resolution of the first display picture at a second frame rate The first base image; the first frame rate is greater than the second frame rate;

The synthesizing the first display image with the second resolution includes: synthesizing the first display image with the second resolution at the first frame rate.
The method of claim 6, wherein the fourth size is smaller than a first size of the first image in the first user interface image at the first resolution. The dimensions of the layer.
The method according to claim 6, wherein the fourth size is greater than the size of all dynamic interface elements of the first user interface in the first user interface image of the second resolution Sum.
The method according to claim 15, wherein the fourth size is determined by the terminal device based on the sum of the sizes.
The method according to claim 3 or 6, wherein the first dynamic interface element is a single dynamic interface element or a composite dynamic interface element composed of multiple dynamic interface elements.
A terminal device, including a touch screen, memory, one or more processors, multiple application programs, and one or more programs; wherein, the one or more programs are stored in the memory; its features That is, when the one or more processors execute the one or more programs, the terminal device implements the method according to any one of claims 1 to 17.
A computer storage medium, characterized by comprising computer instructions, and when the computer instructions are run on a terminal device, the electronic device is made to execute the method according to any one of claims 1 to 17.
A computer program product, characterized in that, when the computer program product is run on a computer, the computer is made to execute the method according to any one of claims 1 to 17.