CN117785345A

CN117785345A - Application display method and related equipment

Info

Publication number: CN117785345A
Application number: CN202211204215.6A
Authority: CN
Inventors: 王晓东; 王泽智; 陈志彬; 王晓锋
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-09-29
Filing date: 2022-09-29
Publication date: 2024-03-29

Abstract

The embodiment of the application provides an application display method and related equipment. The Android system of the client captures a layer creating message of an application program, and creates a layer window notification and sends the layer window notification to the host; the host machine creates a target layer window according to the layer window notification and returns a handle return of the target layer window. The Android system associates a layer of an application program with a handle of a target layer window, acquires application image content according to a synchronous periodic signal of a host machine, and renders the application image content in the target layer window corresponding to the handle of the target layer window according to the handle of the target layer window; and the host responds to the display notification sent by the Android system, and displays the application image content in the target layer window corresponding to the handle of the target layer window. According to the method and the device, the problem of time delay existing in the situation that display content of an Android application program is sent to a host side window under an ecological fusion scene can be solved.

Description

Application display method and related equipment

Technical Field

The present disclosure relates to the field of display technologies, and in particular, to an application display method and related devices.

Background

With further ecological integration of a personal computer (Personal Computer, PC) and a mobile phone, an Android system is operated as a client on the PC (host), so that an Android application program can be operated on a PC operating system, and a user can conveniently use the Android application program on the PC. However, due to the limitation of the existing Android system in terms of native rendering and sending and displaying mechanisms, a user triggers the display content change of the Android application program, and a large time delay exists from the completion of sending and displaying of the Android application program on the PC, so that the display change of the Android application program cannot be presented on the PC in time, and the user experience is affected.

Disclosure of Invention

In view of the foregoing, it is necessary to provide an application display method and related devices to solve the problem that there is a relatively large delay in the display content transmission and display of the Android application program.

An embodiment of the present application provides a method for displaying an application, where the application is applied to a host and a client communicatively connected to the host, and the method includes: creating a layer in the running process of an application program of the Android system; capturing a layer creating message of an application program by an Android system of the client, and sending a layer creating window notification to the host according to the layer creating message; the host creates a target layer window according to the layer window notification, and returns a handle of the target layer window to the client; the Android system associates a layer of the application program with a handle of the target layer window; the host synchronizes a hardware Vsync period signal to the client; the Android system acquires application image content of the application program according to the hardware Vsync periodic signal, and renders the application image content in the target layer window corresponding to the handle of the target layer window; and the host responds to the display notification sent by the Android system, and displays the application image content in the target layer window corresponding to the handle of the target layer window. According to the technical scheme, the Android system acquires application image content of the application program according to the hardware Vsync periodic signal sent by the host machine, renders the application image content in the target layer window corresponding to the handle of the target layer window, and the host machine responds to the display notification sent by the Android system to display the application image content in the target layer window corresponding to the handle of the target layer window, so that the problem of time delay in sending and displaying the display content of the Android application program to the host machine window in an ecological fusion scene of the host machine and the client machine is solved.

In an implementation, the rendering the application image content in the target layer window corresponding to a handle of the target layer window includes: the Android system call eglCreate Window Surface interface creates egl window surface associated with the target layer window of the host; and rendering the application image content on the egl window surface. In the technical scheme, the Android system call eglCreate Window Surface interface directly draws the content of the layer of the Android system of the client to the window of the host for display, so that a layer buffer area required when the Android system synthesizes the image of the layer is saved, and the cost of memory space is saved.

In an implementation, the egl window surface includes a front end buffer and a back end buffer, the rendering the application image content on the egl window surface includes: the application image content is rendered at the back-end buffer of the egl window surface. In the technical scheme, the Android system directly renders the application image content in the rear-end buffer area of egl window surface corresponding to the target layer window, so that the overhead of the memory space is saved.

In an implementation manner, the host machine responds to the display notification sent by the Android system, and displaying the application image content in the target layer window corresponding to the handle of the target layer window includes: the Android system calls an eglSwapBuffer interface to send an eglSwapBuffer instruction to the host, wherein the eglSwapBuffer instruction comprises at least one of an index of an address of the back-end buffer area or the application image content and a handle of the target layer window of the host; the host computer displays the application image content on the target layer window of the host computer by exchanging the content of the front end buffer and the back end buffer. According to the technical scheme, the host machine displays the application image content on the target image layer window of the host machine by exchanging the content of the front end buffer zone and the content of the rear end buffer zone, so that a data transmission process of the application image content between the host machine and the client machine is avoided, and the display efficiency of the application image content on the host machine is improved.

In one implementation, the host synchronizing the hardware Vsync period signal to the client includes: the host acquires a hardware Vsync periodic signal sent by the host through a shared memory module with the client.

In an implementation manner, creating the layer by the application program of the Android system in the running process includes: the host receives the operation of starting the application program input by a user, and generates a notification of starting the application program; the Android system runs the application program according to the notification of starting the application program sent by the host machine; the application creates a layer during the run. According to the technical scheme, the Android system can operate the application program according to the operation of starting the application program by the user detected by the host machine, and a layer is created.

In one implementation, the application creating the layer during the running process includes: the application program calls a startActivity interface of the Android system to start the application program, and a notice for creating a layer is sent to a window management service module of the Android system in the starting process of the application program; and the window management service module creates the layer window according to the notification of creating the layer. In the technical scheme, the Android system creates the layer window of the application program through the window management service module.

In one implementation, the method further comprises: the client acquires state information of a layer window of the application program, determines an operation instruction of the layer window according to the state information, and sends the operation instruction to the host; and the host machine carries out corresponding operation on the target layer window associated with the layer window according to the operation instruction. In the above technical solution, the host machine performs corresponding operation on the target layer window associated with the layer window according to the operation instruction sent by the client machine.

In one implementation, the method further comprises: the client acquires display state information of a layer window of the application program, determines an operation instruction of the layer window as a display instruction according to the display state information, and sends the display instruction to the host; and the host displays the target layer window according to the display instruction. In the above technical solution, the host displays the target layer window associated with the layer window according to the display instruction sent by the client.

In one implementation, the method further comprises: the client acquires hidden state information of a layer window of the application program, determines an operation instruction of the layer window as a hidden instruction according to the hidden state information, and sends the hidden instruction to the host; and hiding the target layer window by the host machine according to the hiding instruction. In the above technical solution, the host machine conceals the target layer window associated with the layer window according to the concealing instruction sent by the client machine.

In one implementation, the method further comprises: the client acquires the moving state information of the layer window of the application program, determines the operating instruction of the layer window as a moving instruction according to the moving state information, and sends the moving instruction to the host; and the host machine moves the target layer window according to the movement instruction. In the above technical solution, the host machine moves the target layer window associated with the layer window according to the movement instruction sent by the client machine.

In one implementation, the method further comprises: the client acquires transparency state information of a layer window of the application program, determines an operation instruction of the layer window as a transparency setting instruction according to the transparency state information, and sends the transparency setting instruction to the host; and the host machine sets the transparency of the target layer window according to the transparency setting instruction. In the above technical solution, the host machine sets the transparency of the target layer window associated with the layer window according to the transparency setting instruction sent by the client machine.

In one implementation, the method further comprises: the client acquires Z-order state information of a layer window of the application program, determines an operation instruction of the layer window to be a Z-order setting instruction according to the Z-order state information, and sends the Z-order setting instruction to the host; and the host machine sets the Z sequence of the target layer window according to the Z sequence setting instruction. In the above technical solution, the host machine sets the Z sequence of the target layer window associated with the layer window according to the Z sequence setting instruction sent by the client machine.

In one implementation, the method further comprises: the host acquires operation information of a user on the target application window of the application program, determines an operation instruction of the target application window according to the operation information, and sends the operation instruction to the client; and the client performs corresponding operation on the application window or the layer window of the application program according to the operation instruction.

In a second aspect, embodiments of the present application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the above-described application display method.

In a third aspect, an embodiment of the present application provides an electronic device, where the electronic device includes a processor and a memory, where the memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory, so that the electronic device executes the application display method described above.

In a fourth aspect, an embodiment of the present application provides a chip coupled to a memory in an electronic device, where the chip is configured to control the electronic device to execute the application display method described above.

It will be appreciated that the computer readable storage medium according to the second aspect, the electronic device according to the third aspect, and the chip according to the fourth aspect correspond to the application display method, and therefore, the advantages achieved by the method may refer to the advantages in the corresponding method provided above, and will not be described herein.

Drawings

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Fig. 2 is a schematic software structure of an electronic device according to an embodiment of the present application.

FIG. 3a is a timing simulation diagram of rendering and sending by the application display system according to an embodiment of the present application.

FIG. 3b is a timing simulation diagram of rendering and sending by the application display system according to another embodiment of the present application.

Fig. 4 is a block diagram of an application display system according to an embodiment of the present application.

Fig. 5 is a block diagram of an application display system according to another embodiment of the present application.

Fig. 6 is a block diagram of an application display system according to another embodiment of the present application.

Fig. 7 is a block diagram of an application display system according to another embodiment of the present application.

Fig. 8 a-8 b are schematic flow diagrams of an application display method according to an embodiment of the present application.

Fig. 9 is a flowchart of an application display method according to another embodiment of the present application.

Detailed Description

It should be noted that "at least one" in this application means one or more, and "a plurality" means two or more. "and/or", describes an association relationship of an association object, and the representation may have three relationships, for example, a and/or B may represent: a alone, a and B together, and B alone, wherein a, B may be singular or plural. The terms "first," "second," "third," "fourth" and the like in the description and in the claims and drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.

In the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as examples, illustrations, or descriptions. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion. The following embodiments and features of the embodiments may be combined with each other without conflict.

For ease of understanding, a description of some of the concepts related to the embodiments of the present application are given by way of example for reference.

Host (Host): refers to the host operating system running the application.

Client (Guest): a client operating system comprisingThe system.

Window management service (Window Manager Service, WMS):the window management service allocates a layer (Surface) for all windows and is responsible for life cycle management of the windows, including creation, hiding, moving, scaling, rotating, destroying and the like.

Surfeflinger: responsible for managementFrame Buffer of system>The surfeflink service in the system draws a user interface in the application by writing content to the frame buffer; in a mobile application engine scene, the SurfaceFlinger service forms all layers in the application into one frame of image and notifies a Host side The rows are displayed.

Layer (Surface): at the position ofIn the method, a window can monopolize a Surface, the Surface is a Canvas, and an application program draws content on the Surface through Canvas or OpenGL; after being drawn on the Surface, the contents of a plurality of Surface are mixed by the Surface eFlink and output to the Frame Buffer, thereby the Surface is processedThe page is presented to the user.

Embedded graphics library (Embedded-System Graphics Library, EGL): the system is an interface between the OpenGL ES and the underlying Native platform window system, and plays a role of a bridge between the OpenGL and the Native window system.

The embedded OpenGL (OpenGL for Embedded Systems, openGL ES) is a subset of the OpenGL three-dimensional graphics API, and is designed for embedded devices such as mobile phones, PDAs, game hosts, etc., and can access an image processor (GPU) of a hardware device through OpenGL ES instructions to complete rendering of the layer content. The electronic device may communicate with other electronic devices or servers over a communication network. The electronic device of the application program of the embodiment of the application program may be a communication device, for example, may be a server or a terminal device. The terminal device may include at least one of a mobile phone, a foldable electronic device, a tablet computer, a personal computer (personal computer, PC), a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, a cellular phone, a personal digital assistant (personal digital assistant, PDA), an augmented reality (augmented reality, AR) device, a Virtual Reality (VR) device, an artificial intelligence (artificial intelligence, AI) device, a wearable device, a vehicle-mounted device, a smart home device, and a smart city device, and the specific type of the electronic device is not particularly limited in the embodiments of the present application. The communication network may be a wired network or a wireless network. For example, the communication network may be a local area network (local area networks, LAN) or a wide area network (wide area networks, WAN), such as the internet. When the communication network is a local area network, the communication network may be, for example, a wireless fidelity (wireless fidelity, wi-Fi) hotspot network, a Wi-Fi P2P network, a bluetooth network, a zigbee network, or a near field communication (near field communication, NFC) network, or the like. When the communication network is a wide area network, the communication network may be, for example, a third generation mobile communication technology (3 rd-generation wireless telephone technology, 3G) network, a fourth generation mobile communication technology (the 4th generation mobile communication technology,4G) network, a fifth generation mobile communication technology (5 th-generation mobile communication technology, 5G) network, a future evolution public land mobile network (public land mobile network, PLMN), the internet, or the like.

In some embodiments, the electronic device may install one or more APP (Application). APP may be abbreviated as an application, a software program capable of performing some or more specific functions. For example, instant messaging applications, video applications, audio applications, image capture applications, cloud desktop applications, and the like. The instant messaging applications may include, for example, short message applications,WhatsApp />Photo sharing->Kakao/>Etc. Image capture class applications may include, for example, camera applications (system cameras or third party camera applications). Video-type applications may for example comprise +.>Etc. An audio-class application is provided which,for example, may include->Etc. The application mentioned in the following embodiments may be a system application installed when the electronic device leaves the factory, or may be a third party application that is downloaded from a network or obtained from other electronic devices by a user during the process of using the electronic device.

Electronic devices include, but are not limited to, on-boardWindowsOr other operating system.

Fig. 1 illustrates a schematic structure of an electronic device 100.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 130, a wireless communication module 140, an audio module 150, a sensor module 160, a camera module 170, a display screen 180, and the like.

It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The processor can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. The memory may hold instructions or data that are used or used more frequently by the processor 110. If the processor 110 needs to use the instruction or data, it can be called directly from the memory, avoiding repeated access, reducing the latency of the processor 110 and thus improving the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, and/or a universal serial bus (universal serial bus, USB) interface, among others. The processor 110 may be connected to modules such as an audio module, a wireless communication module, a display, a camera, etc. through at least one of the above interfaces.

It should be understood that the interfacing relationship between the modules illustrated in the embodiments of the present application is only illustrative, and does not limit the structure of the electronic device 100. In other embodiments of the present application, the electronic device 100 may also use different interfacing manners, or a combination of multiple interfacing manners in the foregoing embodiments.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 130, the wireless communication module 140, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 130 may provide a solution for wireless communication including 2G/3G/4G/5G, etc., applied on the electronic device 100. The mobile communication module 130 may include at least one filter, switch, power amplifier, low noise amplifier, etc. The mobile communication module 130 may receive electromagnetic waves from the antenna 1, perform processes such as filtering, amplifying, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 130 can amplify the signal modulated by the modem processor, and convert the signal into electromagnetic waves through the antenna 1 to radiate. In some embodiments, at least some of the functional modules of the mobile communication module 130 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 130 may be disposed in the same device as at least some of the modules of the processor 110.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs a sound signal through the audio module or displays an image or video through the display 180. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 130 or other functional module, independent of the processor 110.

The wireless communication module 140 may provide solutions for wireless communication including wireless local area network (wireless local area networks, WLAN), bluetooth (BT), bluetooth low energy (bluetooth low energy, BLE), ultra Wide Band (UWB), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), NFC, infrared technology (IR), etc. applied to the electronic device 100. The wireless communication module 140 may be one or more devices integrating at least one communication processing module. The wireless communication module 140 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signals, filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 140 may also receive a signal to be transmitted from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2.

In some embodiments, antenna 1 and mobile communication module 130 of electronic device 100 are coupled, and antenna 2 and wireless communication module 140 are coupled, such that electronic device 100 may communicate with networks and other electronic devices through wireless communication techniques. The wireless communication techniques may include the Global System for Mobile communications (global system for mobile communications, GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC, FM, and/or IR techniques, among others. The GNSS may include a global satellite positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a beidou satellite navigation system (beidou navigation satellite system, BDS), a quasi zenith satellite system (quasi-zenith satellite system, QZSS) and/or a satellite based augmentation system (satellite based augmentation systems, SBAS).

The electronic device 100 may implement display functions through a GPU, a display screen 180, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 180 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The sensor module comprises a touch sensor, a pressure sensor, a fingerprint sensor and the like. The camera module 170 includes a camera. The display 180 is used to display images, videos, and the like. The display 180 includes a display panel. The display panel may employ a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED) or an active-matrix organic light-emitting diode (matrix organic light emitting diode), a flexible light-emitting diode (flex), a mini, a Micro led, a Micro-OLED, a quantum dot light-emitting diode (quantum dot light emitting diodes, QLED), or the like. In some embodiments, electronic device 100 may include 1 or more display screens 180.

The electronic device 100 may implement camera functions through the camera module 170, isp, video codec, GPU, display 180, and application processor AP, neural network processor NPU, etc.

The digital signal processor is used for processing digital signals, and can also process other digital signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to fourier transform the frequency bin energy, or the like.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record video in a variety of encoding formats, such as: dynamic picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The NPU is a neural-network (NN) computing processor, and can rapidly process input information by referencing a biological neural network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent awareness of the electronic device 100 may be implemented through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, etc.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to enable expansion of the memory capabilities of the electronic device 100. The external memory card communicates with the processor 110 through an external memory interface 120 to implement data storage functions. For example, files such as music, video, etc. are stored in an external memory card. Or transfer files such as music, video, etc. from the electronic device to an external memory card.

The internal memory 121 may be used to store computer executable program code that includes instructions. The internal memory 121 may include a storage program area and a storage data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc. The storage data area may store data created during use of the electronic device 100 (e.g., audio data, phonebook, etc.), and so on. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (universal flash storage, UFS), and the like. The processor 110 performs various functional methods or data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The audio module 150 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The audio module 150 may also be used to encode and decode audio signals. In some embodiments, the audio module 150 may be disposed in the processor 110, or some functional modules of the audio module 150 may be disposed in the processor 110.

The software system of the electronic device 100 may employ a layered architecture, an event driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. Embodiments of the present application are in a layered architectureThe system is an example illustrating the software architecture of the electronic device 100.

Fig. 2 is a software configuration block diagram of the electronic device 100 according to the embodiment of the present application.

The layered architecture divides the software into several layers, each layer having distinct roles andand (5) dividing labor into work. The layers communicate with each other through a software interface. In some embodiments, it willThe system is divided into five layers, namely an application program layer, an application program framework layer and a An Zhuoyun line (& gt) from top to bottom>run, ART) and native C/c++ libraries, hardware abstraction layer (Hardware Abstract Layer, HAL), and kernel layer.

The application layer may include a series of application packages.

As shown in fig. 2, the application package may include applications for cameras, gallery, calendar, phone calls, maps, navigation, WLAN, bluetooth, music, video, short messages, etc.

The application framework layer provides an application programming interface (application programming interface, API) and programming framework for application programs of the application layer. The application framework layer includes a number of predefined functions.

As shown in FIG. 2, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, and so forth.

The window manager provides window management services (Window Manager Service, WMS) that may be used for window management, window animation management, surface management, and as a transfer station to the input system.

The content provider is used to store and retrieve data and make such data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebooks, etc.

The view system includes visual controls, such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, a display interface including a text message notification icon may include a view displaying text and a view displaying a picture.

The resource manager provides various resources for the application program, such as localization strings, icons, pictures, layout files, video files, and the like.

The notification manager allows the application to display notification information in a status bar, can be used to communicate notification type messages, can automatically disappear after a short dwell, and does not require user interaction. Such as notification manager is used to inform that the download is complete, message alerts, etc. The notification manager may also be a notification in the form of a chart or scroll bar text that appears on the system top status bar, such as a notification of a background running application, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in a status bar, a prompt tone is emitted, the electronic device vibrates, and an indicator light blinks, etc.

The activity manager may provide activity management services (Activity Manager Service, AMS) that may be used for system component (e.g., activity, service, content provider, broadcast receiver) start-up, handoff, scheduling, and application process management and scheduling tasks.

The input manager may provide input management services (Input Manager Service, IMS), which may be used to manage inputs to the system, such as touch screen inputs, key inputs, sensor inputs, and the like. The IMS retrieves events from the input device node and distributes the events to the appropriate windows through interactions with the WMS.

The android runtime includes a core library and An Zhuoyun rows. The android runtime is responsible for converting source code into machine code. Android runtime mainly includes employing Advanced Or Time (AOT) compilation techniques and Just In Time (JIT) compilation techniques.

The core library is mainly used for providing the functions of basic Java class libraries, such as basic data structures, mathematics, IO, tools, databases, networks and the like. The core library provides an API for the user to develop the android application.

The native C/c++ library may include a plurality of functional modules. For example: surface manager (surface manager), media Framework (Media Framework), libc, openGL ES, SQLite, webkit, etc.

The surface manager is used for managing the display subsystem and providing fusion of 2D and 3D layers for a plurality of application programs. Media frames support a variety of commonly used audio, video format playback and recording, still image files, and the like. The media library may support a variety of audio and video encoding formats, such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES provides for drawing and manipulation of 2D graphics and 3D graphics in applications. SQLite provides a lightweight relational database for applications of the electronic device 100.

The hardware abstraction layer runs in a user space (user space), encapsulates the kernel layer driver, and provides a call interface to the upper layer.

The kernel layer is a layer between hardware and software. The inner core layer at least comprises a display driver, a camera driver, an audio driver and a sensor driver.

With the further ecological convergence of personal computers (Personal Computer, PC) and cell phones,the system runs as a client on a PC (host machine) and so on>The application of the system may run on the PC operating system so that the user may conveniently use the application on the PC. However, due to the existing->And the system is limited in terms of a native rendering and sending mechanism, a user triggers the display content of the application program to change, and at least two Vertical synchronization (VSync) periods are needed for layer rendering, synthesizing and sending, wherein at least one VSync period is used for completing layer rendering and drawing, and at least one VSync period is used for completing layer synthesizing and sending. Because the window for transmitting the display content of the application program to the PC side needs at least two VSync periods, the interface of the application program will not be rendered after finishing the rendering of the updated content of the layer after the application program receives one VSync period The updated content of the image layer can be displayed on the interface of the application program after the next VSync period after the completion of synthesis and display, so that the time delay of displaying the display content of the application program on the PC is increased. Referring to fig. 3a, the display content of the application program is created by the layers, rendered, drawn, synthesized, sent and finally displayed on the interface of the application program, and at least 2 VSync periods are consumed, so that the time delay of displaying the display content of the application program on the PC is increased.

As shown in fig. 4, a block diagram of an application display system 10 according to an embodiment of the present application is shown. The application display system 10 includes a client 11 and a host 12. The client 11 comprisesThe system. The host machine 12 comprisesAn operating system. The client 11 is communicatively connected to the host 12 via a communication module 13. In an embodiment of the present application, the client 11 and the host 12 of the application display system 10 are the same hardware device, such as the electronic device 100 described in fig. 1-2 above. The communication module 13 includes at least one of a shared memory module or a virtual device communication module. In one embodiment of the present application, the application of client 11 may run on the operating system of host 12 through virtualization, container techniques. For example, huacheng's mobile application engine scene +. >An application is run on the operating system. For another example, netease->An Zhuozi System (Windows Subsystem for Android, WSA) etc ≡>SimulationThe machine runs on the operating system of the host machine 12 through virtualization, container technology.

In another embodiment of the present application, the client 11 and the host 12 of the application display system 10 are different hardware devices. The communication module 13 comprises a network communication module. The client 11 transmits the display content of the application program to the host 12 through the network communication module, and the host 12 performs windowed display of the display content of the application program. For example, in the multi-screen collaboration scenario of Huacheng, the display content of the application program running in the mobile phone is transmitted to the PC to run through the network communication moduleAnd the operating system performs windowed display and supports the use of a key mouse of the PC to control an application program running on the mobile phone. In an embodiment of the present application, the network communication module includes at least one of a 5G communication module, a 4G communication module, a Wi-Fi communication module, or a bluetooth communication module.

In the mobile application engine scenario and the multi-screen collaboration scenario, a host 12 is establishedOperating system created window Handle and client 11 +. >Mapping relation of application windows (Virtual Dsiplay) of system, each application program independently runs at +.>In the Virtual Display of the system, the host 12 displays the Virtual Display content associated with the window Handle on the window of the host 12, so that the application program can be in the form of multiple windows in +_ of the host 12>Running on an operating system.

Referring to FIG. 4, a client 11The system includes an application 111, a Goldfish OpenGL module (virtual OpenGL module) 112, a WMS module 113, a Buffer Queue cache 114, a SurfaceFlinger module 115, and a virtual screen 116. The host 12 includes a rendering module 124 and a rendering module 125. The application 111 obtains the image content of the application and sends the image content to the Goldfish OpenGL module 112. In the present embodiment, the application 111 receives +.>The system simulated first Vsync period signal then converts the image content into a GLES/EGL instruction stream and sends the GLES/EGL instruction stream to the Goldfish OpenGL module 112. The image content includes one or more layer updates.

In one embodiment of the present application, application 111 sends window setting instructions to WMS module 113 during execution or when image content changes, where the window setting instructions include the number of layers of the image content. WMS module 113 creates a canvas (Surface) corresponding to the number of layers according to the number of layers in the window setting instruction, where each Surface is associated with one layer.

WMS module 113 sends a storage space application instruction to the Goldfish OpenGL module 112. The Goldfish OpenGL module 112 applies for a PBuffer surface memory space for each layer according to the memory space application instruction, where each PBuffer surface memory space is associated with a layer. The Goldfish OpenGL module 112 renders the image content into a PBuffer surface storage space associated with the layer and populates the rendered image content into a Buffer Queue cache 114 associated with the layer.

In an embodiment of the present application, after receiving the second Vsync period signal simulated by the software, the surface eflinger module 115 traverses all the layers to obtain the rendered image content from the Buffer Queue 114 associated with the layers, and synthesizes a frame of display image. The SurfaceFlinger module 115 sends the display image to the virtual screen 116. The virtual screen 116 transmits the display image to the display module 125 of the host 12 through the communication module 13. The display module 125 displays the display image on a Windows window associated with the virtual screen 116.

In the above scheme, the client 11The first Vsync period signal and the second Vsync period signal of the system are simulated by software, are inconsistent with the Vsync period of the hardware Vsync on the host 12, and have unstable time delay of sending and displaying the send and display content of the application program to the Windows of the host 12.

In order to solve the above-mentioned technical problems, the present application provides an application display method, with which a client machineThe system obtains application image content of an application program, creates a layer window according to the application image content, sends a notification of creating the layer window to the host, and the host creates a target layer window according to the notification of the layer window and returns a Handle (Handle) of the target layer window to the client. Said->And the system associates the application image content with the handle of the target layer window and sends the associated handle of the target layer window to the host. And the host acquires the application image content and renders the application image content in the target layer window corresponding to the handle of the target layer window according to the handle of the target layer window. Said host responds to said +.>The display notice sent by the system displays the application image content in the target layer window corresponding to the handle of the target layer window, so that the display of the application program triggered by the user is reducedAnd (5) changing the capacity, and completing the time delay of display. In order to facilitate description of the application display method provided in the embodiments of the present application, an application environment of the application display method is described below with reference to fig. 5 to 6, respectively.

To solve the problem of unstable delay of sending the display content of the application program to the Windows of the host 12, the application display system 10 of another embodiment is provided. Referring to fig. 5, a block diagram of an application display system 10 according to another embodiment of the present application is shown. The client 11 includes a Goldfish OpenGL module 112, an application 111, a WMS module 113, and a window management client 119. The Goldfish OpenGL module 112 includes an eglceatewindowsurface interface 117, an eglswappuffer interface 118. The host 12 includes a rendering module 124, a sending module 125, a Vsync signal capturing module 122, and a window management server 123.

In one embodiment of the present application, application 111 notifies WMS module 113 to create an application window during the launch run. WMS module 113 transmits the create application window notification to window management client 119. The window management client 119 sends a create application window notification to the window management server 123 of the host 12. The window management server 123 creates a target application window according to the created application window notification, and returns a handle of the target application window to the window management client 119. The window management client 119 associates the application window with the handle of the target application window and sends the handle of the target application window to the Goldfish OpenGL module 112. After the completion of the application 111, an application start completion notification is sent to the WMS module 113.WMS module 113 transmits an application launch completion notification to window management client 119. The window management client 119 transmits an application start completion notification to the window management server 123. The window management server 123 displays the target application window and displays the target application window in the foreground window of the host 12.

The Vsync signal capture module 122 obtains the hardware Vsync period signal of the host 12 through a preset interface (e.g., IDXGIOutput:: waitForVBlank interface). The Vsync signal capturing module 122 transmits the hardware Vsync period signal to the guest through the communication module 13Hwcomposer (HWC) process of the subscriber 11. The HWC process triggers the client 11 upon receipt of the hardware Vsync period signalVsync period signal of system, and for +.>The system simulated Vsync period signal is calibrated. In an embodiment of the present application, the hardware Vsync period signal includes a hardware VBlank signal.

The application 111, upon receivingWhen the Vsync period signal of the system is received, the image content of the application program is acquired, and a window setting instruction is transmitted to the WMS module 113. Wherein the window setting instruction includes a number of layers of the image content. WMS module 113 creates layers based on the number of layers in the window setting instruction. The window management client 119 acquires creation information of the layer of the application program from the WMS module 113 and transmits the creation information of the layer of the application program to the window management server 123 of the host 12. The window management server 123 creates an associated layer window according to the creation information of the layer, and returns the handle of the layer window to the window management client 119. Window management client 119 sends the handle of the layer window to WMS module 113. The WMS module 113 associates a handle of a layer window with a layer and sends the handle of the layer window associated with the layer to the Goldfish OpenGL module 112.

In an embodiment of the present application, the window management client 119 is further configured to obtain the state information of the layer of the application program from the WMS module 113, and send the state information of the layer of the application program to the window management server 123 of the host 12 through the communication module 13. The window management server 123 adjusts the window of the associated layer according to the state information of the layer of the application program. The adjustment includes adding, deleting, modifying, moving the windows of the layers, and synthesizing the windows of the layers.

In an embodiment of the present application, WMS module 113 sends a storage space application instruction (create instruction) to Goldfish OpenGL module 112. The Goldfish OpenGL module 112 calls the eglceateendowsurface interface 117 to apply a Window surface memory space (egl Window surface) for each layer according to the memory space application instruction, where each Window surface memory space is associated with a target layer Window of the host 12. The Goldfish OpenGL module 112 renders the image content into a Window surface storage space associated with the target layer Window of the host 12. In another embodiment of the present application, the Goldfish OpenGL module 112 sends rendered image content to the rendering module 124 of the host 12. Rendering module 124 renders the image content in a layer window associated with the handle of the target layer window. After the image content of the image layer is rendered, the Goldfish OpenGL module 112 calls an eglswappuffer interface 118 to send an eglswappuffer instruction to the host 12, where the eglswappuffer instruction includes at least one of an index of an address of the back-end buffer or the application image content, and a handle of the target image layer window of the host. The sending and displaying module 125 displays the image content on the associated layer according to the eglsweeppuffer instruction. In the above solution, the rendering module 124 renders the image content in the layer window associated with the handle of the layer window, thereby rendering the image content on the client The content of the layer of the system is directly drawn on the window of the host machine for display, so that +.>The image layer buffer area needed when the SurfaceFlinger of the system synthesizes the image of the image layer saves the memory. />The system displays the application image content in the target layer window corresponding to the handle of the target layer window, so that user trigger +_f is reduced>And the display content of the application program of the system changes and the time delay from the completion of the display. Referring to fig. 3b, rendering, synthesizing and transmitting of the display content of the application program are completed within one period, so that the time delay of displaying the display content of the application program on the PC can be reduced.

Referring to fig. 6, a block diagram of a display system 10 in another embodiment of the present application is shown. The client 11 and the host 12 in the application display system 10 are the same hardware device. The host 12 has a mobile application engine 14 disposed thereon. The client 11 is provided on the mobile application engine 14. The Goldfish OpenGL module 112, the application 111, the WMS module 113, the eglceateendowsurface interface 117, the eglsweeppuffer interface 118, and the window management client 119 of the client 11 refer to the description contents of the respective corresponding modules. The rendering module 124, the display module 125, the Vsync signal capturing module 122, and the window management server 123 of the host 12 are disposed on the mobile application engine 14. The functions of the rendering module 124, the display module 125, the Vsync signal capturing module 122, and the window management server 123 are described above. In an embodiment of the present application, the window management server 123 further includes a countercontrol module, layerDim, layerBlur window simulation, screen capturing, and screen recording functions. The back control module can receive the input operation input by the user through the keyboard and the mouse of the host, and send the summer-heat promotion operation to the input management module of the client 11, and the input management module loads the input operation on the application program.

Referring to FIG. 7, an embodiment of the present application is shownThe system draws the content of the layer directly into the host's window. />The application window of the system displays at least one layer window, each layer window corresponds to a target layer windowIs a handle to the computer. />The system sends an eglSwapBuffer instruction to the send display module 125 of the host 12, the eglSwapBuffer instruction including an index of the address of the back-end buffer. Said egl window surface comprises a front end buffer and a back end buffer, said +.>The system renders the application image content in the back-end buffer. The content of the front-end buffer is the content currently displayed by the target layer window of the host 12. The sending and displaying module 125 exchanges the content of the front end buffer with the content of the back end buffer according to the eglswappuffer instruction, and displays the application image content on the target layer window of the host 12.

Referring to fig. 8a-8b, a flow chart of an application display method according to an embodiment of the present application is shown. The method specifically comprises the following steps.

The window management server 123 of the host 12 receives 601 an operation of starting an application program input by a user, and generates a notification of starting the application program.

In one embodiment of the present application, a user clicks an icon of an application on the desktop of host 12 to perform an operation to launch the application. The window management server 123 generates a notification to launch the application in response to a user clicking on an icon of the application.

In another embodiment of the present application, the window management server 123 generates a notification of the start application in response to an operation of the start application performed by the user.

602, the window management server 123 transmits a notification of starting an application to the window management client 119 of the client 11.

In one embodiment of the present application, window management server 123 sends a notification to client 11 to window management client 119 to launch the application via the QEMU PIPE high speed channel.

603, the window management client 119 sends a notification to start the application 111.

604, the application 111 launches the application, creates an application window, and sends a notification to the window management client 119 to create the application window.

In an embodiment of the present application, the application 111 invokes a startActivity interface to start the application, and sends a notification of creating an application window to the WMS module 113 during the start of the application. WMS module 113 creates an application window from the notification of creating the application window and transmits the notification of creating the application window to window management client 119.

605, the window management client 119 transmits a notification of creating the application window to the window management server 123.

In an embodiment of the present application, window management client 119 sends a notification to create an application window to window management server 123 via the QEMU PIPE high speed channel.

606, the window management server 123 creates a target application window according to the notification of creating the application window, and returns a handle of the target application window to the window management client 119.

In one embodiment of the present application, window management server 123 creates the target application window via a WindowsAPI function. The WindowsAPI function includes a Creatwindow function. After the target application window is created, the window management server 123 returns a handle of the target application window to the window management client 119.

607, window management client 119 returns the handle of the target application window to WMS module 113.

608, wms module 113 associates the application window with the handle of the target application window and sends the handle of the application window to Goldfish OpenGL module 112.

In one embodiment of the present application, window management client 119 returns the handle of the target application window to WMS module 113.WMS module 113 associates the target application window with the handle of the target application window and sends the handle of the target application window to Goldfish OpenGL module 112.

609, the application program 111 transmits an application start completion notification to the window management client 119.

In an embodiment of the present application, the application program 111 transmits an application start completion notification to the window management client 119 after the completion of application start.

The window management client 119 transmits 610 an application startup completion notification to the window management server 123.

611, the window management server 123 displays a target application window in response to the application start completion notification.

In an embodiment of the present application, the window management server 123 responds to the application start completion notification to place the target application window on the foreground window of the host 12 for display.

612, wms module 113 obtains the application image content, creates a layer window from the application image content, and sends a notification of creation of the layer window to window management client 119.

In an embodiment of the present application, the WMS module 113 obtains the application image content from the application 111 according to the hardware Vsync period signal synchronized by the host 11, creates a corresponding number of layer windows according to the number of layers of the application image content, and sends a notification of creating the layer windows to the window management client 119. In an embodiment of the present application, the create layer window informs of the number of layers comprising the application image content.

613, the window management client 119 sends a create layer window notification to the window management server 123.

In one embodiment of the present application, window management client 119 sends a create layer window notification to window management server 123 via the QEMU PIPE high speed channel.

614, the window management server 123 creates a target layer window according to the layer window notification, and returns the handle of the target layer window to the window management client 119.

In an embodiment of the present application, the target layer window created by the management server 123 is displayed on the target application window, that is, the target application window carries the target layer window.

615, window management client 119 sends the handle of the target layer window to WMS module 113.

In an embodiment of the present application, the window management server 123 creates a corresponding number of target layer windows according to the number of layers notified by the layer window, and returns a handle of each target layer window to the window management client 119. Window management client 119 sends the handle for each target layer window to WMS module 113.

616, wms module 113 may associate the layer window with the handle of the target layer window and send the handle of the target layer window to Goldfish OpenGL module 112.

In an embodiment of the present application, WMS module 113 associates a layer window with a handle of a corresponding target layer window and sends the handle of the target layer window associated with the layer window to Goldfish OpenGL module 112.

The 617,Goldfish OpenGL module 112 retrieves application image content from the application 111 and invokes the eglCreateWindowSurface interface 117 to create egl Window surface, where egl Window surface is associated with the target layer window of the host 12.

In one embodiment of the present application, the Vsync signal capture module 122 obtains the hardware Vsync period signal of the host 12 through the IDXGIOutput: waitForVBlank interface. The Vsync signal capturing module 122 transmits the hardware Vsync period signal to the HWC process of the client 11 through the communication module 13. The HWC process triggers the client 11 upon receipt of the hardware Vsync period signalVsync period signal of system. The application 111, upon receivingThe Vsync period signal of the system then sends the application image content to the Goldfish OpenGL module 112. The Goldfish OpenGL module 112 invokes the eglreatewindowsurface interface 117 to create egl Window surface associated with the target layer window of the host 12 and render the image content into egl Window surface associated with the target layer window of the host 12.

The 618,Goldfish OpenGL module 112 renders the application image content into egl Window surface associated with the target layer window of the host 12.

In an embodiment of the present application, the egl window surface includes a front-end buffer and a back-end buffer, and the Goldfish OpenGL module 112 renders the application image content at the back-end buffer of the egl window surface. The contents of the front-end buffer are displayed on the destination layer window of the host 12.

In an embodiment of the present application, the rendering module 124 creates a screen buffer as a back-end buffer according to the handle of the target layer window, renders the application image content in the back-end buffer, and returns the index of the address of the back-end buffer to the Goldfish OpenGL module 112.

619,Goldfish OpenGL module 112 invokes eglswappuffer interface 118 to send a display notification to send display module 125.

620, the sending and displaying module 125 displays the application image content in the target layer window corresponding to the handle of the target layer window in response to the display notification.

In an embodiment of the present application, the Goldfish OpenGL module 112 invokes an eglswappuffer interface to send an eglswappuffer instruction to the host, where the eglswappuffer instruction includes at least one of an index of an address of the back-end buffer or the application image content, and a handle of the target layer window of the host. The display module 125 displays the application image content on the target layer window of the host by exchanging the content of the front-end buffer with the content of the back-end buffer.

In one embodiment of the present application, the front-end buffer stores the current image content of the application 111.

In an embodiment of the present application, the method further comprises: the WMS module 113 of the client 11 acquires the state information of the layer window of the application 111, determines an operation instruction or a window management instruction of the layer window according to the state information, and sends the operation instruction to the window management client 119; the window management client 119 sends the operation instruction of the layer window to the window management server 123 of the host 12; the window management server 123 performs corresponding operations on the target layer window according to the operation instruction.

In an embodiment of the present application, the operation instruction includes a display instruction, and the window management server 123 displays the target layer window according to the display instruction. Specifically, the WMS module 113 of the client 11 obtains display status information of the layer window of the application 111, determines an operation instruction of the layer window as a display instruction according to the display status information, and sends the display instruction to the window management client 119; the window management client 119 transmits a display instruction to the window management server 123 of the host 12; the window management server 123 displays the target layer window according to the display instruction.

In an embodiment of the present application, the operation instruction includes a hiding instruction, and the window management server 123 hides the target layer window according to the hiding instruction. Specifically, the WMS module 113 of the client 11 obtains the hidden state information of the layer window of the application 111, determines the operation instruction of the layer window as a hidden instruction according to the hidden state information, and sends the hidden instruction to the window management client 119; window management client 119 sends a hidden command to window management server 123 of host 12; the window management server 123 conceals the target layer window according to the concealing instruction.

In an embodiment of the present application, the operation instruction includes a movement instruction, and the window management server 123 moves the target layer window according to the movement instruction. Specifically, the WMS module 113 acquires movement status information of the layer window of the application 111, determines an operation instruction of the layer window as a movement instruction according to the movement status information, and sends the movement instruction to the window management client 119; the window management client 119 transmits a move instruction to the window management server 123 of the host 12; the window management server 123 moves the target layer window according to the movement instruction.

In an embodiment of the present application, the operation instruction includes a set transparency instruction, an instruction thatThe window management server 123 sets the transparency of the target layer window according to the transparency setting instruction. Specifically, the client 11The system acquires transparency state information of a layer window of the application program, determines an operation instruction of the layer window as a transparency setting instruction according to the transparency state information, and sends the transparency setting instruction to a window management client 119; the window management client 119 sends a transparency setting instruction to the window management server 123 of the host 12; the window management server 123 sets the transparency of the target layer window according to the transparency setting instruction.

In an embodiment of the present application, the operation instruction includes a set Z-order instruction. The window management server 123 sets the Z order of the target layer window according to the Z order setting instruction. Specifically, the WMS module 113 acquires Z-order status information of the layer window of the application 111, determines an operation instruction of the layer window to be a Z-order setting instruction according to the Z-order status information, and sends the Z-order setting instruction to the window management client 119; the window management client 119 sends a set Z-order instruction to the window management server 123 of the host 12; the window management server 123 sets the Z-order of the target layer window according to the Z-order setting instruction.

In an embodiment of the present application, the method further comprises: the window management server 123 obtains operation information of a target application window of the application program from a user, determines an operation instruction of the target application window according to the operation information, and sends the operation instruction to the window management client 119; the window management client 119 transmits an operation instruction to the WMS module 113 of the client 11; the WMS module 113 performs corresponding operations on the application window and the layer window of the application program according to the operation instruction.

For example, the window management server 123 obtains the size adjustment information of the target application window of the application program from the user, determines the operation instruction of the target application window as the size adjustment instruction according to the size adjustment information, and sends the size adjustment instruction to the window management client 119; window management client 119 sends a resizing instruction to WMS module 113 of client 11; WMS module 113 resizes the application window according to the resizing instruction and resizes the layer window of the application according to the resizing instruction.

In an embodiment of the present application, after the window management server 123 creates the target application window, the method further includes: the window management server 123 sends a switch focus window notification to the window management client 119; window management client 119 sends a switch focus window notification to WMS module 113; WMS module 113 sets the application window associated with the target application window as the focus window.

Referring to fig. 9, a flowchart of a display method is shown in another embodiment of the present application. The method comprises the following steps.

71, client 11The application of the system creates a layer during the run-time.

72，The system captures the create layer message for the application and sends a create layer window notification to the host 12 based on the create layer message.

73, the host 12 creates a target layer window according to the layer window notification, and returns the handle of the target layer window to the client 11.

74，The system associates the layer of the application with the handle of the target layer window.

75, the host 12 synchronizes the hardware Vsync period signal to the client 11.

76，The system is based on the hardware Vsync periodAnd acquiring application image content of the application program by a signal, and rendering the application image content in the target layer window corresponding to the handle of the target layer window.

And 77, the host responds to the display notification sent by the Android system to display the application image content in the target layer window corresponding to the handle of the target layer window.

The embodiment of the application also provides a computer storage medium, in which computer instructions are stored, which when executed on an electronic device, cause the electronic device to execute the related method to implement the task management method in the embodiment.

The present application also provides a computer program product which, when run on a computer, causes the computer to perform the method of measuring pointing angle between devices in the above embodiments.

In addition, embodiments of the present application also provide an apparatus, which may be specifically a chip, a component, or a module, and may include a processor and a memory connected to each other; the memory is configured to store computer-executable instructions, and when the apparatus is running, the processor may execute the computer-executable instructions stored in the memory, so that the chip performs the method for measuring the pointing angle between devices in the above method embodiments.

The electronic device, the computer storage medium, the computer program product, or the chip provided in the embodiments of the present application are used to execute the corresponding methods provided above, so that the beneficial effects thereof can be referred to the beneficial effects in the corresponding methods provided above, and are not described herein.

From the foregoing description of the embodiments, it will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of functional modules is illustrated, and in practical application, the above-described functional allocation may be implemented by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to implement all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are illustrative, and the module or division of the units, for example, is a logic function division, and may be implemented in other manners, such as multiple units or components may be combined or integrated into another apparatus, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and the parts displayed as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated unit may be stored in a readable storage medium if implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions for causing a device (may be a single-chip microcomputer, a chip or the like) or a processor (processor) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Finally, it should be noted that the above embodiments are merely for illustrating the technical solution of the present application and not for limiting, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present application may be modified or substituted without departing from the spirit and scope of the technical solution of the present application.

Claims

1. An application display method applied to a host and a client communicatively connected to the host, the method comprising:

creating a layer in the running process of an application program of the Android system;

capturing a layer creating message of the application program by an Android system of the client, and sending a layer creating window notification to the host according to the layer creating message;

the host creates a target layer window according to the layer window notification, and returns a handle of the target layer window to the client; the Android system associates a layer of the application program with a handle of the target layer window;

the host synchronizes a hardware Vsync period signal to the client;

the Android system acquires application image content of the application program according to the hardware Vsync periodic signal, and renders the application image content in the target layer window corresponding to the handle of the target layer window;

and the host responds to the display notification sent by the Android system, and displays the application image content in the target layer window corresponding to the handle of the target layer window.

2. The application display method of claim 1, wherein the rendering the application image content in the target layer window corresponding to a handle of the target layer window comprises:

the Android system calls an eglCreateWindowSurface interface to create egl window surface associated with the target layer window of the host; and

The application image content is rendered on the egl window surface.

3. The application display method of claim 2, wherein the egl window surface includes a front-end buffer and a back-end buffer, the rendering the application image content on the eglwindow surface comprising:

the application image content is rendered at the back-end buffer of the egl window surface.

4. The application display method according to claim 3, wherein the host machine displaying the application image content in the target layer window corresponding to the handle of the target layer window in response to the display notification sent by the Android system includes:

the Android system calls an eglSwapBuffer interface to send an eglSwapBuffer instruction to the host, wherein the eglSwapBuffer instruction comprises at least one of an index of an address of the back-end buffer area or the application image content and a handle of the target layer window of the host; the host computer displays the application image content on the target layer window of the host computer by exchanging the content of the front end buffer and the back end buffer.

5. The application display method of claim 1, wherein the host synchronizing a hardware Vsync period signal to the client comprises:

the host acquires a hardware Vsync periodic signal sent by the host through a shared memory module with the client.

6. The application display method according to claim 1, wherein creating a layer during the running of the application program of the Android system includes:

the host receives the operation of starting the application program input by a user, and generates a notification of starting the application program;

the Android system runs the application program according to the notification of starting the application program sent by the host machine;

the application creates a layer during the run.

7. The application display method of claim 6, wherein the application creating a layer during execution comprises:

the application program calls a startActivity interface of the Android system to start the application program, and a notice for creating a layer is sent to a window management service module of the Android system in the starting process of the application program;

and the window management service module creates the layer window according to the notification of creating the layer.

8. The application display method of claim 1, wherein the method further comprises:

the client acquires state information of a layer window of the application program, determines an operation instruction of the layer window according to the state information, and sends the operation instruction to the host;

and the host machine carries out corresponding operation on the target layer window associated with the layer window according to the operation instruction.

9. The application display method of claim 1, wherein the method further comprises:

the client acquires display state information of a layer window of the application program, determines an operation instruction of the layer window as a display instruction according to the display state information, and sends the display instruction to the host;

and the host displays the target layer window according to the display instruction.

10. The application display method of claim 1, wherein the method further comprises:

the client acquires hidden state information of a layer window of the application program, determines an operation instruction of the layer window as a hidden instruction according to the hidden state information, and sends the hidden instruction to the host;

And hiding the target layer window by the host machine according to the hiding instruction.

11. The application display method of claim 1, wherein the method further comprises:

the client acquires the moving state information of the layer window of the application program, determines the operating instruction of the layer window as a moving instruction according to the moving state information, and sends the moving instruction to the host;

and the host machine moves the target layer window according to the movement instruction.

12. The application display method of claim 1, wherein the method further comprises:

the client acquires transparency state information of a layer window of the application program, determines an operation instruction of the layer window as a transparency setting instruction according to the transparency state information, and sends the transparency setting instruction to the host;

and the host machine sets the transparency of the target layer window according to the transparency setting instruction.

13. The application display method of claim 1, wherein the method further comprises:

the client acquires Z-order state information of a layer window of the application program, determines an operation instruction of the layer window to be a Z-order setting instruction according to the Z-order state information, and sends the Z-order setting instruction to the host;

And the host machine sets the Z sequence of the target layer window according to the Z sequence setting instruction.

14. The application display method of claim 1, wherein the method further comprises:

the host acquires operation information of a user on the target application window of the application program, determines an operation instruction of the target application window according to the operation information, and sends the operation instruction to the client;

and the client performs corresponding operation on the application window or the layer window of the application program according to the operation instruction.

15. A computer readable storage medium comprising computer instructions which, when run on an electronic device, cause the electronic device to perform the application display method of any one of claims 1 to 14.

16. An electronic device comprising a processor and a memory, the memory for storing instructions, the processor for invoking the instructions in the memory to cause the electronic device to perform the application display method of any of claims 1-14.

17. A chip coupled to a memory in an electronic device, wherein the chip is configured to control the electronic device to perform the application display method of any one of claims 1 to 14.