WO2017167125A1

WO2017167125A1 - Graphic composition method, information interaction method, and system

Info

Publication number: WO2017167125A1
Application number: PCT/CN2017/078035
Authority: WO
Inventors: 吴涛; 吴昌松
Original assignee: 阿里巴巴集团控股有限公司; 吴涛; 吴昌松
Priority date: 2016-03-31
Filing date: 2017-03-24
Publication date: 2017-10-05
Also published as: CN107291561B; CN107291561A

Abstract

Disclosed are a graphics composition method, an information interaction method, and a system. In the present application, a first control element in a first subsystem receives graphical data transmitted by a graphical object corresponding to the first control element in a graphics compositor of a second subsystem. The first control element transmits the received graphical data to a graphics compositor of the first subsystem. The first control element in the first subsystem corresponds to the graphical object in the graphics compositor of the second subsystem. The first control element provides a function for transmitting graphical data between the two subsystems, thereby realizing compatibility between different operating systems in a graphics processing layer.

Description

Graphic synthesis method, information interaction method and system

Technical field

The present application relates to the field of communications technologies, and in particular, to a graphics synthesis method, an information interaction method, and a system.

Background technique

The cloud operating system, also known as cloud OS, is an operating system supported by cloud computing and cloud storage technologies. The cloud operating system will be widely used in the future because it can integrate massive computing power and provide powerful resource allocation capabilities.

Cloud operating systems can be used for both traditional desktop systems (such as PCs) and mobile devices. Applications or services based on cloud operating systems, as well as applications or services based on traditional operating systems, will operate within the framework of the cloud operating system, which requires the cloud operating system framework to integrate applications or services based on different operating systems.

At present, there is no corresponding solution for how applications or services based on different operating systems are integrated at the graphics synthesis level or integrated at the control level within an operating system framework.

Application content

The embodiment of the present application provides a graphics synthesis method and system for implementing compatibility between different operating systems at a graphics processing level.

The graphic synthesis method provided by the embodiment of the present application includes:

The first control in the first subsystem receives the graphic data sent by the graphic object corresponding to the first control in the second subsystem graphics synthesizer;

The first control transmits the graphics data to a first subsystem graphics synthesizer.

Further, the method further includes the first subsystem graphics synthesizer receiving graphics data transmitted by the graphics object in the first subsystem.

Further, the method further includes: the first subsystem graphics synthesizer synthesizes the graphics data sent by the first control and the graphics data sent by the graphics object in the first subsystem.

Preferably, the first control is a plurality, and the first control sends the graphic data to the first Subsystem graphics synthesizer, including:

The plurality of first controls send the respective received graphic data to the graphic objects in the first subsystem;

The graphic object in the first subsystem sends the graphic data sent by the plurality of first controls to the first subsystem graphic synthesizer as the graphic data corresponding to the graphic object.

Preferably, the method further includes: the first control receives graphic object setting information sent by the application component in the first subsystem; the first control sends the received graphic object setting information to the first subsystem graphic synthesis Device.

The graphic object is an application window, and the graphic object setting information includes window attribute information.

Further, before the first control in the first subsystem receives the graphic data sent by the second subsystem graphics synthesizer, the method further includes: creating the first control.

The creating the first control includes: the application component in the first subsystem is configured to create a graphic object corresponding to the graphic object in the second subsystem for providing graphic data according to the request for using the graphic data in the second subsystem. The first control.

Further, after the first control sends the graphic data sent by the second subsystem graphics synthesizer to the first subsystem graphics synthesizer, the method further includes: destroying the first control.

Destroying the first control, comprising: the application component in the first subsystem receiving the information that the graphic object corresponding to the first control in the second subsystem is destroyed, and destroying the first control; or The application component in the first subsystem receives an indication to end the use of the graphics data in the second subsystem, and then destroys the first control corresponding to the graphics object in the second subsystem for providing the graphics data.

Further, before the first control in the first subsystem receives the graphic data sent by the graphic object corresponding to the first control in the second subsystem graphics synthesizer, the method further includes:

The first application component in the first subsystem sends control information to the first system service component in the first subsystem, the control information is used to indicate that the graphic data in the second subsystem is acquired;

The first system service component sends the control information to a second system service component of the second subsystem;

The second system service component sends the control information to a second one of the second subsystems.

Preferably, the first subsystem is a Host subsystem, and the second subsystem is a Guest subsystem.

The graphics synthesis system provided by the embodiment of the present application includes: a first subsystem graphics synthesizer, a second subsystem graphics synthesizer, and a first control in the first subsystem;

a first control in the first subsystem, configured to receive graphic data sent by the graphic object corresponding to the first control in the second subsystem graphics synthesizer, and send the graphic data to the first subsystem graphic synthesis Device.

Further, the first subsystem graphics synthesizer is further configured to: receive graphics data sent by the graphic object in the first subsystem.

Further, the first subsystem graphic synthesizer is further configured to: synthesize the graphic data sent by the first control and the graphic data sent by the graphic object in the first subsystem.

Preferably, the first control is a plurality of, the plurality of first controls send the respective received graphic data to the graphic object in the first subsystem;

The first subsystem graphics synthesizer is specifically configured to: receive graphic data sent by the graphic object in the first subsystem; wherein the graphic object corresponds to the graphic data sent by the plurality of first controls The graphics data is sent to the first subsystem graphics synthesizer.

Further, the first control is further configured to:

Receiving graphic object setting information sent by an application component in the first subsystem;

The received graphic object setting information is sent to the first subsystem graphic synthesizer.

Preferably, the first subsystem further includes an application component; and the application component in the first subsystem is configured to create the first control.

The application component in the first subsystem is specifically configured to: create a first control corresponding to the graphic object for providing graphic data in the second subsystem according to the request for using the graphic data in the second subsystem .

Preferably, the first subsystem further includes an application component; and the application component in the first subsystem is configured to destroy the first control.

The application component in the first subsystem is specifically configured to:

Receiving information that the graphic object corresponding to the first control in the second subsystem is destroyed, Destroying the first control; or,

Upon receiving an indication to end the use of graphics data in the second subsystem, the first control corresponding to the graphical object in the second subsystem for providing the graphics data is destroyed.

Preferably, the first subsystem further includes a first application component, and the second subsystem further includes a second application component;

a first application component in the first subsystem, configured to send control information to the first system service component in the first subsystem, where the control information is used to indicate that the graphic data in the second subsystem is acquired;

The first system service component is specifically configured to: send the control information to a second system service component in the second subsystem;

The second system service component is specifically configured to send the control information to a second application component in the second subsystem.

In the above embodiment of the present application, the first control in the first subsystem receives the graphic data sent by the graphic object corresponding to the first control in the second subsystem graphics synthesizer, and the first control will receive the graphic data. Sending to the first subsystem graphics synthesizer, it can be seen that, in the first subsystem, there is a corresponding first control corresponding to the graphic object in the second subsystem graphics synthesizer, and the first control provides between the two subsystems The transfer function of graphic data enables compatibility between different operating systems at the graphics processing level.

The embodiment of the present application provides an information interaction method and system for implementing compatibility between different operating systems at a control level.

The information interaction method provided by the embodiment of the present application includes:

The first system service component in the first subsystem receives the first information sent by the first application component in the first subsystem;

The first system service component sends the first information to a second system service component of the second subsystem;

The second system service component transmits the first information to a second one of the second subsystems.

Preferably, the method further comprises:

The second system service component receives the second information sent by the second application component, where the second information is obtained by the first application component according to the first information;

The second system service component sends the second information to the first system service component;

The first system service component sends the second information to the first application component.

Preferably, the first application component sends the first information to the first system service component in one or more of the following cases:

Need to use the graphic data in the second subsystem;

The application components in the second subsystem need to be notified for processing.

Preferably, the method further comprises: creating the first system service component.

The first system service component is created when the first application component is created.

Preferably, the method further comprises: destroying the first system service component.

The first system service component is destroyed when the first application component is destroyed.

Preferably, the information comprises: control information and/or data.

Preferably, the first subsystem is a Host subsystem, and the second subsystem is a Guest subsystem; or the first subsystem is a Guest subsystem, and the second subsystem is a Host subsystem.

The information interaction system provided by the embodiment of the present application includes:

a first system service component in the first subsystem, configured to receive first information sent by the first application component in the first subsystem;

The first system service component, configured to send the first information to a second system service component in the second subsystem;

The second system service component is configured to send the first information to a second application component in the second subsystem.

Preferably, the second system service component is further configured to: receive second information sent by the second application component, and send the second information to the first system service component; wherein the second information Obtained for the first application component according to the first information;

The first system service component is further configured to: send the second information to the first application component.

Preferably, the first application component is specifically configured to: in one or more of the following cases, The first system service component sends the first information:

Need to use the graphic data in the second subsystem;

Preferably, the information comprises: control information and/or data.

In the above embodiment of the present application, the first system service component in the first subsystem receives the information sent by the first application component in the first subsystem; the first system service component sends the information to the second subsystem. The second system service component, the second system service component sends the information to the second application component in the second subsystem, thereby implementing information interaction between the application components in the different subsystems.

DRAWINGS

1 is a schematic diagram of an operating system framework applicable to an embodiment of the present application;

2 is a schematic diagram of an information interaction architecture of a cross-system according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a dual-system compatible architecture of a graphics synthesis layer according to an embodiment of the present application; FIG.

4 is a schematic diagram of a user interface of a video playback application provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a graphics synthesis system according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of an information interaction system according to an embodiment of the present application.

detailed description

In order to provide a more natural and smooth experience transition for the user in the framework of the first operating system, the embodiment of the present application provides integration and unification at the graphics processing level in the framework of the first operating system. To provide users with a seamless dual system user experience. The embodiment of the present application can be applied to an application that needs to run across operating systems, and is also applicable to a scenario in which graphics data or control information needs to be transmitted between applications based on different operating systems.

The first operating system and the second operating system are different operating systems. Specifically, the first operating system may be a cloud operating system, and the second operating system may be a traditional operating system. More Specifically, the first operating system may be YunOS, and the second operating system may be Android.

The embodiments of the present application are described in detail below with reference to the accompanying drawings.

FIG. 1 is a schematic diagram of an operating system framework applicable to an embodiment of the present application. Taking YunOS as the first operating system and Android as the second operating system as an example, FIG. 1 shows a dual system compatible architecture within the YunOS framework.

As shown in Figure 1, the system framework includes two subsystems: Host and Guest. Among them, YunOS is the Host subsystem, Android is the Guest subsystem, and the two subsystems are isolated by the container technology. The Hardware Abstraction Layer (HAL) within the framework provides interfaces and driver services for graphical display and other aspects.

Glibc in the Host subsystem, the C runtime library, is the lowest API (Application Programming Interface) in Linux. Bioric in the Guest subsystem is the Linux kernel library for Android.

Libhybris is a compatibility layer that enables Glibc-based operating systems to reuse existing Android driver services; in graphics processing, libhybris also implements the EGL platform. EGL platform This is a backend-independent graphical platform that follows the EGL interface and multiple backend implementations.

The Gralloc module is located in the hardware abstraction layer and encapsulates all access operations to the framebuffer.

The InputManageerService (input management module) in the Guest subsystem is mainly used to monitor and manage input events.

SurfaceFlinger is the graphics synthesizer of the Guest subsystem; Wayland Compositor (or Weston) is the graphics synthesizer of the Host subsystem, where Weston is the reference implementation of the Wayland Compositor.

In the embodiment of the present application, the Wayland protocol can be used for communication between the graphics synthesizer of the Guest subsystem and the graphics synthesizer of the Host subsystem. The Wayland protocol defines how to communicate with the kernel and how to communicate with the client; Wayland reuses the graphics, input and output technologies of all Linux kernels.

In the embodiment of the present application, the Wayland protocol can be communicated between the graphic synthesizer of Andriod and the graphic synthesizer of YunOS, so that the graphics system in Android is overhead and bridged to the graphic system of YunOS, and the unified synthesis rendering of the dual system is realized. This provides better dual system compatibility and a seamless user experience for dual systems.

It should be noted that the above system architecture is only illustrated by YunOS and Andriod. For other types of operating systems, the compatibility and effective integration of dual systems at the graphics synthesis level can also be realized based on the above principles.

Based on the above framework, the embodiments of the present application provide dual system compatibility on the following two levels:

(1) Dual system compatibility scheme at the information interaction level

(2) Dual system compatibility scheme at the graphics synthesis level

In the actual application scenario, the above two dual system compatibility schemes may be used independently or in combination.

The following takes the first subsystem as the Host subsystem and the second subsystem as the Guest subsystem as an example to describe the two dual system compatibility schemes. Among them, the Host subsystem is YunOS, and the Guest subsystem is Android.

(1) Dual system compatibility scheme at the information interaction level

In order to achieve uniformity in interaction and control, the embodiment of the present application establishes a connection path for data and/or control information of the cross-operating system environment from the first subsystem to the second subsystem, that is, provides an IPC (Inter -process communication, mechanism to implement information exchange between application components (or processes) of different subsystems.

Referring to FIG. 2, an information interaction architecture provided by an embodiment of the present application is provided.

As shown in Figure 2, a Bridge Proxy is established in the Host subsystem, and a Bridge Service is established in the Guest subsystem. Among them, Bridge Proxy and Bridge Service are system service components.

Generally, the Android system uses the Binder mechanism as the IPC (Inter-Process Communication) mechanism, and the YunOS uses the socket or dbus as the IPC communication mechanism, in order to make information between application components (that is, application processes) of different operating systems. In the embodiment of the present application, the Bridage Proxy is used as a system service to convert the IPC mechanism between the two systems.

An application component is one of the components of an application. It is called an Activity in the Android system and a Page in the YunOS. It is the main body of the application. A process is an application or application component that is running.

The Bridge Service is usually a resident service and the Bridge Proxy can be created or enabled when needed. For example, in some embodiments, when an application component (process) is created in the Host subsystem, Bridge Proxy is created or enabled to enable information interaction between different subsystems through the Bridge Proxy. Further, when an application component (process) is destroyed in the Host subprocess, the Bridge Proxy is destroyed or closed. The application component (process) described in the embodiment of the present application indicates that the application component and the process are in an alternative relationship, and the process herein can be understood as an application component in an operational state.

In some embodiments, application components (processes) in the Host subsystem can send information, such as data and/or control information, to the Bridge Proxy, which sends the data and/or control information sent by the Bridge Proxy to Bridge. The Service, Bridge Service sends the data and/or control information to the application components (processes) in the Guest subsystem.

Among them, the Bridge Proxy and the Bridge Service can communicate based on the Binder mechanism. The Bridge Service can interact with application components (processes) in the Guest subsystem through intent or socket methods.

Further, after the application component (process) in the guest subsystem processes according to the received data and/or control information, the data and/or control information as a result of the processing may be sent to the Bridge Service, and the Bridge Service compares the data and / or control information is sent to the Bridge Proxy, the Bridge Proxy sends the data and / or control information to the application components (processes) in the Host subsystem.

In other embodiments, application components (processes) in the Guest subsystem may send information, such as data and/or control information, to the Bridge Service, which sends the data and/or control information to the Bridge Proxy, Bridge. The Proxy sends the data and/or control information to the application components (processes) in the Host subsystem.

Further, after the application component (process) in the Host subsystem processes the received data and/or control information, the data and/or control information as a result of the processing may be sent to the Bridge Proxy, and the Bridge Proxy may / or control information is sent to the Bridge Service, the Bridge Service sends the data and / or control information to the application components (processes) in the Guest subsystem.

In the above IPC architecture, the information exchanged between processes in different subsystems may be data or control information, and the control information may include an event, or request information, or window attribute information.

In the specific implementation, based on the above IPC architecture, when the application component in the Host subsystem If you need to use the data of the graphic object in the Guest subsystem, or you need to notify the application component (process) in the Guest subsystem to perform some or some operations, you can send the control information to the Bridge Service through Bridge Proxy, by Bridge. The service is sent to the application component (process) in the guest subsystem, and the application component (process) performs corresponding processing according to the control information, and sends the processing result to the Bridge Proxy through the Bridge Service, and is sent by the Bridge Proxy to the Host subsystem. Application component (process).

The above information interaction method can be applied to some functions of an application that need to be implemented based on the Host subsystem, and some functions need to be based on the scenario implemented by the Guest subsystem. For example, taking a video playback app as an example, when a user submits a request to play a movie through the video playback app user interface, the event is acquired by the video playback app in the process of the Host subsystem due to the video content. Provider encryption and copyright restrictions, only provide the SDK (Software Development Kit) of the Android platform, so the logic related to video content is developed in the Android environment, that is, the object requested by the event is in the Guse subsystem. Graphical data, therefore, the process of the video playback app in the Host subsystem sends the event to the Bridge Proxy, which sends the event to the Bridge Service, which is sent by the Bridge Service to the video playback app in the Guest subsystem. Thus, the process of the video playback app in the Guest subsystem sends the requested video data to the graphics synthesizer of the Host subsystem for processing.

It can be seen that the IPC architecture provided by the embodiment of the present application implements information exchange across operating systems, so that different operating systems are merged at the control level, thereby making the application-level fusion interactive experience seamless.

(2) Dual system compatibility scheme at the graphics synthesis level

FIG. 3 is a schematic diagram of a dual-system compatible architecture of a graphics synthesis layer according to an embodiment of the present application.

As shown in FIG. 3, a layer is created in the graphics synthesizer SurfaceFlinger of the Guest subsystem, and only two layers are exemplarily shown in the figure, namely layer1 and layer2.

In the Host subsystem, p_surface1 and p_surface2 are created, where p_surface1 corresponds to layer1 in the Guest subsystem and corresponds to surface1 in the graphics synthesizer of the Host subsystem (not shown in the figure) for connecting layer1 And the graphics synthesizer Weston of the Host subsystem, more specifically for connecting surface1 in layer1 and Weston; P_surface2 corresponds to layer2 and corresponds to surface2 in the graphics synthesizer of the Host subsystem (not shown in the figure), the graphics synthesizer Weston for connecting layer2 and the Host subsystem, more specifically for connecting layer2 and Surface2 in Weston.

P_surface can be a control, which is used to connect the layer synthesizer of the layer and Host subsystem in the guest subsystem graphics synthesizer, as a proxy of the layer in the guest subsystem, and pass the graphics data of the guest subsystem to the host sub The graphics synthesizer of the system. Specifically, layer1 can send graphics data to p_surface1, p_surface1 can send graphics data sent by layer1 to the graphics synthesizer Weston of Host subsystem; layer2 can send graphics data to p_surface1, and p_surface2 can send graphics data sent by layer2 The graphics synthesizer Weston sent to the Host subsystem.

Further, in the Host subsystem, the application component (process) can also create a local surface in the Host subsystem, and send the graphic data in the Host subsystem to the graphic synthesizer Weston of the Host subsystem through the local surface. Where local surface is the surface in the Host subsystem. In this way, the graphics synthesizer of the Host subsystem can synthesize the graphics data of the Host subsystem, and synthesize the graphics data of the Guest subsystem, and realize the graphics synthesis across operating systems.

Further, in the Guest subsystem, the application component (process) can also create layer3. Layer3 can send the graphics data in the Guest subsystem to the graphics synthesizer Weston in the Host subsystem, and the graphics data is synthesized by Weston.

Based on the characteristics of the Android framework, most of the graphics data is submitted to the Surfaceflinger for composition in the form of a layer. Therefore, the embodiment of the present application uses layer content as the basic unit of the integration of the two subsystems, that is, the graphic data sent through the p_surface may be The layer content, more specifically, may be the graphics data stored in the GraphicBuffer queue corresponding to the layer. A layer is a graphical object created by an application process. Generally, a layer represents a layer, and multiple layers can be superimposed in a Z-order order to form a user interface. A layer can correspond to a GraphicBuffer queue, and a GraphicBuffer queue contains multiple GraphicBuffers. In this embodiment, the GraphicBuffer is created by the application component in the process of the Guest subsystem, where the graphic data is stored, such as the graphic data stored by the process of the application component in the Guest subsystem.

Of course, the architecture of Figure 3 is described by YunOS and Android, and the graphics data is The layer is submitted to the Surfaceflinger for synthesis. In other cases, the graphics object in the Guest subsystem and the control of the Host subsystem graphics synthesizer can be connected to other types of graphic objects. The data is sent to the Host Subsystem Graphics Synthesizer for synthesis.

Further, the p_surface may further receive graphic object setting information sent by the application component in the Host subsystem, and send the graphic object setting information to the graphic synthesizer Weston of the Host subsystem. Weston can set or update graphic objects based on the graphic object setting information. Specifically, the graphic object may be an application window, and the graphic object setting information may be window attribute information, and Weston may update the corresponding window according to the window attribute information, such as changing a window transparency.

Specifically, the p_surface may provide an interface for calling the application component in the Host subsystem. When the application component calls the graphic object to process the related interface, the corresponding operation of the interface is performed to implement the graphic object setting, such as window property update.

As an example, after the application component in the Host subsystem receives the video window attribute update event (which can be triggered by the user by adjusting the video window size or transparency, the application component sends the updated window attribute information to the video) The surface of the window corresponds to p_surface1, and the window attribute information is sent to Weston by p_surface1, and Weston redraws the window according to the window attribute information.

The lifecycle of p_surface can be managed by application components in the Host subsystem.

Specifically, taking p_surface1 as an example, the application component in the Host subsystem may create p_surface1 according to the request for displaying the graphics data obtained by the Guest subsystem; when the graphic object corresponding to p_surface1 in the Guest subsystem is destroyed, p_surface1 is destroyed.

In a further embodiment, when the application component in the Host subsystem receives an indication to end the use of the graphics data in the Guest subsystem, the p_surface1 corresponding to the graphics object in the Guest subsystem for providing the graphics data is destroyed. .

Preferably, p_surface can be created based on the classes provided by the system, such that the functional properties of the parent class (ie, the classes provided by the system) can be inherited. Further, considering different application scenarios, there may be different requirements. Therefore, when a p_surface is created based on a system-provided class, some parameters or attributes are reset in combination with the current application scenario.

For example, the video playback application can be based on the system when creating p_surface The class is created and set to the sub-surface of the h5surface, controlling the width, position, and so on.

For example, in other embodiments, graphics data sent by multiple p_surfaces may be sent to a local surface (ie, a surface in the Host subsystem) to obtain synthesized graphics data, which is sent to Weston by the local surface, and finally submitted. A surface to the weston.

The above graphic synthesis method can be applied to some applications, and some functions need to be implemented based on the Host subsystem, and some functions need to be implemented based on the Guest subsystem. Taking the video playback app as an example, when the video playback app acquires a video play event in the process in the Host subsystem (this event can be triggered by the user submitting a video play request through the user interface), create p_surface1, P_surface1 corresponds to the surface of the video window in Weston; further, the process of the video playback app in the Host subsystem sends the event to the video playback app in the guest subsystem (see the information interaction between operating system processes) In the foregoing embodiment, the process of the video playback app in the guest subsystem acquires video data from the network side according to the event, and the video decoded content is transmitted to the VideoView control through the MediaCodec (media decoder), and then submitted to the surfaceflinger, and passed through the p_surface1. The graphic data is sent to the weston for synthesis to be displayed in the video window.

When the video is played, the process in the guest subsystem of the video playback app sends the event to the process in the video subsystem, and the process in the video subsystem app destroys p_surface1 according to the event.

When the user requests to play the video again, it can be executed according to the above procedure to create p_surface again, and transfer the graphic data from the Guest subsystem to the graphics synthesizer in the Host subsystem through p_surface.

It can be seen that the p_surface in the Host subsystem receives the graphics data sent by the layer corresponding to the p_surface in the guest subsystem graphics synthesizer by the above-mentioned embodiment of the present application, and the p_surface will receive the received data. The graphics data is sent to the host subsystem graphics synthesizer. It can be seen that in the Host subsystem, there is a corresponding p_surface corresponding to the layer in the guest subsystem graphics synthesizer, and the p_surface provides the transmission function of the graphics data between the two subsystems. This achieves compatibility between different operating systems at the graphics processing level.

For a more clear understanding of the above-mentioned embodiments of the present application, the implementation process of the embodiment of the present application is described below by taking a video playback application as an example.

Tom's mobile phone uses the YunOS system, and its system architecture is shown in Figure 1. The video playback app is installed on the mobile phone. The video data downloading and decoding related functions in the application are based on Android, and the playback control related functions are implemented based on YunOS.

The process by which Tom uses the video playback app to watch a movie movie and the mobile phone's response to Tom's operational behavior are described below.

Tom's operational behavior 1:

Tom clicks the video playback app icon displayed on the main interface of the operating system on the mobile phone screen, the video playback app is activated, and the main interface of the video playback app is displayed on the screen of the mobile phone. The main interface is shown in FIG. 4, which mainly includes the video window 100. The play control bar 200 and the window control bar 300 are displayed. The play control bar 200 is provided with a play control button, and specifically includes a pause button 201, a stop button 202, a play button 203, and a process drag bar 204. A window maximizer key 301 and a window restore key 302 are disposed on the window control strip 300. The main interface of the video playback app also includes an exit key 400.

Based on the above operation behavior of Tom 1, click the event of the video playback app icon to trigger the Host subsystem to open the video playback app, and create a video playback app process in the Host subsystem and the Guest subsystem respectively. In the Host subsystem, create the local surface corresponding to the application, draw the main interface of the application through the weston and display it. In addition, in the Host subsystem, a Bridge Proxy is also created for the video playback app, and the Bridge Proxy can be connected to the Bridge Service corresponding to the video playback app process in the Guest subsystem for control information and/or data interaction.

Tom's operational behavior 2:

Tom selects a movie and submits a request to view the movie, after which the movie is played in the video window 100.

Based on Tom's above operation behavior 2, the video playback app process in the Host subsystem creates a p_surface; the video playback app process in the Guest subsystem creates a buffer queue, obtains video data of the movie from the network side, and performs the obtained video data. Decoding, the decoded video data is stored in the buffer queue.

In the Guest subsystem, the layer in the graphics synthesizer SurfaceFlinger sends the video data in the buffer queue to the p_surface; in the Host subsystem, the p_surface sends the video data to the graphics synthesizer weston, which combines the video data, and Displayed in the video window 100 by the display driver.

Tom's operational behavior 3:

Tom clicks on the window maximize button 301 on the window control bar 300 and the video window 100 is maximized.

Based on Tom's above operational behavior 3, in the Host subsystem, the video playback app process sends the relevant information of the maximized video window 100 (such as the indication information of the maximized window and/or the window attribute information, etc.) to the p_surface, which p_surface will The information is sent to weston, which repaints the video window 100 based on this information to maximize display.

Tom's operational behavior 4:

Tom clicks the pause button 201 in the playback control bar 200, and the movie being played in the video window 100 is paused.

Based on the above operation behavior of Tom 4, the video playback app process in the Host subsystem sends an event generated by clicking the pause button 201 to the Bridge Proxy, and the Bridge Proxy sends the event to the Bridge corresponding to the video playback app process in the Guest subsystem. Service, Bridge Service sends the event to the video playback app process in the Guest subsystem. The video playback app process responds to the event and no longer sends video data to the p_surface through the layer.

Tom's operational behavior 5:

Tom clicks the play button 203 in the play control bar 200, and the video data in the video window 100 resumes playing.

Based on Tom's above operation behavior 5, the video playback app process in the Host subsystem sends an event generated by clicking the play button 203 to the Bridge Proxy, and the Bridge Proxy sends the event to the Bridge corresponding to the video playback app process in the Guest subsystem. Service, Bridge Service sends the event to the video playback app process in the Guest subsystem. The video playback app process responds to the event and continues to send video data to the p_surface through the layer.

When the movie is played, the video window 100 is closed.

Based on the event, the video playback app process in the Guest subsystem sends the event to the Bridge Proxy through the Bridge Service, and the Bridge Proxy sends the event to the video playback app process in the Host subsystem; the video playback app process in the Host subsystem. In response to this event, the video window 100 is closed and the p_surface is destroyed.

Tom's operational behavior 6:

Tom clicks to exit the exit button 400 of the video playback app.

Based on the above operation behavior of Tom 6, the video playback app process in the Host subsystem acquires the event of clicking the exit key 400, destroys the Bridge Proxy, and closes the video playback app.

In summary, when the embodiment of the present application is applied to a scenario in which the Android and the YunOS are integrated, the Android application can be made transparent, and the Android application can compile the corresponding APK or binary code through the SDK or the NDK without performing an Android application. Too many improvements.

In addition, the embodiments of the present application can implement diversification of cross-system integration. For different types of Host SDKs, for example, h5, qt, domoless, native, etc., the input access method can be customized.

In addition, the embodiment of the present application provides a fusion of application-level content, which can display the content processed by the application process of YunOS and Android in one window, and organically integrate two different operating systems into one, and a unified interactive experience does not need to be performed. Switch windows to manage all content in a unified manner.

Based on the same technical concept, the embodiment of the present application provides a graphics synthesis system, which can implement the graphics synthesis process described in the foregoing embodiment.

5 is a schematic structural diagram of a graphics synthesis system according to an embodiment of the present application. The system may include: a first subsystem graphics synthesizer 51, a second subsystem graphics synthesizer 52, and a first one of the first subsystems. Control 53;

a first control 53 in the first subsystem, configured to receive graphic data sent by the graphic object corresponding to the first control in the second subsystem graphic synthesizer 52, and send the graphic data to the first subsystem Graphic synthesizer 51.

Further, the first subsystem graphics synthesizer 51 is further configured to: receive graphics data sent by the graphic object in the first subsystem.

Further, the first subsystem graphics synthesizer 51 is further configured to: synthesize the graphics data sent by the first control 53 and the graphics data sent by the graphics object in the first subsystem.

Preferably, the first control is a plurality of, the plurality of first controls send the respective received graphics data to the graphic objects in the first subsystem; correspondingly, the first subsystem graphics synthesizer 51 is specifically configured to: And receiving the graphic data sent by the graphic object in the first subsystem; wherein the graphic object sends the graphic data sent by the plurality of first controls as the graphic data corresponding to the graphic object to the first subsystem graphic synthesizer.

Further, the first control 53 is further configured to: receive graphic object setting information sent by the application component in the first subsystem; and send the received graphic object setting information to the first subsystem graphic synthesizer 51.

The application component in the first subsystem is specifically configured to: create a first control corresponding to the graphic object for providing graphic data in the second subsystem according to the request for using the graphic data in the second subsystem 53.

Preferably, the first subsystem further includes an application component; and the application component in the first subsystem is configured to destroy the first control 53.

The application component in the first subsystem is specifically configured to: after receiving information that the graphic object corresponding to the first control 53 in the second subsystem is destroyed, destroying the first control 53; or Upon receiving an indication to end the use of the graphical data in the second subsystem, the first control 53 corresponding to the graphical object in the second subsystem for providing the graphical data is destroyed.

Preferably, the first subsystem further includes a first application component, and the second subsystem further includes a second application component; correspondingly, the first application component of the first subsystem is configured to be in the first subsystem The first system service component 54 sends control information, the control information is used to indicate that the graphic data in the second subsystem is acquired; the first system service component 54 is specifically configured to send the control information to the second subsystem. The second system service component 55 is configured to send the control information to the second application component in the second subsystem.

Based on the same technical concept, the embodiment of the present application provides an information interaction system, and the system The information interaction method described in the foregoing embodiment can be implemented.

Referring to FIG. 6 , it is a schematic diagram of an information interaction system architecture provided by an embodiment of the present application. The system may include: a first system service component 61 in the first subsystem, and a second system service component 62 in the second subsystem, where:

a first system service component 61 in the first subsystem, configured to receive first information sent by the first application component in the first subsystem, and send the first information to a second system in the second subsystem Service component 62;

The second system service component 62 is configured to send the first information to a second application component in the second subsystem.

Preferably, the second system service component 62 is further configured to: receive the second information sent by the second application component, and send the second information to the first system service component 61; wherein, the The second information is obtained by the first application component according to the first information. Correspondingly, the first system service component 61 is further configured to: send the second information to the first application component.

Preferably, the first application component is specifically configured to: send the first information to the first system service component 61 in one or more of the following cases:

Need to use the graphic data in the second subsystem;

Preferably, the information comprises: control information and/or data.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

These computer program instructions can also be stored at a computer or other programmable data location. The device is readable in a computer readable memory that operates in a particular manner such that instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device implemented in a flow or a flow and/or block diagram of the flowchart The function specified in the box or in multiple boxes.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

While the preferred embodiment of the present application has been described, it will be apparent that those skilled in the art can make further changes and modifications to the embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and the modifications and

It will be apparent to those skilled in the art that various modifications and changes can be made in the present application without departing from the spirit and scope of the application. Thus, it is intended that the present invention cover the modifications and variations of the present invention.

Claims

A graphic synthesis method, comprising:

The first control in the first subsystem receives the graphic data sent by the graphic object corresponding to the first control in the second subsystem graphics synthesizer;

The first control transmits the graphics data to a first subsystem graphics synthesizer.
The method of claim 1 further comprising:

The first subsystem graphics synthesizer receives graphics data transmitted by the graphics object in the first subsystem.
The method of claim 2, further comprising:

The first subsystem graphics synthesizer synthesizes the graphics data sent by the first control and the graphics data sent by the graphics object in the first subsystem.
The method of claim 1, wherein the first control is a plurality, and the first control sends the graphic data to the first subsystem graphics synthesizer, including:

The plurality of first controls send the respective received graphic data to the graphic objects in the first subsystem;

The graphic object in the first subsystem sends the graphic data sent by the plurality of first controls to the first subsystem graphic synthesizer as the graphic data corresponding to the graphic object.
The method of claim 1 further comprising:

The first control receives graphic object setting information sent by an application component in the first subsystem;

The first control sends the received graphic object setting information to the first subsystem graphic synthesizer.
The method of claim 5 wherein said graphical object is an application window and said graphical object setting information comprises window attribute information.
The method of claim 1, wherein before the first control in the first subsystem receives the graphics data sent by the second subsystem graphics synthesizer, the method further comprises: creating the first control.
The method of claim 7 wherein creating the first control comprises:

An application component in the first subsystem creates a first control corresponding to the graphical object in the second subsystem for providing graphical data in accordance with a request to use the graphical data in the second subsystem.
The method of claim 1, wherein after the first control sends the graphic data sent by the second subsystem graphics synthesizer to the first subsystem graphics synthesizer, the method further comprises: destroying the first control.
The method of claim 9 wherein destroying said first control comprises:

The application component in the first subsystem receives the information that the graphic object corresponding to the first control in the second subsystem is destroyed, and then destroys the first control; or

The application component in the first subsystem receives an indication to end the use of the graphical data in the second subsystem, and then destroys the first control corresponding to the graphical object in the second subsystem for providing the graphical data.
The method of claim 1, wherein before the first control in the first subsystem receives the graphic data sent by the graphic object corresponding to the first control in the second subsystem graphics synthesizer, the method further includes:

The first application component in the first subsystem sends control information to the first system service component in the first subsystem, the control information is used to indicate that the graphic data in the second subsystem is acquired;

The first system service component sends the control information to a second system service component of the second subsystem;

The second system service component sends the control information to a second one of the second subsystems.
The method according to any one of claims 1 to 11, wherein the first subsystem is a Host subsystem and the second subsystem is a Guest subsystem.
An information interaction method, comprising:

The first system service component in the first subsystem receives the first information sent by the first application component in the first subsystem;

The first system service component sends the first information to a second system service component of the second subsystem;

The second system service component transmits the first information to a second one of the second subsystems.
The method of claim 13 further comprising:

The second system service component receives the second information sent by the second application component, where the second information is obtained by the first application component according to the first information;

The second system service component sends the second information to the first system service component;

The first system service component sends the second information to the first application component.
The method of claim 13 wherein said first application component sends first information to said first system service component in one or more of the following cases:

Need to use the graphic data in the second subsystem;

The application components in the second subsystem need to be notified for processing.
The method of claim 13 further comprising: creating said first system service component.
The method of claim 16 wherein said first system service component is created when said first application component is created.
The method of claim 13 further comprising: destroying said first system service component.
The method of claim 18 wherein said first system service component is destroyed when said first application component is destroyed.
The method according to any of claims 13 to 19, wherein the information comprises: control information and/or data.
The method according to any one of claims 13 to 19, wherein the first subsystem is a Host subsystem and the second subsystem is a Guest subsystem; or

The first subsystem is a Guest subsystem, and the second subsystem is a Host subsystem.
A graphics synthesis system, comprising: a first subsystem graphics synthesizer, a second subsystem graphics synthesizer, and a first control in the first subsystem;

a first control in the first subsystem, configured to receive graphic data sent by the graphic object corresponding to the first control in the second subsystem graphics synthesizer, and send the graphic data to the first subsystem graphic synthesis Device.
The system of claim 22 wherein the first subsystem graphics synthesizer is further configured to: receive graphics data transmitted by the graphics object in the first subsystem.
The system of claim 23, wherein the first subsystem graphics synthesizer is further configured to: synthesize the graphics data sent by the first control and the graphics data sent by the graphics object in the first subsystem.
The system of claim 22 wherein the first control is a plurality of The plurality of first controls send the respective received graphics data to the graphic objects in the first subsystem;

The first subsystem graphics synthesizer is specifically configured to: receive graphic data sent by the graphic object in the first subsystem; wherein the graphic object corresponds to the graphic data sent by the plurality of first controls The graphics data is sent to the first subsystem graphics synthesizer.
The system of claim 22 wherein said first control is further for:

Receiving graphic object setting information sent by an application component in the first subsystem;

The received graphic object setting information is sent to the first subsystem graphic synthesizer.
The system of claim 26 wherein said graphical object is an application window and said graphical object setting information comprises window attribute information.
The system of claim 22, wherein the first subsystem further comprises an application component;

An application component in the first subsystem for creating the first control.
The system of claim 28 wherein the application component of the first subsystem is specifically for:

A first control corresponding to the graphical object in the second subsystem for providing graphical data is created in accordance with a request to use the graphical data in the second subsystem.
The system of claim 22, wherein the first subsystem further comprises an application component;

An application component in the first subsystem for destroying the first control.
The system of claim 30 wherein the application component of the first subsystem is specifically for:

Receiving information that the graphic object corresponding to the first control in the second subsystem is destroyed, destroying the first control; or

Upon receiving an indication to end the use of graphics data in the second subsystem, the first control corresponding to the graphical object in the second subsystem for providing the graphics data is destroyed.
The system of claim 22, wherein the first subsystem further comprises a first application component, and the second subsystem further comprises a second application component;

a first application component in the first subsystem, configured to send control information to the first system service component in the first subsystem, where the control information is used to indicate that the graphic data in the second subsystem is acquired;

The first system service component is specifically configured to: send the control information to a second system service component in the second subsystem;

The second system service component is specifically configured to send the control information to a second application component in the second subsystem.
A system according to any one of claims 22 to 32, wherein the first subsystem is a Host subsystem and the second subsystem is a Guest subsystem.
An information interaction system, comprising:

a first system service component in the first subsystem, configured to receive first information sent by the first application component in the first subsystem;

The first system service component, configured to send the first information to a second system service component in the second subsystem;

The second system service component is configured to send the first information to a second application component in the second subsystem.
The system of claim 34 wherein:

The second system service component is further configured to: receive second information sent by the second application component, and send the second information to the first system service component; wherein the second information is The first application component is obtained according to the first information;

The first system service component is further configured to: send the second information to the first application component.
The system of claim 34, wherein the first application component is specifically configured to: send the first information to the first system service component in one or more of the following cases:

Need to use the graphic data in the second subsystem;

The application components in the second subsystem need to be notified for processing.
The system of any of claims 34 to 36, wherein the information comprises control information and/or data.
The system according to any one of claims 34 to 36, wherein the first subsystem is a Host subsystem and the second subsystem is a Guest subsystem; or

The first subsystem is a Guest subsystem, and the second subsystem is a Host subsystem.