WO2021244452A1

WO2021244452A1 - Multi-device interaction method and system

Info

Publication number: WO2021244452A1
Application number: PCT/CN2021/097004
Authority: WO
Inventors: 代海涛
Original assignee: 荣耀终端有限公司
Priority date: 2020-05-30
Filing date: 2021-05-28
Publication date: 2021-12-09
Also published as: CN111666075B; CN111666075A

Abstract

A multi-device interaction method and system. In the method, a first application program is installed in a first device (530, 630), and the first application program may not be installed in a second device (540, 640). The first device (530, 630) displays a user interface of the first application program, and sends UI description information of the user interface of the first application program to the second device (540, 640). The UI description information can be used for describing attributes of assemblies in the user interface of the first application program. The second device (540, 640) displays the user interface of the first application program according to the UI description information and in combination with a screen shape and size of the second device (540, 640). The second device (540, 640) can also detect a user operation on the user interface, and feeds the user operation back to the first device (530, 630). The first device (530, 630) can process a task indicated by the user operation. By means of the method, the second device (540, 640) does not need to install the first application program, but provides a service in the first application program for a user by means of interacting with the first device (530, 630), thereby effectively reducing development costs.

Description

Method and system for multi-device interaction

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on May 30, 2020, the application number is 202010480943.4, and the application name is "a method and system for multi-device interaction", the entire content of which is incorporated herein by reference Applying.

Technical field

This application relates to the field of terminal technology, and in particular to a method and system for multi-device interaction.

Background technique

With the increase in the number of information devices, there are more and more application scenarios for multi-device interaction. Among them, in the process of multi-device interaction, multiple devices can display a user interface corresponding to the same service. When any one of the devices detects a user operation, these multiple devices can perform related processing according to the user operation. In this way, multiple different devices can notify each other, exchange information, and control each other, so as to jointly complete an application scenario. For example, the mobile phone pushes incoming call notifications or instant messages to the screen of the smart watch and assists with vibration reminders, and the smart watch controls the music played on the mobile phone.

Currently, multiple devices install applications that can implement the same service. These multiple devices can run their respective installed applications to implement the above-mentioned interaction process. However, this method needs to install applications on multiple sides, and the development cost is relatively high.

Summary of the invention

This application provides a method and system for multi-device interaction. The first application is installed in the first device. The second device can display the user interface of the first application program by receiving the user interface UI description information of the first application program in the first device and act on the user operation on the user interface accordingly. In this way, the second device can provide the user with the service of the first application without installing the first application, thereby reducing the development cost of the developer developing the first application on the second device.

In the first aspect, an embodiment of the present application provides a multi-device interaction method. The method includes: a second device receives first UI description information from a first device; the first UI description information is used to describe the information of the first application. The first UI component, the first UI component is determined according to the first user interface of the first application; the first UI component is displayed on the first user interface with the first component attribute; the first application is The application installed on the first device; the second device can instantiate the first UI component according to the second component attribute and the executable underlying code of the first UI component; the second component attribute is according to the first UI component Determined by the UI description information, the executable underlying code of the first UI component is determined from the component library according to the component type of the first UI component, and the component library contains executables for drawing different types of components Low-level code; the second device can display a second user interface, and the second user interface can include the first UI component. The first UI component can be displayed in the second user interface with the aforementioned second component attribute; wherein, the first component attribute and the second component attribute both include the component type and one or more of the following parameters: component size, Component location, component appearance.

In this application, both the first device and the second device may be devices such as mobile phones, computers, smart watches, smart TVs, and smart cars. In different application scenarios, the first device and the second device may be different.

In some possible implementations, the first device may be a smart car, and the second device may be a smart watch. Among them, a fatigue driving detection system can be installed in a smart car. When it is detected that the user is driving in fatigue, the smart car can send UI description information containing warning information to the smart watch. The smart watch can display warning messages on the screen and assist with vibration reminders.

In other possible embodiments, the first device may be a mobile phone, and the second device may be a smart watch. When the mobile phone receives an incoming call notification or instant message, the mobile phone can send to the smart watch the UI description information corresponding to the user interface such as the incoming call notification interface or instant message. Smart watches can display incoming call notifications or instant messages on the screen.

In this embodiment of the application, the first device establishes a communication connection with the second device. Among them, the connection mode may include: classic Bluetooth, low energy Bluetooth, and near field communication.

In the embodiment of the present application, the second device includes a component library. The second device can obtain the component library through one or more of the following methods: the second device obtains the component library from the first device, the second device obtains the component library from the cloud (such as a content server), and the second device obtains the component library from the cloud (such as a content server). A component library is stored in the way.

In some embodiments, the component library included in the second device includes executable low-level codes for drawing some or all types of components.

In the embodiment of the present application, the UI description information, such as the first UI description information, is a description of each component in the user interface. The amount of data is relatively small. In this way, the first device may not require high bandwidth to transmit the UI description information to the second device, thereby reducing the requirement for data transmission bandwidth. In addition, the UI description information is not a program directly executable by the processor, which can reduce uncontrollable security risks caused by the direct download of the executable program by the second device.

With reference to the first aspect, the above-mentioned first user interface may be displayed on the first device.

For example, the first device is a mobile phone, and the second device is a smart watch. When the mobile phone receives an incoming call notification, the incoming call notification interface can be displayed on the mobile phone. And the smart watch can receive the UI description information of the incoming call notification interface from the mobile phone. Furthermore, the smart watch may display the components in the incoming call notification interface on the user interface according to the UI description information. For example, answer the component, hang up the component. And these components can respond to user operations. In this way, both the mobile phone and the watch can display an incoming call notification interface to remind the user to answer the call.

With reference to the first aspect, when a first user operation acting on the first UI component is detected in the second user interface, the second device can change the component size and/or component appearance of the first UI component in the second user interface. The appearance of the aforementioned components may include one or more of the following: color, foreground, and background.

Specifically, the above-mentioned first UI component may be associated with a listener. The listener can be used to monitor user operations that act on the first UI component, such as click, long press, and drag. When the listener detects that a user operation is acting on the first UI component, the second device can change the component size and/or component appearance of the first UI component. That is, the display state of the first UI component itself changes. Among them, the display state of the first UI component in this application may undergo one or more of the following changes: the size of the first UI component changes, the foreground icon of the first UI component changes, the first UI component and the first UI component change The size of the sub-components changes.

Exemplarily, when the UI description information includes a description of an application icon on the main interface of the mobile phone, the smart watch may display the application icon on the main interface of the first device according to the UI description information. For example, dial icon, WeChat icon, Google Maps icon and settings icon, etc. When a click operation on the Google Maps icon is detected, the Google Maps icon can become larger. That is, the Google Maps icon can be enlarged to indicate that the Google Maps icon is in a selected state.

In other words, the first UI component of the aforementioned second user interface can respond to user operations, which is different from a static picture.

With reference to the first aspect, in response to the detected user operation on the first UI component in the second user interface, the second device may send the first event to the first device. Wherein, the first event can be used to indicate the occurrence of the first user operation. The second device receives the second UI description information from the first device; the second UI description information is used to describe the second UI component of the first application, and the second UI component is based on the first application The third user interface is determined; the second UI component is displayed on the third user interface with the third component attribute; the second device can instantiate the second UI component according to the fourth component attribute and the executable underlying code of the second UI component The second UI component. The fourth component attribute is determined according to the second UI description information, and the executable underlying code of the second UI component is determined from the component library according to the component type of the second UI component. The second device may display a fourth user interface, and the fourth user interface may include the second UI component. The second UI component may be displayed on the fourth user interface with the above-mentioned fourth component attribute.

That is to say, in response to a user operation on the first UI component detected in the second user interface, the second device can jump to the next user interface or refresh the second user interface, and display the fourth user interface .

Exemplarily, when receiving UI description information from the incoming call notification interface of the mobile phone, the smart watch may display the components in the incoming call notification interface according to the UI description information. For example, answering components and hanging up components. When detecting a user operation that acts on the hang-up component in the smart watch user interface, the smart watch can send an event indicating that the hang-up component has a user operation to the mobile phone. Further, the smart watch can receive the UI description information from the phone hang-up interface of the mobile phone. The smart watch can display the components of the phone hang up interface according to the UI description information of the phone hang up interface. Among them, the hang-up interface may be displayed or not displayed on the mobile phone.

With reference to the first aspect, the second device can find the executable low-level code of the first UI component from the component library according to the component type of the first UI component; the second device can use the executable low-level code of the first UI component The underlying code obtains the component object of the component type of the first UI component; the second device can use the second component attribute to set one or more of the following one or more of the component object of the component type of the first UI component in the second user interface Parameters: component size, component position, component appearance, get the first UI component instantiated.

In some embodiments, the first UI component includes at least one sub-component, and the second device can find the executable underlying code of the first UI component from the component library according to the component type of the first UI component, and according to The component type of the subcomponent finds the executable underlying code of the subcomponent from the component library; the second device can use the executable underlying code of the first UI component to obtain the component object of the component type of the first UI component , And use the executable underlying code of the subcomponent to obtain the component object of the component type of the subcomponent; the second device can use the second component property to set the component object of the component type of the first UI component in the second user interface One or more of the following parameters in: component size, component position, component appearance, the first UI component to be instantiated, and the component object that uses the component property of the sub-component to set the component type of the sub-component in the second user One or more of the following parameters in the interface: component size, component position, component appearance, get instantiated subcomponents.

With reference to the first aspect, in a possible implementation manner, the first UI description information may include the above-mentioned first component attribute, and the second device may determine the second component according to the first component attribute and the screen information of the second device Properties; the screen information includes one or more of the following: screen size, screen shape, and screen resolution.

With reference to the first aspect, in a possible implementation manner, the first UI description information may include the foregoing second component attribute. The second component attribute may be determined by the first device according to the foregoing first component attribute and screen information of the second device. Wherein, the second device may send the screen information of the second device to the first device.

It can be seen that the second component attribute of the first UI component can be determined by using the screen information of the second device. In other words, the first device or the second device may determine the component size and component position of the first UI component displayed on the second device according to the screen information of the second device. In this way, not only the problem of screen mismatch is reduced, but a set of UI description information can be applied to multiple devices.

In some embodiments, the first UI component may be determined according to some or all of the components in the first user interface.

Exemplarily, the incoming call notification interface may include an avatar component, a contact component, a hang-up component, and an answer component. Among them, the avatar component can indicate the avatar of the contact. The contact component can indicate the name of the contact. The hang-up component can be used to hang up the phone, and the answering component can be used to answer the phone. When receiving the UI description information of the incoming call notification interface, the smart watch can display the components in the incoming call notification interface. The components in the incoming call notification interface may have a priority order. That is, the smart watch can display these components in descending order of their priority. When the screen size of the smart watch is not large enough to display all the components in the incoming call notification interface, components with low priority may not be displayed in the smart watch. For example, the priority order of the hanging up component and the answering component may be the first priority. The priority order that the contact component may be may be the second priority. The priority order that the avatar component can be may be the third priority. The smart watch can first display the hang-up component and the answer component, then display the contact component, and finally display the avatar component.

With reference to the first aspect, before the second device displays the second user interface, the method further includes: the second device may be in a screen off state;

or,

The second device may display a third user interface, the third user interface is a user interface of a second application, and the second application is an application installed on the second device.

With reference to the first aspect, the first application program may include a system application program, a home screen application program, and a third-party application program. The third-party application is an application downloaded and installed from a third-party application market. The system application programs are, for example, a dialing application, a short message application, and a contact application. The third-party applications are, for example, WeChat applications, sports health applications, and Google Maps applications.

In other words, the second device may receive UI description information from the user interface of the system application in the first device, UI description information of the main screen application user interface, and UI description information of the third-party application user interface. To display the user interface of the system application in the first device, the user interface of the home screen application, and the user interface of a third-party application, so that the user interface can be provided to the user by communicating with the first device without installing these applications. The services of these applications. In this way, developers do not need to develop application programs on the second device side, thereby effectively reducing development costs.

In the embodiment of the present application, the first UI component of the first application can be displayed on the first user interface of the first device with the first component attribute, and can also be displayed on the second user of the second device with the second component attribute. Interface. The first component attribute may be the same as the second component attribute. For example, the first device and the second device are two devices with the same screen information, and the first UI component may be displayed on the user interfaces of the two devices with the same component size, component position, and component appearance. The vertical orientation of the first component may be different from that of the second component. For example, the first device and the second device are a mobile phone and a smart watch, respectively, and the screen information of the two devices is different. The component size, component position, and component appearance of the first UI component displayed on the mobile phone and the smart watch may be different.

In a second aspect, the embodiments of the present application also provide a method for multi-device interaction. The method includes: a first device detects a first event and can send a first user interface UI description information to a second device; the first UI description The information can be used to describe the first UI component of the first application, the first UI component is determined according to the first user interface of the first application; the first UI component can be displayed on the first user interface as the first UI component A component attribute display; the first application is an application installed on the first device; the second device can instantiate the first UI component according to the second component attribute and the executable underlying code of the first UI component . The second component attribute may be determined according to the first UI description information, the executable underlying code of the first UI component may be determined from the component library according to the component type of the first UI component, the component library Contains executable low-level code for drawing different types of components. The second device may display a second user interface; the second user interface includes the first UI component. The first UI component can be displayed in the second user interface with the second component attribute. Wherein, the first component attribute and the second component attribute both include the component type and one or more of the following parameters: component size, component location, and component appearance.

With reference to the second aspect, in a possible implementation manner, the first device may determine the second device from multiple devices according to a first preset rule. These multiple devices have established a communication connection with the first device.

Specifically, the first device may determine the second device to interact with it according to the distance between the first device and the first device or the priority order between the multiple devices. In other words, the first device can select the closest device from a plurality of devices with which it has established a communication connection as the second device, and then send the user interface information in the first application to the selected second device. UI description information. Or, there is a priority order among multiple devices that have established a communication connection with the first device. The first device may select the device with the highest priority as the second device, and then send the UI description information of the user interface in the first application to the selected second device.

Exemplarily, in an application scenario where a user receives an incoming call notification while driving, the first device may be a mobile phone. Among them, both the smart car and the smart watch worn by the user have established a communication connection with the mobile phone, and in this application scenario, the priority of the smart car is higher than the priority of the smart watch. When the mobile phone receives the incoming call notification, the mobile phone can send the UI description information such as the incoming call notification interface to the smart car with higher priority. The smart car can display the components in the incoming call notification interface on the display screen according to the UI description information. In addition, the on-board computer can answer calls in response to user operations (such as touch operations, voice control operations).

With reference to the second aspect, in a possible implementation manner, the first device can detect the operation of the first user. The first user operation can be used to select the second device. The first device may determine that the device selected by the first user's operation is the second device.

Exemplarily, the first device may be a mobile phone, and a communication connection is established between the mobile phone and the user's tablet and smart watch. The first device can display a music playing interface. In response to a user operation of sending the UI description information of the music playing interface to the smart watch, the mobile phone may send the UI description information to the smart watch. The smart watch can display the music playing interface. The mobile phone can also send the audio data of the played music to the smart watch. In this way, the smart watch can play music and respond to user operations on the music playing interface.

In a third aspect, an embodiment of the present application provides a device including a display screen, a communication device, a memory, and a processor. Wherein: the display screen is used to display a user interface; the memory is used to store one or more programs; the processor is used to execute the one or more programs, so that the device implements any of the A possible implementation, or any possible implementation as in the second aspect.

In a fourth aspect, an embodiment of the present application provides a chip that is applied to a device. The chip includes one or more processors for invoking computer instructions to make the device perform any possible Implementation, or any possible implementation as in the second aspect.

In a fifth aspect, an embodiment of the present application provides a computer program product containing instructions, characterized in that, when the computer program product is run on a device, the device is caused to execute any possible implementation manner in the first aspect, or Such as any possible implementation in the second aspect.

In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, characterized in that, when the foregoing instructions are executed on a device, the foregoing device executes any possible implementation manner as in the first aspect, or Such as any possible implementation in the second aspect.

Understandably, the device provided in the third aspect, the chip provided in the fourth aspect, the computer program product provided in the fifth aspect, and the computer-readable storage medium provided in the sixth aspect are all used to execute the methods provided in the embodiments of the present application. . Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

Description of the drawings

FIG. 1A is a schematic diagram of the component composition of a user interface provided by an embodiment of the present application;

FIG. 1B and FIG. 1C are schematic structural diagrams of the logical relationship between components of a user interface provided by an embodiment of the present application;

2A to 2E, 3A to 3D, 4A, and 4B are schematic diagrams of a series of user interfaces for multi-device interaction provided by an embodiment of the application;

FIG. 5 is a flowchart of a method for multi-device interaction provided by an embodiment of the present application;

6A to 6E are schematic diagrams of some multi-device interaction user interfaces provided by embodiments of the present application;

FIG. 7 is a flowchart of a method for instantiating a component of a device according to an embodiment of the present application;

FIG. 8 is a flowchart of a method for determining whether a device is adapted to a screen according to an embodiment of the present application;

9A, FIG. 9B, FIG. 10A, and FIG. 10B are schematic diagrams of application scenarios for adapting components provided by an embodiment of the application to the screen of a device;

11A to 11C are schematic diagrams of other multi-device interaction user interfaces provided by embodiments of the present application;

FIG. 12 is a schematic diagram of a structure for developing a multi-device interaction system provided by an embodiment of the present application;

FIG. 13 is a schematic structural diagram of another development multi-device interaction system provided by an embodiment of the present application;

FIG. 14 is a schematic structural diagram of a device provided by an embodiment of the present application;

FIG. 15 is a schematic structural diagram of another device provided by an embodiment of the present application.

detailed description

The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and appended claims of this application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to also Including plural expressions, unless the context clearly indicates to the contrary. It should also be understood that the term "and/or" used in this application refers to and includes any or all possible combinations of one or more of the listed items.

At present, there are more and more application scenarios for realizing multi-device interaction. There are three main types of existing technologies for realizing multi-device interaction.

The first method is to install applications that can implement the same service on multiple devices to achieve multi-device interaction. For example, in an application scenario where a user uses WeChat to communicate with others on mobile phones and smart watches, the WeChat application needs to be installed on both mobile phones and smart watches. In this way, for the same application, developers need to develop applications that can be applied to different devices, and the development cost is relatively high.

The second method is to install an application on a device or deploy an application on the cloud (such as a server), and other devices can interact with the above one device by dynamically downloading an executable program (such as a hot patch). For example, an application is installed on a device, and multiple functions of the application are respectively contained in multiple program packages. When another device wants to provide a certain function of the application, the other device can run the program package corresponding to this function downloaded from the above-mentioned one device. However, there will be uncontrollable security risks when executing dynamically downloaded executable programs.

The third method is to install an application on a device. This device can run all applications and services and send the results of the operation (such as pixel information or drawing commands) to other devices. Other devices can display based on these running results to realize multi-device interaction. However, in this implementation manner, the operation result of the above one device does not take into account information such as the screen size of other devices, and the problem of screen mismatch may occur.

The embodiment of the present application provides a method for multi-device interaction. In this method, a first application (such as a dial-up application) is installed in the first device, and the first application may not be installed in the second device. When the first device interacts with the second device, the first device may send user interface (UI) description information of the user interface of the first application to the second device. When the aforementioned UI description information is obtained, the second device can run a service program to parse the UI description information, and use the component library to instantiate each component in the UI description information. When all the components in the UI description information are instantiated, the second device can display these components on the user interface according to the logical relationship between these components. Among them, the aforementioned component library contains executable low-level codes that can be used to draw various components. Further, when a user operation acting on the second device is detected, for example, a user's touch operation, text input operation, etc., the second device may feed back the user operation to the first device. The first device can process the user operation.

In some possible multi-device interaction application scenarios, the first device may be a smart car, and the second device may be a smart watch. Among them, a fatigue driving detection system can be installed in a smart car. When it is detected that the user is driving in fatigue, the smart car can send UI description information containing warning information to the smart watch. The smart watch can display warning messages on the screen and assist with vibration reminders. Alternatively, the first device may be a mobile phone, and the second device may be a smart watch. When the mobile phone receives an incoming call notification or instant message, the mobile phone can send to the smart watch the UI description information corresponding to the user interface such as the incoming call notification interface or instant message. Smart watches can display incoming call notifications or instant messages on the screen. In these multi-device interaction application scenarios, the process of multi-device interaction can use the above-mentioned multi-device interaction method provided in the embodiment of the present application.

It should be noted that in the above-mentioned multi-device interaction application scenario, the first device may first determine the second device that needs to interact from multiple devices, and then send the UI description information of the first application to the second device . The first application program may not need to be installed in the second device. When receiving the aforementioned UI description information from the first device, the second device can provide the user with the service in the first application.

During the interaction between the first device and the second device, the user interface of the first application in the first device may be displayed on the second device. The second device can detect the user operation, and feed back the user operation to the first device. Then, the first device can process the task indicated by the user operation. In this way, the first device and the second device can notify each other, exchange information, and control each other. In addition, the second device does not need to install the first application program, and the developer does not need to develop the application program on the second device side, thereby effectively reducing development costs.

Compared with the first device, the second device may be a lightweight device with weaker computing capability, storage capability, or human-computer interaction capability, such as a smart watch. In this way, when the lightweight device does not install the first application, the lightweight device can also provide users with the services and applications of the first application. The aforementioned multi-device interaction method not only reduces development costs, but also reduces the requirements for various capabilities of the second device, such as computing capabilities, storage capabilities, or human-computer interaction capabilities.

In the embodiment of the present application, the above-mentioned first device and second device may both be devices such as mobile phones, computers, smart watches, smart TVs, and smart cars. In the foregoing method of interaction between the first device and the second device, both the first device and the second device may be multiple devices, which are not limited in the embodiment of the present application.

The following describes some concepts involved in the introduction of the method of multi-device interaction.

UI description information: text conforming to the UI description format, used to describe the attributes of each component in the user interface, and can also be used to describe the processing of input events or bound data by each component. Script or bytecode can be inserted in the UI description information. The aforementioned UI description format is a text format, for example, an XML format or a JSON format. The UI description format defines the syntax that needs to be followed to describe each component of the user interface.

The UI description information can be transmitted between the second device and the first device or the cloud (such as the application market). The UI description information is loaded and analyzed by the second device, and the component library is used to implement the display of each component in the user interface.

Service program: A resident program or module preset on the device. The service program can be used to load and parse the UI description information, so as to display the components in the UI description information on the user interface of the device. The service program can also be used to call a virtual machine or a script engine to execute the bytecode or script accessed in the UI description information.

Component: Refers to the basic UI elements that have independent appearance, can respond to input events, and can bind data. For example, buttons, sliders, check boxes, text boxes, icons, etc. The aforementioned input event may be a user operation on the component detected by the device, such as a touch operation, a text input operation, and so on. The above-mentioned binding data may indicate that there is a corresponding relationship between the component and the data. When detecting changes in the data bound to the component, the device can update the component. For example, for a component used to display the current time on the device, when the current time changes (for example, from 10:00 to 10:01), the device can update the time on the component. Alternatively, the device can also update the data when a change in the component is detected.

In the embodiment of the present application, the second device is associated with a listener on the component displayed on the user interface according to the UI description information. The listener can be used to monitor user operations that act on the component, such as click, long press, and drag. When the listener detects that a user operation is acting on the component, the component can respond to the user operation. The specific response manner may include: the display state of the component itself changes, for example, the appearance of the component changes, and the second device can jump to the next user interface or refresh the current user interface according to the user operation.

In some embodiments, in response to a user operation on a component displayed by the second device according to the UI description information, the size of the component may change. As shown in Figure 2E, when the UI description information contains a description of the application icon on the main interface of the first device (ie mobile phone), the second device (ie smart watch) can display the icon on the main interface of the first device according to the UI description information. Application icon. For example, the dial icon 721, the WeChat icon 722, the Google Maps icon 723, the setting icon 724, and so on. When a click operation acting on the Google Maps icon 723 is detected, the Google Maps icon 723 can become larger. In other words, the Google Maps icon 723 can be enlarged to indicate that the Google Maps icon 723 is in a selected state.

In other embodiments, in response to a user operation on a component displayed by the second device according to the UI description information, the foreground icon of the component may change. When the UI description information includes a description of each component in the music playing interface of the first device, the second device may display each component on the music playing interface according to the UI description information. For example, playback control keys. If the initial foreground icon of the playback control key is an icon indicating that music is playing, in response to a user operation of the playback control key acting on the foreground icon as an icon indicating that the music is playing, the playback control key can transform the foreground icon to indicate that the music is playing. The icon of the paused playback state. In other words, the playback control key can indicate that the music is in a playing or paused state by changing the foreground icon.

In another embodiment, in response to a user operation on a component displayed by the second device according to the UI description information, the size of the component and the size of the subcomponents contained in the component may both change. When the UI description information includes a description of the application icon on the desktop of the first device, the second device may display the application icon on the desktop of the first device according to the UI description information. Among them, these application icons may be subcomponents of page components in the desktop. In response to a long-press operation on the page component in the user interface of the second device, the page component and its subcomponents can all become smaller. Further, in response to a drag operation on the page component, the page component and its subcomponents can exchange positions displayed on the second device with other page components and its subcomponents.

The embodiment of the present application does not limit the manner in which the display state of the component itself changes in response to the user operation described above, and the display state of the component may undergo other transformations.

It can be seen that the components displayed by the second device according to the UI description information are associated with listeners. These components can respond based on user operations monitored by the listener, which is different from the static pictures displayed by the second device according to the drawing commands from the first device. .

Among them, the component may include a first component and a second component. The first component may be a component that contains other components inside. The relationship between the first component and these other components is the relationship between the parent component and the child component. And the first component can manage the arrangement of its internal sub-components. The second component may be a component that does not contain other components inside.

In the embodiment of the present application, the UI description information may include a description of the attribute of the first component. The attribute of the first component may include the type of the first component, and the type of the first component may include linear layout or relative layout. For example, a linear layout may specify that the alignment of its internal sub-components is left-aligned and the arrangement is top-to-bottom, and the sub-components in the linear layout are arranged one by one in a left-aligned and top-to-bottom manner. The attributes of the first component may also include the desired size, desired position, and appearance (such as background, foreground, border line) of the first component. The attributes of the first component may also include size and position information of the sub-components contained in the first component. For example, the width and height of the child component should match the width and height of the parent component as much as possible, and the width and height of the child component should match the width and height of the content in the child component as much as possible.

In the embodiment of the present application, the UI description information may include a description of the attribute of the second component. The attributes of the second component may include the type of the second component, and the type of the second component may include a button, a slider, a check box, a text box, and an icon. The attributes of the second component may also include the desired size, desired position, and appearance (such as background, foreground, border line) of the second component, and so on.

Fig. 1A exemplarily shows the component composition of a user interface.

As shown in FIG. 1A, the user interface includes a component 101, a component 102, a time component 103, a heart rate component 104, a weather component 105, and a temperature component 106.

Among them, the component 101 includes a component 102, a time component 103, and a heart rate component 104, and the component 101 is the first component. The component 102 includes a weather component 105 and a temperature component 106, and the component 102 is the first component. Since the time component 103, the heart rate component 104, the weather component 105 and the temperature component 106 do not contain sub-components, the time component 103, the heart rate component 104, the weather component 105 and the temperature component 106 are the second components. Both the component 101 and the component 102 can have a linear layout, and the sub-components contained in the component 101 and the component 102 can be arranged in a manner of left-aligned, left-to-right, and automatic line wrapping. Both the time component 103 and the heart rate component can be text boxes, which are used to display the current time and heart rate data, respectively. The weather component 105 may be an icon for indicating current weather conditions (such as sunny, cloudy, heavy rain, etc.). The temperature component 106 can be a text box for displaying the current ambient temperature.

FIG. 1B shows the logical relationship of the components in the UI description information of the user interface shown in FIG. 1A. As shown in FIG. 1B, the first component 101 includes sub-components 102-104. And the sub-component 102 of the first component 101 is also the first component. The first component 102 includes a sub-component 105 and a sub-component 106.

The second device (smart watch) can run a service program to parse the UI description information, so as to determine the attributes of the components 101-106. When the first component 101 is instantiated, the second device (smart watch) can determine the size and position of the first component in combination with the screen information of the smart watch (such as shape, size, resolution, etc.), and display it on the smart watch screen as shown in the figure The first assembly 101 shown in 1A. The smart watch can also determine the arrangement of the sub-components 102 to 104 in the first component 101, and the size and position of these sub-components in the smart watch. When the first component 102 is instantiated, since the smart watch determines the size and position of the component 102 when instantiating the first component 101, the smart watch can determine the background color, border line and other attributes of the first component 102, and the first component 102 in the size and position of the sub-assembly 105 and the sub-assembly 106.

When the second components 103 to 106 are instantiated, the smart watch can display the time component 103, the heart rate component 104, the weather component 105 and the temperature component 106 as shown in FIG. 1A on the screen of the smart watch according to the attributes of these second components.

In the embodiment of the present application, the logical relationship between the components in the UI description information can be described through the UI view tree.

Figure 1C shows a UI view tree. The UI view tree may indicate the logical relationship of the components in the UI description information as shown in FIG. 1A and FIG. 1B. As shown in FIG. 1C, node 101 and node 102 may represent component 101 and component 102, respectively, and node 103, node 104, node 105, and node 106 may represent time component 103, heart rate component 104, weather component 105, and temperature component 106, respectively. The relationship between node 101 and node 102, node 103, and node 104 is a parent node and a child node. Similarly, the relationship between the node 102 and the node 105 and the node 106 is a parent node and a child node.

That is to say, when each component in the UI description information is instantiated, the second device can display each component on the user interface according to the aforementioned UI view tree. The embodiment of the present application does not limit the manner in which the user describes the logical relationship between the components in the UI description information.

Component library: a module that contains executable low-level code (such as machine code, bytecode, etc.) corresponding to each component. Among them, the above executable low-level code can be used to draw various components on the user interface. The above-mentioned machine code can be directly run on the central processing unit (CPU) of the device. The above bytecode can be run in the virtual machine of the device. The executable low-level code in the component library can include an instruction part and a data part. When running the service program to parse the UI description information, the device can determine the attributes of the component. Further, the device can obtain the executable low-level code corresponding to the component from the component library according to the type of the component in the above attribute, and initialize the data part of the executable low-level code according to other attributes of the component. When the data part of the executable low-level code corresponding to the component is initialized and the device runs the initialized executable low-level code, the device can instantiate the component, that is, draw the component on the user interface. The component has attributes that describe the component in the UI description information. For example, a button with a background color of gray, a black border line, and a thickness of 2px.

The component library can be stored in the first device or the second device in a preset manner. In addition, the component library may also be downloaded to the first device when the first device downloads the application program, or sent to the second device when the first device sends UI description information to the second device. The embodiment of the application does not limit this.

In some embodiments, the aforementioned component library stored in the first device and the second device in a preset manner or in a download manner may include executable low-level codes corresponding to a part of the components. This part of the components provides services for the first device and the second device for users, and displays the components required by the corresponding user interface. In other words, the component libraries stored in the first device and the second device may not necessarily contain executable low-level codes corresponding to all components.

According to the above introduction to the method of multi-device interaction and some concepts involved, it can be seen that in the application scenario where the first device interacts with the second device, the first device can install the first application, and the second device There is no need to install the first application. This can effectively reduce development costs.

In addition, during multi-device interaction, the first device may send the UI description information of the user interface in the first application to the second device. The second device may parse the UI description information to display the user interface of the first application, and feedback the user operation detected on the user interface to the first device. The first device can process the task indicated by the user operation. In this way, the user can indirectly control the first device through the second device. The UI description information transferred between the first device and the second device is not an executable program, thereby reducing uncontrollable security risks.

When the second device is running the service program to parse the UI description information, it can obtain the second device’s own screen information (such as shape, size, resolution), and then determine the size and position of each component in the UI description information on the second device . In this way, determining the size and position of each component according to the shape and size of the screen of the device can not only reduce the problem of screen mismatch, but also this set of UI description information can be applied to a variety of devices.

Since the second device can parse the UI description information, high-overhead steps such as drawing various components of the user interface mainly rely on the component library, that is, the second device can directly run the executable low-level code in the component library to draw each component. This reduces the requirement on the processing capacity of the second device. In addition, compared to the transmission of lower-level pixel information or drawing commands, the transmission of UI description information occupies less bandwidth. It can be seen from the above two points that the method for multi-device interaction provided by the embodiments of the present application is very suitable for application scenarios that include lightweight devices.

When the first device interacts with the second device, the first device may send UI description information to the second device, and the UI description information may describe the attributes of the user interface components in the first application. Wherein, the above-mentioned first application program may be a system application or a third-party application.

The system application may be an application program preset in the device when the first device or the second device leaves the factory. For example, main interface application, dialing application, messaging application, contact application, camera, settings, etc.

The third-party application can be an application that the first device or the second device downloads and installs from the cloud (such as an application market) or other places. For example, sports health, WeChat, Google Maps, etc.

Hereinafter, taking the first application as a system application or a third-party application as an example, an embodiment in which the second device displays the user interface of the first application in the first device is introduced.

2A to 2E show that when the first application is a main interface application in the system application, the second device displays some user interfaces of the main interface of the first device.

Before the first device interacts with the second device, the first device can select the second device to interact with, and the first application.

In the embodiment of the present application, the first device is a mobile phone and the second device is a smart watch. Both the first device and the second device may be other devices, which are not limited in the embodiment of the present application.

As shown in FIG. 2A, the mobile phone may display a main interface 710. The main interface 710 may include a status bar 711, a component 715 with a list of commonly used applications, a calendar component 712, a weather component 713, a navigation bar 716, and other application icons 714. in:

The status bar 711 may include one or more signal strength components of a mobile communication signal, one or more signal strength components of a wireless fidelity (wireless fidelity, WiFi) signal, a Bluetooth component, a battery status component, and a time component.

The calendar component 712 can be used to indicate the current time, such as date, day of the week, hour and minute information, and so on.

The weather component 713 can be used to indicate the type of weather, such as cloudy to clear, and can also be used to indicate information such as temperature.

The component 715 with commonly used application icons may include: a dial icon 715A, an information icon 715B, a contact icon 715C, and a camera icon 715D. The component 715 with commonly used application program icons may include more or fewer application program icons.

Other application icons 614 may include: sports health icon 714A, Google Maps icon 714B, WeChat icon 713C, and settings icon 714D. The other application icons 714 may include more or fewer application icons.

The navigation bar 716 may include components used for system navigation, such as a return component, a main interface component, and a call-out task history component.

In response to a user operation on the status bar 711, such as a sliding operation to slide down from the status bar, the mobile phone can display a drop-down notification bar component 717 as shown in FIG. 2B on the main interface 710. The drop-down notification bar component 717 may include a linkage control icon 717A. The linkage control icon 717A can be used to select the second device that interacts with the first device. The drop-down notification bar component 717 may also include more content, such as a WiFi signal icon, a Bluetooth icon, a flashlight icon, and so on. The embodiment of the present application does not limit the content in the drop-down notification bar component 717.

In response to a user operation on the linkage control component 717A, such as a touch operation, the mobile phone can automatically search for a second device that can interact with the mobile phone, and display the linkage device option component 718 as shown in FIG. 2C on the main interface 710. The linked device option component 718 may include an icon indicating the second device searched by the mobile phone, for example, a smart watch icon 718A, a tablet icon 718B, and a car computer icon 718C. Among them, each second device indicated by the icon in the linked device option component 718 may be a device that has established a communication connection (such as a Bluetooth connection) with a mobile phone.

In response to a user operation of selecting one or more second devices in the linked device option component 718, the mobile phone may send the UI description information of the user interface in the first application to the one or more second devices.

The embodiment of the present application does not limit the user operation for calling the drop-down notification bar component 717.

Further, when a user operation, such as a touch operation, acting on the smart watch icon 718A is detected, the mobile phone can determine that the second device is a smart watch, and can display a linked application option component 719 as shown in FIG. 2D in the main interface 710.

The linkage application option component 719 may include a system application 719A and a third-party application 719B. Among them, the system application 719A may include a main interface icon, a dial icon, an information icon, and a contact icon. The third-party application 719B can include WeChat icons, sports health icons, and Google Maps icons. In response to a user operation acting on the icon of one or more applications in the linked application option component 719, the mobile phone can send the UI description information of the user interface in the one or more applications to the second selected in FIG. 2C. equipment.

Exemplarily, when it is determined that the second device is a smart watch, and a user operation acting on an icon of the main interface is detected, the mobile phone may send the UI description information of the main interface 710 to the main interface 710 when displaying the main interface 710 as shown in FIG. 2A smart watch.

When receiving the UI description information of the main interface 710, the smart watch may display the user interface 720 as shown in FIG. 2E. The user interface 720 may include icons of application programs displayed on the main interface 710, such as a dial icon 721, a WeChat icon 722, a Google Maps icon 723, and a setting icon 724. The smart watch may respond to a user operation of an icon of any application program in the user interface 720, and feedback the user operation to the mobile phone. The mobile phone can process the task indicated by the user's operation, and send the updated UI description information of the user interface to the smart watch. For example, in response to a user operation acting on the dial icon 721, the smart watch may feed back to the mobile phone that a user operation acts on the dial icon 721. Then the mobile phone can send the UI description information of the user interface 730 for dialing as shown in FIG. 3A to the smart watch. The smart watch can parse the UI description information of the user interface 730 and update the user interface 720 to display the user interface 740 as shown in FIG. 3B.

It is not limited to the icons of the applications displayed on the main interface 710, and the user interface 720 may also include more icons of other applications installed on the mobile phone.

It should be noted that the mobile phone can reselect or cancel the second device and the first application that interact with the mobile phone multiple times. In a possible implementation, when a touch operation on the linkage control component 717A is detected, the mobile phone can display the linkage device option component 718 as shown in FIG. 2C, and further display the linkage application option component 719 as shown in FIG. 2D . This allows the user to re-select the second device and the first application. When a long press operation on the linkage control component 717A is detected, the mobile phone can cancel all interactions with the second device. Not limited to the foregoing implementation manners, the embodiment of the present application may also reselect or cancel the second device and the first application program that interact with the mobile phone in other manners.

3A to 3D show schematic diagrams of the smart watch displaying the user interface of the dialing application.

When it is determined that the second device is a smart watch, and a user operation acting on the dial icon in FIG. 2D is detected, the mobile phone may send UI description information of the user interface 730 to the smart watch when the user interface 730 shown in FIG. 3A is displayed . The smart watch may display a user interface 740 as shown in FIG. 3B.

As shown in FIG. 3A, the user interface 730 may include a historical call record component 731, a dialing component 732, a telephone component 733, a contact component 734, and a favorite component 735. Among them, the historical call record component 731 may include call records before the current moment, such as the call record with Jane at 9:00. In response to a user operation that acts on any call record, such as a touch operation, the mobile phone can dial a user corresponding to the call record. The dialing component 732 can include a dialing keyboard, which can be used for the user to input a telephone number and make a call. The phone component 733 can be used for a mobile phone to display a user interface 730 as shown in FIG. 3A. The contact component 734 can be used to display the contact information (such as mobile phone number, email address, etc.) of the contact stored in the mobile phone. The favorite component 735 can be used to display the contact information of the favorite contact.

The user interface 730 may include more or fewer components, which is not limited in the embodiment of the present application.

The mobile phone can send the UI description information of the user interface 730 to the smart watch. The smart watch can parse the aforementioned UI description information, and use the component library to instantiate each component in the aforementioned UI description information, so that the size and position of each component are adapted to the screen of the smart watch. The smart watch may display a user interface 740 as shown in FIG. 3B. The user interface 740 may include a number component 741, a dial component 743, and a mobile phone main interface component 742. in:

The number component 741 can be used to display the phone number entered by the user, such as "131********".

The dialing component 743 may include a dialing keyboard, which can be used for the user to input a telephone number and make a call. Specifically, in response to a user operation acting on the dial keypad "1,2,3,4,5,6,7,8,9,*,0,#", the smart watch can display the corresponding symbol on the number component 741 . When a user operation acting on the component for making a call is detected, the smart watch can feed back the user operation to the mobile phone, and send the phone number in the number component 741 to the mobile phone. The mobile communication module in the mobile phone can send instructions to the base station to make a call. It is not limited to the above manner, and the smart watch in the embodiment of the present application may also make a call in other manners.

The mobile phone main interface component 742 can be used to display the main interface of the mobile phone on the smart watch. In response to a user operation on the main interface component 742 of the mobile phone, such as a touch operation, the smart watch may display a user interface 720 as shown in FIG. 2E. The components in the user interface 720 can be obtained by analyzing the UI description information of the main interface 710 by the smart watch. In this way, when the user interface 720 is displayed, the smart watch can open any application included in the main interface 710 of the mobile phone according to the detected user operation, so as to provide the user with a corresponding service.

In the embodiment of the present application, the smart watch may include physical buttons. When a user operation acting on the physical button is detected, the smart watch can display the main interface of the smart watch as shown in FIG. 6B. The embodiment of the present application does not limit the content included in the main interface of the smart watch, and does not limit the manner in which the smart watch displays the main interface of the smart watch.

As shown in FIG. 3C, when the mobile phone uses the module for mobile communication to send an instruction to the base station to make a call, the mobile phone can display a user interface 750. The user interface 750 may include components indicating the name of the call contact, a recording component, a video call component, a hang-up component, a speaker component, and other components. Among them, the recording component can be used to record calls. The video call component can be used to convert the current voice call into a video call. The hang-up component can be used to hang up the phone. The speaker assembly can be used to switch the module for playing sound from the earpiece to the speaker. The embodiment of the present application does not limit the components included in the user interface 750.

When the second device is a smart watch and the first application is a dialing application, the mobile phone may send the UI description information of the user interface 750 to the smart watch. The smart watch can parse the UI description information of the user interface 750, and use the component library to instantiate each component in the UI description information, so that the size and position of each component are adapted to the screen of the smart watch. The smart watch may display a user interface 760 as shown in FIG. 3D. The user interface 760 may include a component indicating the name of the call contact, a hang-up component, a speaker component, and the like. Among them, the hang-up component can be used to hang up the phone. The speaker assembly can be used to switch the module for playing sound from the earpiece to the speaker. Exemplarily, in response to a user operation of the hang-up component acting on the user interface 760, the smart watch may feed back the user operation to the mobile phone. Then the mobile phone can use the module for mobile communication to send instructions to the base station to hang up the phone. In this way, the user can control the mobile phone to hang up the call through the hang-up component on the smart watch.

The user interface 760 may also include a mobile phone main interface component 742. In this way, in response to a user operation acting on the main interface component 742 of the mobile phone, the smart watch can update the user interface 760 to the user interface 720 as shown in FIG. 2E. That is, the user can use the mobile phone main interface component 742 to jump directly from the user interface 760 to the user interface 720.

As shown in FIG. 4A, in response to a user operation acting on the Google Maps icon 714B in the main interface 710, the mobile phone can display the user interface 770. The user interface 770 may include a search box component 771, a positioning component 772, a clearing component 773, a start navigation component 774, and a map component 775. Among them, the search box component 771 can be used to input the address information that needs to be queried by text or voice. The positioning component 772 can be used to locate the current position of the user and display it in the map component 775. The clearing component 773 can be used to cancel the navigation path currently displayed in the map component 775. The start navigation component 774 can be used to start navigating the user according to the navigation path currently displayed in the map component 775. The map component 775 can be used to display information such as map information, the user's current location information, and navigation paths. The embodiment of the present application does not limit the components included in the user interface 770.

When the second device is a smart watch and the first application is Google Maps, the mobile phone can send the UI description information of the user interface 770 to the smart watch. The smart watch can parse the UI description information of the user interface 770, and use the component library to instantiate each component in the UI description information, so that the size and position of each component are adapted to the screen of the smart watch. The smart watch may display a user interface 780 as shown in FIG. 4B. The user interface 780 may include a map component 781 and a start navigation component 782. Among them, the map component 781 can be used to display information such as map information, the user's current location information, and navigation paths. The start navigation component 782 can be used to start navigation for the user according to the navigation path currently displayed in the map component 781. Exemplarily, when a user operation acting on the start navigation component 782 in the user interface 780 is detected, the smart watch may feed back the user operation to the mobile phone. The mobile phone can send the updated UI description information of the user interface to the smart watch according to the user operation. In this way, the smart watch can realize the navigation function for the user in the process of interacting with the mobile phone.

The foregoing second device may also be a device such as a tablet or a vehicle-mounted computer, which is not limited in the embodiment of the present application.

The following specifically takes the application scenario of receiving incoming call notification as an example to introduce the method of multi-device interaction.

Figure 5 shows a flow chart of a method for multi-device interaction. As shown in Fig. 5, the method includes steps S101 to S114. Among them, steps S105 to S107 are the specific process of the second device instantiating the component. Steps S109 to S114 are the interaction process between the first device and the second device after the second device detects the user operation.

S101. The first device establishes a communication connection with the second device.

The first device may establish a communication connection and perform communication in a wired or wireless manner. For example, the first device and the second device may communicate through classic Bluetooth, Bluetooth low energy, or near field communication, which is not limited in the embodiment of the present application.

S102. The first device displays a first user interface.

In an application scenario where the first device interacts with the second device, for example, the first device receives an incoming call notification, short message, or WeChat message, etc., and the first device is installed with a first application (such as a dial-up application, WeChat, etc.). The first device can run the first application to display the first user interface (such as an incoming call notification interface, a short message user interface, etc.).

Exemplarily, the first user interface may be a user interface (incoming call notification interface) 21 as shown in FIG. 6A. The user interface 21 may include a first component 210, an avatar component 211, a contact component 212, a hang-up component 213, and an answer component 214. in:

The first component 210 is the parent component of the avatar component 211, the contact component 212, the hang-up component 213, and the answer component 214.

In response to a user operation on the hang-up component 213, such as a touch operation, the first device can use the module for mobile communication therein to send an instruction to the base station to hang up the phone.

In response to a user operation, such as a touch operation, acting on the answering component 214, the first device may use the module for mobile communication therein to send an instruction to the base station to connect to the phone.

The user interface 21 may also include more or fewer components, which is not limited in the embodiment of the present application.

S103: The first device sends the first UI description information to the second device.

The first device contains first UI description information. The first UI description information is used to describe the attributes of each component in the first user interface.

When the first user interface is displayed, the first device may send the first UI description information to the second device.

Exemplarily, when the first user interface is the user interface 21 as shown in FIG. 6A, the first UI description information may include the first component 210, the avatar component 211, the contact component 212, the hang-up component 213, and the answer Description of the attributes of the component 214. The attribute of the first component 210 may include the type of the first component. The type of the first component 210 may be a linear layout or a relative layout. The attributes of the first component 210 may also include a desired size, a desired position, and appearance (such as background, foreground, border line), and the like. The attributes of the avatar component 211, the contact component 212, the hang-up component 213, and the answer component 214 may include the types of these components. The types of these components can be icons, text boxes, buttons, etc. The attributes of the avatar component 211, the contact component 212, the hang-up component 213, and the answer component 214 may also include a desired size, a desired position, and appearance (such as background, foreground, border line), and the like.

In some embodiments, before the first device sends the first UI description information to the second device, the first device may first send an inquiry instruction to the second device. The query instruction can be used to determine whether the second device can interact with the first device, that is, whether the second device can receive and parse the first UI description information. When receiving the information from the second device that it is determined that interaction is possible, the first device may send the first UI description information to the second device.

Exemplarily, when the first device (mobile phone) receives an incoming call notification and displays a user interface (incoming call notification interface) 21, the mobile phone may send an inquiry instruction to the second device (smart watch). If the call notification function is enabled in the smart watch, the smart watch can send confirmation information to the mobile phone. The determination information may indicate that the smart watch agrees to interact with the mobile phone. Then the mobile phone can send the first UI description information to the smart watch. If the smart watch cannot receive and parse the first UI description information, for example, the call notification function in the smart watch is turned off, or the smart watch contains a priority ordering of the services that can be processed, and the smart watch is processing The priority of the service is higher than the priority of receiving and parsing the first UI description information, and the smart watch may not respond to the inquiry instruction from the mobile phone. Then, when the smart watch does not receive a response to the inquiry instruction of the mobile phone, the mobile phone may not send the first UI description information to the smart watch.

Before the first device sends the UI description information to the second device, it may also determine that the first device and the second device can interact in other ways, which is not limited in the embodiment of the present application.

S104. The second device obtains the component library.

The component library contains executable low-level codes (such as machine code, bytecode, etc.) corresponding to each component (such as text boxes, buttons, etc.). Among them, the executable low-level code can include an instruction part and a data part. The second device can obtain the executable low-level code corresponding to the component from the component library according to the type of the component, and initialize the data part of the executable low-level code according to the attribute of the component. When running the executable low-level code with the initialized data part, the second device can draw the component on the user interface.

The component library can be stored in the second device in a preset manner, and can also be sent to the second device along with the UI description information. Among them, the second device may receive the UI description information sent from the first device, the cloud (such as a content server) and other devices. In other words, the second device can obtain the component library from the first device, cloud and other devices. In addition to the foregoing two ways of obtaining the component library, the second device may also obtain the component library in other ways, which is not limited in the embodiment of the present application.

There is no sequence relationship between step S104 and other steps, and the embodiment of the present application does not limit the occurrence time of step S104.

S105. The second device parses the first UI description information, and determines the attributes of each component in the first user interface.

When receiving the first UI description information, the second device can run the service program. The service program can be used to parse the first UI description information.

After parsing the first UI description information, the second device can determine the attributes of the components in the first user interface. Exemplarily, if the first user interface is the user interface 21 as shown in FIG. 6A, the second device may determine that the component 210 is the first component, the type in the attribute is linear layout, and the background color in the attribute is white Wait. The second device may also determine that the component 213 is the second component, the type in its attribute is a button, and its attribute indicates that the button is associated with a picture or the like.

S106. The second device obtains the executable low-level code of the first component from the component library according to the type of the first component in the first user interface, and uses the executable low-level code of the first component and the attributes of the first component. Instantiate the first component.

The component library contains executable low-level codes corresponding to multiple components. When determining the type of the first component (such as linear layout, relative layout, etc.), the second device can obtain the executable low-level code corresponding to the first component from the component library according to the type of the first component.

The executable low-level code corresponding to the component in the component library includes an instruction part and a data part. When the attributes of the first component are determined (for example, the background color is white, the border line is 2px, etc.), the second device can initialize the data part of the executable underlying code according to the attributes of the first component. In this way, the second device can complete the instantiation of the first component. And the instantiated first component has the attributes describing the first component in the first UI description information (for example, the background color is white, the border line is 2px, etc.).

It should be noted that for the topmost first component, the second device can determine the size and position of the topmost first component according to the screen information (such as shape, size, resolution, etc.) of the second device. The above-mentioned top-most first component means that the first component is not a subcomponent of any other components, that is, the first component is not included in any other components.

In addition, because the first component contains child components, when instantiating the first component, the second device can also match the parent component as much as possible according to the size and position of the first component and preset rules (such as the width and height of the child component) The width and height of the sub-component and the width and height of the sub-component should match the width and height of the content in the sub-component as much as possible) to determine the size and position of each sub-component.

In this way, the second device can display the components on the first user interface in a suitable size on a suitable position on the screen of the second device.

S107. The second device obtains the executable low-level code of the second component from the component library according to the type of the second component in the first user interface, and uses the executable low-level code of the second component and the attributes of the second component. Instantiate the second component.

When determining the type of the second component (such as a button, a text box, etc.), the second device can obtain the executable low-level code corresponding to the second component from the component library according to the type of the second component.

When the attributes of the second component are determined (for example, the background color is white, the border line is 2px, etc.), the second device can initialize the data part of the executable underlying code according to the attributes of the second component. In this way, the second device can complete the instantiation of the second component. And the instantiated second component has the attributes describing the first component in the first UI description information (for example, the background color is white, the border line is 2px, etc.).

S108. The second device displays the first component and the second component in the instantiated first UI description information.

When the instantiation of all the first components and all the second components in the first UI description information is completed, the second device may instantiate the first component and the second component in the first UI description information according to the UI view tree display. The UI view tree may indicate the logical relationship between the components in the first UI description information.

In some embodiments, the UI view tree may be a child node of an existing UI view tree, that is, each component in the first UI description information may be displayed in a certain position of an existing user interface . This existing user interface can be the user interface or desktop of any application. This existing UI view tree can indicate the logical relationship between components on this existing user interface.

The second device can embed the UI view tree as a child node in an existing UI view tree, and display each of the first UI description information at the position of the child node on the existing user interface. Components.

Exemplarily, a music playing interface is displayed on the second device. The music playing interface contains a pause component. In response to a user operation on the pause component, the second device may send an instruction indicating that there is a user operation on the pause component to the first device, and receive UI description information from the first device. The UI description information may include a playback component. The first UI view tree may indicate the logical relationship between the components in the UI description information. The second UI view tree may indicate the logical relationship between the components in the existing music playing interface. Then the above-mentioned first UI view tree may be embedded in the second UI view tree as a child node. In this way, the second device can display the playback component at the location of the pause component on the music playback interface, that is, replace the pause component with the playback component, and other components on the music playback interface are still displayed on the music playback interface.

In other embodiments, the UI view tree may be a separate UI view tree, that is, each component in the first UI description information may be separately displayed on a new user interface.

The second device can create a new user interface separately, and display the components in the first UI description information on the new user interface.

Exemplarily, as shown in FIG. 6B, a user interface 30 is displayed on the second device (smart watch). The user interface 30 may include a first component 301, a weather component 301A, an exercise data component 301B, and a time component 301C. Among them, the first component 301 is the parent component of the weather component 301A, the motion data component 301B, and the time component 301C. The weather component 301A can indicate the current weather and the current temperature. The exercise data component 301B can indicate the user's calories burned, number of steps, and heart rate information. The time component 301C can indicate time information, calendar information, and the like. The embodiment of the present application does not limit the components included in the user interface 30. When the first device (mobile phone) receives the incoming call notification, the mobile phone can display the user interface 21 as shown in FIG. 6A, and send UI description information to the smart watch. The smart watch can receive and parse the UI description information. When instantiating each component in the UI description information, the smart watch can display each component on the user interface according to the UI view tree. The UI view tree may indicate the logical relationship between the components in the UI description information. Since it can be determined that the UI view tree is a separate UI view tree, the smart watch can create a new user interface, such as the user interface 31 shown in FIG. 6C. In this way, the smart watch can display the components in this UI view tree on the user interface 31.

S109. The second device detects a user operation acting on the component.

Each component in the first UI description information can respond to input events. The aforementioned input event may be a user operation that acts on the component, etc.

When each component in the first UI description information is displayed on the user interface, the second device can detect user operations on the components, such as touch operations.

In some embodiments, when a user operation is detected, such as a touch operation on the screen of the second device, the second device may determine the component associated with the user operation. Specifically, the second device may first determine the screen coordinates of the touch operation acting on the screen, and convert the screen coordinates into component coordinates on the user interface, thereby determining the component acting on the touch operation, and associating the user operation with the component . The embodiment of the present application does not specifically limit the manner of determining the component associated with the user operation.

Exemplarily, if the first user interface is the user interface 21 as shown in FIG. 6A, the first UI description information may describe the attributes of each component in the user interface 21. When each component in the first UI description information is instantiated, the second device (smart watch) can display the user interface 31 as shown in FIG. 6C. The user interface 31 includes a first component 310, a contact component 311, a hang-up component 312, and an answer component 313.

When the user touches the location of the hang-up component 312 on the smart watch screen, the smart watch can obtain the screen coordinates of the user's operation on the screen. Each component on the user interface 31 may have component coordinates on the user interface 31. The smart watch can convert the screen coordinates of the user operation to the component coordinates on the user interface 31, so as to determine that the component associated with the user operation is the hang-up component 312.

S110. The second device sends an instruction to the first device.

When the component associated with the user operation is determined, the second device may send an instruction to the first device. The instruction may indicate the user operation detected by the second device.

Exemplarily, the second device may send the detected screen coordinates of the user operation, the text or audio input by the user, and other content to the first device. As shown in FIG. 6C, when the second device (smart watch) detects a user operation that acts on the hang-up component 312 in the user interface 31, such as a touch operation, the smart watch can send the touch operation to the mobile phone and act on the smart watch screen. Screen coordinates.

Alternatively, the second device may send the information of the component associated with the detected user operation to the first device. As shown in FIG. 6C, when the second device (smart watch) detects a user operation that acts on the hang-up component 312 in the user interface 31, such as a touch operation, the smart watch can confirm the component associated with the user operation in step S109. , It is determined that there is a user operation in the hang-up component 312. Further, the smart watch may send an instruction to the mobile phone to indicate that the hang-up component 312 has a user operation. The instruction can instruct the mobile phone to hang up the phone in the mobile communication network.

The embodiments of the present application do not limit the instructions sent by the second device to the first device, except for the information used to indicate the components operated by the user, the screen coordinates of the user operation detected by the second device, the text input by the user, audio, etc. The content can also be other.

S111. The first device displays a second user interface.

When receiving an instruction from the first device, the first device can respond to the instruction and display the second user interface.

Exemplarily, as shown in FIGS. 6A and 6D, when the first device (mobile phone) receives an incoming call notification and receives an instruction from the second device (smart watch) to hang up the phone, the mobile phone can display the user interface 22 . The user interface 22 may include a first component 210, an avatar component 211, a contact component 212, and a call ending component 217. Among them, the first component 210 is the parent component of the avatar component 211, the contact component 212, and the call end component 217. The call end component 217 can indicate that the call has been hung up.

In addition, the mobile phone can also use the mobile communication module in the mobile phone to send an instruction to hang up the phone to the base station, so as to hang up the phone in the mobile communication network. The mobile phone can first hang up the phone in the mobile communication network, and then display the user interface 22 as shown in FIG. 6D, or first display the user interface 22 as shown in FIG. 6D and then hang up the phone in the mobile communication network. The user interface 22 as shown in FIG. 6D can be displayed while hanging up the phone in the mobile communication network. The embodiment of the application does not limit this.

S112. The first device sends the second UI description information to the second device.

The second UI description information can be used to describe the attributes of each component in the second user interface.

Exemplarily, when the second device (smart watch) detects a user operation acting on the hang-up component 312 as shown in FIG. 6C, in addition to sending a hang-up instruction to the first device (mobile phone), the smart watch can also display A user interface indicating that the phone has been hung up (the user interface 32 shown in FIG. 6E). However, the dial application may not be installed in the smart watch, and the smart watch may not be able to update the user interface according to the detected user operation. But the smart watch can interact with mobile phones. The mobile phone stores UI description information corresponding to each user interface, and the mobile phone can send the second UI description information to the smart watch. The second UI description information may describe each component in the user interface where the phone has been hung up. In this way, the smart watch can update the user interface to display the user interface that the phone has been hung up.

It should be noted that there is no sequence between step S111 and step S112.

S113. The second device parses the second UI description information, and uses the component library to instantiate the first component and the second component in the second UI description information.

When receiving the second UI description information, the second device can run a service program to parse the second UI description information. The components in the second UI description information may include a first component and a second component. The second device may obtain the executable underlying code corresponding to the component from the component library to instantiate the first component and the second component in the second UI description information.

Exemplarily, if the second UI description information describes each component in the user interface 22 as shown in FIG. 6D, the second UI description information may include the first component 210, the avatar component 211, the contact component 212, and the call The description of the attributes of the component 217 ends. The avatar component 211, the contact component 212, and the call ending component 217 are included in the first component 210 and are the second component.

The second device parses the second UI description information and instantiates the first component and the second component by using the component library. For the specific implementation process, please refer to steps S105 to S107, which will not be repeated here.

S114. The second device displays the first component and the second component in the instantiated second UI description information.

When instantiating each component in the second UI description information, the second device may display each component in the second UI description information according to the UI view tree. The UI view tree may indicate the logical relationship between the components in the second UI description information. The UI view tree can be embedded in an existing UI view tree as a child node, or can be used as a separate UI view tree.

Exemplarily, if the second UI description information describes the components in the user interface 22 as shown in FIG. 6D, the second device (smart watch) may display the components in the second UI description information after instantiating each component in the second UI description information. The user interface 32 shown in FIG. 6E. The user interface 32 may include a first component 310, a contact component 311, and a call end component 314. Among them, the first component 310 is the parent component of the contact component 311 and the call end component 314. The call end component 314 can indicate that the call has been hung up.

For the specific display method, refer to step S108, which will not be repeated here.

According to the above-mentioned method of interaction between the first device and the second device, it can be seen that the second device may not install the first application (such as a dial-up application, an application for playing music, etc.), but by receiving and parsing the user in the first application The UI description information of the interface can provide users with the services in the first application. This can effectively reduce development costs. The second device can run a preset service program to parse the UI description information. The UI description information is a description of each component in the user interface, rather than an executable program. This not only reduces the bandwidth requirement for data transmission between the first device and the second device, but also reduces uncontrollable security risks caused by the second device directly downloading executable programs. In addition, when the second device parses the UI description information, it can determine the size and position of each component in the UI description information on the user interface of the second device according to the screen information of the second device itself. In this way, not only the problem of screen mismatch is reduced, but a set of UI description information can be applied to multiple devices. Since the second device can use the executable underlying code in the component library to draw each component, this reduces the requirement on the processing capability of the second device, and is suitable for application scenarios that include lightweight devices.

Here, an application scenario of receiving notification of incoming calls is taken as an example to illustrate an implementation method of the instantiated component of the second device.

When the first device (mobile phone) receives the incoming call notification, the mobile phone can send the first UI description information to the second device (smart watch). The first UI description information may describe various components in the user interface 21 as shown in FIG. 6A. Exemplarily, the first UI description information may specifically be:

Fig. 7 shows a flow chart of a method for instantiating a component of a second device. As shown in Figure 7, the method includes steps S201 to S207.

S201. The second device identifies the first component in the UI description information.

The description will be made by taking the UI description information as the above-mentioned first UI description information.

The second device (smart watch) parses the first UI description information, and can first identify the first component (such as component 210) in the first UI description information.

If the first component is recognized, the smart watch can execute step S202. Otherwise, the smart watch executes step S205.

Obviously, according to the above-mentioned first UI description information, the smart watch can recognize the first component 210 and execute step S202.

S202: The second device uses the component library to instantiate the first component according to the parsed attribute of the first component.

The second device parses the first UI description information to obtain the attributes of the first component. From the description of the first component 210 described above by the first UI description information, it can be determined that the type in the attributes of the first component 210 is linear layout, and the expected size in the attributes of the first component 210 and the arrangement of the sub-components contained therein. Cloth way. The second device may obtain the executable low-level code corresponding to the linear layout from the component library according to the type in the attribute of the first component 210, and initialize the data part of the executable low-level code by using the attribute of the first component. In this way, the second device can instantiate the first component 210 and determine the size and position of its internal sub-components on the user interface of the second device.

In the above process of determining the size and position of the sub-components inside the first component on the user interface of the second device, if the first component is the top-most first component in the UI description information, that is, the first component is not included in the Among other components, the second device can determine the size of the first component and the size and position of the sub-components of the first component on the user interface of the second device in combination with the size of the screen.

It should be noted that when it is detected that the screen of the second device is circular, the second device can determine the size of the first component and the size of the sub-components of the first component according to the method flowchart shown in FIG. 8 And location. The method includes steps S301 to S304.

S301: The second device sets the value of the initial margin of the subcomponent of the first component to 0.

The margin of the sub-component may represent the distance between each side of the sub-component and each side of the first component. Among them, margin can include top margin, bottom margin, left margin and right margin.

S302. The second device calculates the maximum possible width and maximum possible height of each subcomponent.

The second device may calculate the maximum possible width (maxWidth) and maximum possible height (maxHeight) of each subcomponent according to the width and height of the content contained in each subcomponent and the expected size of the subcomponent. In this way, the second device can ensure that a subcomponent with a width of maxWidth and a height of maxHeight can accommodate the content it contains.

S303. According to the maximum possible width and maximum possible height of the subcomponent, the second device calculates the width and height of the first component.

According to the maxWidth and maxHeight of the sub-components, and in accordance with preset rules, for example, the size of the first component should match the size of the screen as much as possible, and the size of the first component should match the size of the sub-component as much as possible. The second device can calculate the first component. The width (measuredWidth) and height (measuredHeight). The measuredWidth and measuredHeight are the final dimensions of the first component. The top-most first component is the top-most first component in the UI description information.

S304. The second device adjusts the value of the outer margin of each subcomponent according to the size of the screen and the size of the first component, and determines the size and position of each subcomponent.

When the final size of the first component is determined, the second device needs to calculate the size and position of the sub-components of the first component. Specifically, the size and position of the sub-component can be determined by the margin value of the sub-component. The second device may obtain the value of Inset according to the formula Inset=0.146*max[min(measuredWidth, screenWidth), min(measuredHieght, screenHieght)], and set the value of Inset to the new margin value. Among them, screenWidth and screenHieght represent the width and height of the screen, respectively.

According to the new margin value, the second device can determine the size and position of each subcomponent.

Exemplarily, the first component 210 is not included in other components, and is the top-most first component. When the second device (smart watch) instantiates the first component 210, it can determine the size and position of the first component 210 and the subcomponents 211 to 214 of the first component 210 on the smart watch user interface according to the above steps S301 to S304. Size and location. In this way, the smart watch can display the first component 310 and the subcomponents 311-313 of the first component as shown in FIG. 6C. The first component 310 may be the first component 210 after instantiation. The component 311 may be the contact component 212 after instantiation. The component 312 may be the hang-up component 213 after instantiation. The component 313 may be the answering component 214 after instantiation. The sizes and positions of the components 310 to 313 are adapted to the screen of the smart watch.

It should be noted that determining the size and position of each component above means determining the size and position of each component on the user interface of the second device.

As can be seen from the method flowchart shown in Figure 8, when the second device instantiates the top-most first component in the UI description information, it can determine the size of the first component in conjunction with the size of the second device’s screen. And the size and position of the sub-components of the first component. In this way, the first component and its subcomponents can be adapted to the screen of the second device, and the same set of UI description information can be applied to multiple different devices.

When the screen of the second device is circular, in addition to determining the size of the first component and the size and position of the sub-components of the first component according to the method flowchart shown in FIG. The method of adapting the circular screen to determine the size and position of each component is not limited in the embodiment of the present application.

When it is detected that the screen of the second device is of another shape (such as a rectangle), the second device can determine the size and position of each component according to the method of adapting each component to the screen of the other shape in the prior art. This embodiment of the present application I will not go into details about this.

In addition, in the above process of determining the size and position of the subcomponent of the first component, if the first component is not the topmost first component in the UI description information, that is, the first component is a subcomponent of a certain first component. When instantiating one of the above-mentioned first components, the second device has already determined the size and position of the first component. When instantiating the first component, the second device can determine the size and position of the sub-components of the first component. Specifically, according to the size of the first component, the desired size of the child components of the first component, and preset rules (such as the width and height of the child component as much as possible to match the width and height of the parent component, and the width and height of the child component as far as possible to match the child The width and height of the content in the component), the second device can determine the size and position of the sub-component of the first component.

S203: The second device recognizes the second component in the UI description information.

The description will still be made by taking the UI description information as the above-mentioned first UI description information.

After the first component 210 is instantiated according to step S202, the second device (smart watch) can further identify the sub-components of the first component 210 in the first UI description information. If a component that has not been instantiated can be identified, the smart watch can execute step S204. When the smart watch fails to identify a component that has not been instantiated, that is, the sub-components of the first component 210 have been instantiated, the second device can return to step S201 to identify other first components in the first UI description information .

S204. The second device uses the component library to instantiate the second component according to the parsed attribute of the second component.

When identifying the sub-components of the first component in step S202, the second device may instantiate these sub-components as sub-components of the second component. That is, the second device can instantiate sub-components that do not contain other components. For the sub-components of the first component, that is, the sub-components that contain other components, the second device can instantiate all the sub-components of the second component and then compare the sub-components of the first component according to step S201 and step S202. Perform instantiation, and further instantiate other components contained in the sub-components that are the first component according to step S203 and step S204.

Specifically, the second device may determine the executable low-level code corresponding to the sub-component from the component library according to the type of the sub-component, and initialize the data part of the executable low-level code with data related to the attribute of the component. When running the initialized executable low-level code, the second device can instantiate the sub-component, and display the sub-component at the position of the sub-component determined in step S202. The sub-component has attributes that describe the sub-component in the UI description information. For example, a button with a background color of gray, a black border line, and a thickness of 2px.

When the instantiation of a sub-component is completed, the second device may perform step S203 to identify other sub-components that are not instantiated in the first component including this sub-component. The second device may perform step S203 and step S204 in a loop until all subcomponents of the second component in the first component are instantiated. Then the second device can perform step S201 to identify other first components in the UI description information. If the other first components are not recognized, it may indicate that the second device has instantiated all the recognized first components and second components. In this way, the second device can execute step S205.

Exemplarily, if the UI description information is the aforementioned first UI description information, the second device (smart watch) may first recognize the avatar component 211 and instantiate the avatar component 211. The type of the avatar component 211 is an icon, and the attributes include a desired size and associated picture information. The smart watch can obtain the executable low-level code corresponding to the icon from the component library according to the type of the icon, and use the attributes of the avatar component 211 to initialize the data part of the executable low-level code. In this way, the smart watch completes the instantiation of the avatar component 211.

Then, the smart watch may return to step S203 to continue to identify the sub-components in the first component 210, such as the contact component 212, the hang-up component 213, and the answer component 214. The smart watch can instantiate the contact component 212, the hanging up component 213, and the answering component 214 according to the method of instantiating the avatar component 211.

When it is not recognized that the first component 210 has not yet been instantiated and is a sub-component of the second component, that is, the sub-components of the second component in the first component 210 are all instantiated, the smart watch may perform step S201, To identify whether there are other first components that have not been instantiated in the first UI description information. Wherein, the aforementioned first component that is not instantiated may include any sub-component of the first component that is the first component.

Since there is no other first component in the above-mentioned first UI description information, the smart watch may execute step S205.

S205: The second device judges whether the UI description information is parsed.

When the second device instantiates each first component and each second component in the UI description information, that is, the second device does not recognize the components that have not been instantiated, the second device can determine whether the UI description information has been parsed .

If the analysis of the UI description information is completed, the second device can determine the logical relationship between the components in the UI description information, and execute step S206.

If the resolution of the UI description information is not completed, the second device may perform step S207 to perform the resolution failure processing on the UI description information. Wherein, when the UI description information contains unrecognizable items, such as undefined tags or UI description information that does not conform to the UI description format, the second device cannot complete the analysis of the UI description information.

S206. The second device displays the instantiated first component and the second component.

For the method for the second device to display the instantiated first component and the second component, refer to step S108, which will not be repeated here.

It should be noted that, in some embodiments, before instantiating each component in the UI description information or before displaying each component in the UI description information, the second device may change the UI description information according to a preset filtering rule. Part of the components in the filter are filtered out. The second device may display the components that are not filtered out in the UI description information on the user interface. The above filtering rules can be used to filter components that are not important for various application scenarios. For example, components that cannot respond to input events or components that are not bound to data, etc. The embodiments of the present application do not specifically limit the foregoing filtering rules.

Exemplarily, the foregoing first UI description information includes a first component 210, an avatar component 211, a contact component 212, a hang-up component 213, and an answer component 214. When the second device (smart watch) receives the first UI description information, the smart watch can filter out the avatar component 211 according to the filter rule based on the size of its own screen. In this way, the smart watch can display the user interface 31 as shown in FIG. 6C. The user interface 31 displays a first component 210, a contact component 212, a hang-up component 213, and an answer component 214.

S207. The second device performs a parsing failure process on the UI description information.

The foregoing analysis failure processing may be that the second device displays information about the analysis failure on the user interface, etc. The embodiment of the present application does not specifically limit the method for processing the analysis failure.

In another implementation manner, the order in which the second device instantiates the components in the UI description information may be the order in which these components are described in the UI description information. Take the UI description information of the user interface 21 shown in FIG. 6A as an example for description. The order of describing these components in the UI description information is: the first component 210, the avatar component 211, the contact component 212, the hanging up component 213, and the answering component 214, respectively. Then, the second device can instantiate these components in the above sequence and display them on the user interface.

In another implementation manner, the order in which the second device instantiates the components in the UI description information may be determined by the priority order of the components in the UI description information. The priority order of the components in the UI description information can be preset. Still taking the UI description information of the user interface 21 shown in FIG. 6A as an example for description. The priority order of the components in the UI description information can be: the first component 210 has the highest priority; the hang-up component 213 and the answer component 214 have the same priority and are lower than the priority of the first component 210; contact component The priority of 212 is lower than the priority of the hanging up component 213 and the answering component 214; the priority of the avatar component 211 is lower than the priority of the contact component 212. Then, the second device can first instantiate the first component 210, then instantiate the hang-up component 213 and the answer component 214, then instantiate the contact component 212, and finally instantiate the avatar component, and display it in the user interface in the order of the above instantiations. These components are shown one by one. Wherein, if the screen size of the second device is limited, the second device may display components with a higher priority order according to the size of its own screen. As shown in FIG. 6C, since the screen of the second device (smart watch) is small, the second device may not display the above-mentioned avatar component 211 with the lowest priority on the user interface.

The embodiment of the present application does not limit the priority order of the components in the UI description information.

It should be noted that, in the foregoing implementation of determining the order of instantiating components according to the order of the components described in the UI description information and the priority order of the components in the UI description information, the second device can determine whether the UI description information is parsed Complete, and then decide whether to perform the analysis failure processing or display the instantiated component on the user interface according to the result of the completion of the analysis. For the specific implementation process of judging whether the UI description information is parsed, the parsing failure processing, and displaying the instantiated components can refer to steps S205 to S207 in the method embodiment shown in FIG. 7, which will not be repeated here.

The following describes an embodiment in which the components in the UI description information can be adapted to the device screen.

When a user operation acting on the contact icon 715C in the main interface 710 is detected, the first device (mobile phone) may display the user interface 810 as shown in FIG. 9A. The user interface 810 may include a contact list component 811 and an alphabetic index component 812. Wherein, the contact list component 811 may indicate the contact information (such as phone number, email) of each contact stored in the mobile phone. The contact information of each contact can be arranged in alphabetical order according to the first letter of the contact's name. The letter index component 812 can contain 26 letters of the alphabet, and can be used to find a contact whose name is the first letter based on one of the letters. The letters in the letter index component 812 can be arranged vertically in the order of the alphabet. The embodiment of the present application does not limit the components included in the user interface 810.

The mobile phone can send the UI description information of the user interface 810 to the second device (smart watch). The screen of the smart watch is circular. When receiving the UI description information of the user interface 810, the smart watch can parse the UI description information according to the method flowcharts shown in FIGS. 7 and 8 and display the user interface 820 as shown in FIG. 9B. The user interface 820 may include a letter index component 821 and a contact viewing component 822. Among them, the letter index component 821 can contain 26 letters of the alphabet, and can be used to find a contact whose name is the first letter based on one of the letters. For example, in response to a user operation acting on the letter B in the letter index component 821, the smart watch may display the contact “Dad” in the contact viewing component 822. Further, when a user operation acting on "Dad" in the contact viewing component 822 is detected, the smart watch can display the contact information (such as phone number, email) of the contact "Dad" in the contact viewing component 822.

Since the screen of the smart watch is circular, the smart watch can display the letters in the letter index component 821 on the user interface 820 in a circular arrangement. Not limited to the above-mentioned circular arrangement, the letters in the letter index component 821 may also be displayed on the user interface 820 according to other methods adapted to the smart watch screen.

When a user operation acting on the sports health icon 714A in the main interface 710 is detected, the first device (mobile phone) may display a user interface 830 as shown in FIG. 10A. The user interface 830 may include an outdoor running component 831, a mountaineering component 832, a bicycle component 833, a swimming component 834, and a walking component 835. Among them, the outdoor running component 831, mountaineering component 832, bicycle component 833, swimming component 834, and walking component 835 can be used to view the user's outdoor running, climbing, cycling, swimming, and walking exercise records, respectively. These components may be displayed in the user interface 830 in a vertical list arrangement. The user interface 830 may include more or fewer components indicating the user's exercise record, which is not limited in the embodiment of the present application.

The mobile phone can send the UI description information of the user interface 830 to the second device (smart watch). The screen of the smart watch is circular. When receiving the UI description information of the user interface 830, the smart watch can parse the UI description information according to the method flowcharts shown in FIGS. 7 and 8 and display the user interface 840 as shown in FIG. 10B. The user interface 840 may include an outdoor running component 841, a mountain climbing component 842 and other components that indicate the user's exercise record.

Since the screen of the smart watch is circular, the smart watch can display components such as the outdoor running component 841 and the mountain climbing component 842 in the user interface 840 in an arc-shaped list arrangement. It is not limited to the above-mentioned arrangement of the arc-shaped list, and each component indicating the user's exercise record may also be displayed on the user interface 840 according to other methods adapted to the smart watch screen.

The first device may send multiple sets of UI description information involved in one application scenario to the second device all at once.

In a multi-device interaction application scenario, multiple sets of UI description information may be involved. For example, in an application scenario of receiving an incoming call notification, the UI description information sent by the first device to the second device may at least include the first UI description information and the second UI description information corresponding to the user interface 21 and the user interface 22.

In some embodiments, the first device may send multiple sets of UI description information involved in an interactive application scenario to the second device all at once. When an input event is detected and the user interface is updated, the second device can search multiple sets of stored UI description information to determine the UI description information corresponding to the updated user interface.

The application scenario of receiving incoming call notification is still taken as an example for description.

As shown in FIGS. 6A and 6D, when receiving an incoming call notification, the first device (mobile phone) may display the user interface 21. The mobile phone can send all UI description information involved in the application scenario of receiving the incoming call notification to the second device (smart watch). All the UI description information may include the first UI description information corresponding to the user interface 21, the second UI description information corresponding to the user interface 22, and so on.

When all UI description information is received, the smart watch can search for the first UI description information. The smart watch can run a service program, parse the first UI description information, and display the components described by the first UI description information on the user interface 31 as shown in FIG. 6C.

When an input event is detected, the smart watch can determine the component associated with the input event. For example, when it is determined that the component associated with the input event is the hang-up component 312, the smart watch can search for the second UI description information from all the UI description information. The second UI description information may describe various components in the user interface 22 for instructing to hang up the phone as shown in FIG. 6D. In addition, the smart watch can also feed back the instruction to hang up the phone to the mobile phone. When receiving the instruction to hang up the phone, the mobile phone can use the module for mobile communication to send an instruction to the base station to hang up the phone.

It can be seen from the foregoing embodiment that the second device can store all UI description information involved in a certain application scenario, and when an input event is detected, it automatically searches for the UI description information according to the component associated with the input event. In this way, the requirement for the communication connection between the second device and the first device is not high, which reduces the situation that the second device cannot respond to the input event in time due to the disconnection of the communication connection between the first device and the second device.

The first device may send multiple sets of UI description information involved in one application scenario to the second device in stages.

In some embodiments, the first device may send multiple sets of UI description information involved in an application scenario to the second device in stages. That is, when the second device detects an input event and updates the user interface according to the input event, the first device may send the set of UI description information corresponding to the updated user interface to the second device.

As shown in the flowchart of the method for multi-device interaction shown in FIG. 5, in step S102 and step S103, when the first device (mobile phone) receives an incoming call notification, the mobile phone can display the user interface 21 as shown in FIG. 6A, And send the first UI description information to the second device (smart watch). The first UI description information can be used to describe the attributes of each component in the user interface 21. When the first UI description information is received, the smart watch may display the user interface 31 as shown in FIG. 6C.

When a user operation on the decisive component 312 in the user interface 31 is detected, the smart watch can send an instruction to hang up the phone to the mobile phone. Then in step S111 and step S112, the mobile phone can display the user interface 22 as shown in FIG. 6D, use the mobile communication module in the mobile phone to send an instruction to hang up the phone to the base station, and send the second UI description information to the smart watch . The second UI description information can be used to describe the attributes of each component in the user interface 22. When receiving the second UI description information, the smart watch may display the user interface 32 as shown in FIG. 6E.

It can be seen from the foregoing embodiment that in an application scenario where the first device interacts with the second device, the second device may not store all UI description information, but only needs to parse the UI description information from the first device. And the second device can only feed back the detected input event to the first device without actually processing services. In this way, the storage space and processing capability of the second device are not high requirements, and the burden on the second device can be reduced.

The first device may select a second device for interaction in response to a user operation.

In some embodiments, when the first device runs the first application and detects an instruction to interact with the second device selected by the user, the first device may send the first application to the second device selected by the user UI description information of the user interface in the.

11A to 11C show schematic diagrams of some application scenarios of multi-device interaction. Among them, the first device is a mobile phone, and the second device is a smart watch or a tablet.

When the application music player is running, the mobile phone can display the music playing interface as shown in FIG. 11A. The music playing interface may include a first component 410, which may include a linkage control component 411, a song information component 412, an album cover component 413, a lyrics component 414, a progress bar component 415, a play mode component 416, and a previous song. Component 417, pause component 418, next song component 419, and volume component 4110.

Among them, the linkage control component 411 can be used for interaction between the mobile phone and the second device. In response to a user operation on the linkage control component 411, such as a touch operation, the mobile phone can search for devices that have established a communication connection with the mobile phone, and display a user interface as shown in FIG. 11B. The user interface includes a first component 420. The first component 420 may include components indicating device options, such as a smart watch component 421 and a tablet component 422. The device corresponding to the device option included in the first component 420 is a device that has established a communication connection with a mobile phone.

In response to a user operation on the smart watch component 421, such as a touch operation, the mobile phone may send UI description information to the smart watch. The UI description information can include the linkage control component 411, the song information component 412, the album cover component 413, the lyrics component 414, the progress bar component 415, the play mode component 416, the previous song component 417, the pause component 418, and the next A description of the attributes of the header component 419 and the volume component 4110.

When receiving the UI description information, the smart watch can parse the UI description information according to steps S105 to S108 shown in Figure 5, instantiate the components in the UI description information using the component library, and display on the user interface as shown in Figure 11C The components. The user interface shown in FIG. 11C may include a song information component 431, a previous song component 432, a pause component 433, a next song component 434, a play mode component 435, and a volume component 436.

The embodiment of the present application does not limit the foregoing music playing interface, and the music playing interface may also include more or fewer components.

The foregoing embodiment of the second device that the first device selects to interact in response to the user's operation may be combined with the foregoing method embodiment shown in FIG. 5. For example, in the method embodiment shown in FIG. 5, when the first device performs step S102 and displays the first user interface, the first device may perform step S103 after receiving a user operation for determining the second device The second device sends the first UI description information. The first UI description information is used to describe the attributes of each component in the first user interface.

The first device can intelligently select the second device to interact with.

In some embodiments, when the first device runs the first application and detects that multiple devices have established a communication connection with the first device, the first device can be based on the distance between the first device or the multiple devices. The second device interacts with it in such a way as the priority order. In other words, the first device can select the closest device from a plurality of devices with which it has established a communication connection as the second device, and then send the user interface information in the first application to the selected second device. UI description information. Or, there is a priority order among multiple devices that have established a communication connection with the first device. The first device may select the device with the highest priority as the second device, and then send the UI description information of the user interface in the first application to the selected second device.

Exemplarily, in an application scenario where a user receives an incoming call notification while driving, the first device may be a mobile phone. Among them, both the smart car and the smart watch worn by the user have established a communication connection with the mobile phone, and in this application scenario, the priority of the smart car is higher than the priority of the smart watch. When the mobile phone receives the incoming call notification, the mobile phone can send the UI description information of the incoming call notification interface as shown in FIG. 6A to the smart car with higher priority. The smart car can parse the UI description information of the incoming call notification interface, and display the components in the UI description information on the onboard computer. In addition, the on-board computer can answer calls in response to user operations (such as touch operations, voice control operations). For the implementation of the onboard computer answering the phone in response to the user's operation, reference may be made to steps S109 to S114 in the method embodiment shown in FIG. 5.

The foregoing embodiment in which the first device intelligently selects the second device for interaction may be combined with the foregoing method embodiment shown in FIG. 5. For example, in the method embodiment shown in FIG. 5, when the first device executes step S102 and displays the first user interface, the first device can be based on the distance to the first device or the priority between the multiple devices. The second device to interact with is determined by means of level sequence, etc., and then step S103 is executed to send the first UI description information to the second device. The first UI description information is used to describe the attributes of each component in the first user interface.

The embodiment of the present application does not limit the manner in which the first device determines the second device to interact with. In addition to the manner of receiving user operations for determining the second device or the manner of smart selection, the first device may also use other methods. Determine the second device to interact with.

The following introduces a schematic diagram of the development of a multi-device interactive system.

Figure 12 shows a schematic diagram of the development of a multi-device interaction system. As shown in FIG. 12, the structure for developing a multi-device interaction system may include a workstation 510, a cloud 520, a first device 530, and a second device 540. in:

The workstation 510 is a development environment for developers (such as designers and programmers). The workstation may include a simulator 511 and a designer 512.

The designer 512 can be used to generate UI description information. The designer 512 may include a text editor and a visual drag-and-drop component. The above text editor can be used to generate UI description information by writing text. The UI description information conforms to the UI description format. The above-mentioned visual drag-and-drop component can be dragged, change attributes such as background color and border line, and generate UI description information in a visual user interface manner.

That is, developers can use the designer to generate UI description information by writing text or dragging and dropping visual components.

In addition to the way of writing text and the way of dragging and dropping visualized components, the designer 512 may also generate UI description information in other ways. For example, the designer 512 may generate UI description information by receiving a user interface design draft input by a developer. The embodiment of the application does not limit this.

When the UI description information is generated, the designer 512 may send the UI description information to the simulator 511.

The simulator 511 can be used to simulate the output module and the input module of the target device (such as the first device or the second device). For example, the simulator 511 can simulate the screen of a smart watch and an input module (such as a touch sensor, a keyboard, etc.) for responding to an input event (such as a user's touch operation and a text input operation).

The simulator 511 can provide the same binary interface as the target device. That is, developers can regard the simulator as the target device to verify whether the results displayed on the screen of the target device of each component in the UI description can achieve the expected effect, and whether the component associated with the input event is detected when an input event is detected. Able to respond.

The simulator 511 may include a service program 511a and a virtual machine/script engine 511b.

Among them, the service program 511a can be used to load and parse the UI description information. When receiving the UI description information, the simulator 511 can run the service program 511a to obtain the attributes of each component in the UI description information. According to the type in the attribute of the component, the simulator 511 can obtain the executable low-level code corresponding to the component in the component library, and initialize the data part of the executable low-level code by using the attribute of the component. When running the initialized executable low-level code, the simulator 511 can instantiate each component, and display each component in the UI description information on the user interface according to the logical relationship between the components.

The aforementioned component library may be preset in the simulator, or may be sent to the simulator along with the UI description information. The embodiment of the present application does not limit the manner in which the simulator obtains the component library.

When an input event acting on the simulator 511 is detected, the simulator 511 can run a service program to determine the component associated with the input event, and search for UI description information corresponding to the updated user interface. The simulator 511 can parse the aforementioned UI description information corresponding to the updated user interface, and display each component in the UI description information on the user interface. In this way, the developer can check on the simulator whether the second device's response to the input event reaches the expected effect when the second device detects the input event.

In some embodiments, a script or bytecode can be inserted into the UI description information. When the simulator 511 runs the service program to parse the UI description information, it can extract these scripts or bytecodes and send them to the virtual machine/script engine 511b.

The virtual machine/script engine 511a may indicate that the simulator 51 may include only a virtual machine, only a script engine, a virtual machine and a script engine, or a script engine based on a virtual machine. Among them, the virtual machine can be used to execute bytecode. The script engine can be used to execute scripts. If the script or bytecode contains a description of each component of the user interface, the virtual machine/script engine 511a can call the component library, obtain and execute executable low-level code from the component library to instantiate each component. If the script or bytecode contains a description of processing input events, when an input event is detected, the simulator 511 can execute the script or bytecode describing the input event in the virtual machine/script engine 511b.

It should be noted that scripts or bytecodes are executable programs. When the virtual machine/script engine 511a executes a script or bytecode, it can be executed in an isolated space in the simulator memory, and can control various permissions required by the script or bytecode. It can also capture and process Abnormal conditions during operation. In this way, the risks that may exist when the script or bytecode is executed can be reduced.

The workstation 510 may also contain more content, which is not limited in the embodiment of the present application.

When the UI description information is generated, the designer 512 may upload the UI description information to the cloud 520.

The cloud 520 may be used to store UI description information and application programs. The cloud 520 may be a content server, an application market, and so on.

The first device 530 may include UI description information. The UI description information may be preset or obtained by downloading an application that contains the UI description information. The first device can download the application program from the designer 512 in the workstation 510 or download the application program from the cloud 520. The application program may contain UI description information.

In some embodiments, the first device may include a service program and a virtual machine/script engine. The components displayed on the user interface of the first device are the components described in the UI description information. That is, the first device can run the service program to parse the UI description information, and call the component library to instantiate each component in the UI description information. Wherein, when the first device parses the UI description information, it may first obtain its own screen information (such as shape, size, resolution, etc.) of the first device, and calculate the size and position of each component according to the screen information. In this way, the components in the UI description information can be adapted to the screen of the first device.

In other embodiments, the application program contains a drawing command of a user interface adapted to the screen of the first device. The first device can execute these drawing commands to display various components on the user interface.

The embodiment of the present application does not limit the manner in which the first device displays each component on the user interface.

The first device 530 may send all UI description information related to an application scenario to the second device 540 at one time or in batches.

The second device 540 may include a service program 511a and a virtual machine/script engine 511b. The description of the service program 511a and the virtual machine/script engine 511b can refer to the description of the service program 511a and the virtual machine/script engine 511b in the simulator 511.

The second device 540 may obtain the UI description information from the first device 530 or from the cloud 520. Wherein, when the UI description information is obtained from the first device 530, the second device 540 needs to establish a communication connection with the first device 530. For example, the second device 540 may establish a communication connection with the first device 530 in a manner such as classic Bluetooth, Bluetooth low energy, or near field communication. When obtaining UI description information from the cloud 520, the second device 540 needs to establish a communication connection with the cloud 520. For example, the second device 540 may establish a communication connection with the cloud 520 by turning on WiFi and accessing the Internet.

It should be noted that the second device 540 includes a component library. The component library can be preset in the second device, or can be downloaded to the second device 540 along with the UI description information.

The second device 540 may process the UI description information according to steps S105 to S108 in the method flowchart shown in FIG. 5, and display the components in the UI description information on the user interface. I won't repeat it here.

In some embodiments, when a communication connection with the cloud 520 is not established or interaction with the first device 530 is required, the second device 540 may obtain UI description information from the first device 530. For example, in the music playing application scenario shown in FIGS. 11A to 11C, when an instruction to interact with the second device (smart watch) 540 is detected, the first device (mobile phone) 530 needs to communicate with the second device 540. Interactive. The smart watch can receive the UI description information from the mobile phone. The smart watch parses the UI description information, and according to preset filtering rules, can display the first component 430, the song information component 431, the previous component 432, the pause component 433, and the next as shown in FIG. 11C on the user interface. A song component 434, a play mode component 435, and a volume component 436. In this way, the user can control the music played on the mobile phone through the smart watch.

In the foregoing embodiment in which the second device obtains UI description information from the first device, since the channel power consumption for communication between the first device and the second device is low, it is beneficial for the second device to obtain UI description information from the first device. Reduce the power consumption of the second device.

In other embodiments, when working independently from the first device 530 and a communication connection is established with the cloud 520, the second device 540 may obtain UI description information from the cloud 520. For example, in an application scenario where the second device (smart watch) 540 is separated from the first device (mobile phone) 530 to play music, the smart watch can use its own speaker to play music. The smart watch may not install the application music player, but obtain the UI description information of the user interface in the music player from the cloud 520. Smart watches can also obtain other music-related data (such as music audio) from the cloud. Wherein, the UI description information acquired by the smart watch may include UI description information corresponding to the music playing interface and UI description information corresponding to the music list interface as shown in FIG. 11A. The music list interface can be used to select music to be played on the smart watch.

In the foregoing embodiment in which the second device obtains the UI description information of the user interface in the first application (such as a music player) from the cloud, the second device can not only provide the user with the first application without installing the first application; Service in the application, and the second device can work independently from the first device.

The following introduces another structure diagram for the development of a multi-device interactive system.

Figure 13 shows a schematic diagram of another development of a multi-device interactive system. As shown in FIG. 13, the structure for developing a multi-device interaction system may include a workstation 610, a cloud 620, a first device 630, and a second device 640. in:

For the description of the workstation 510 and the cloud 520, reference may be made to the description of the workstation 510 and the cloud 520 shown in FIG. 12, respectively. I won't repeat it here.

The first device 630 may include a virtual screen 631. The first device 630 may obtain in advance the screen information of the second device 640 or the window information of a specific window in the second device 640 (such as the shape, size, and resolution of the window), and set the screen information or window information of the second device It is the screen information of the virtual screen 631.

When the first device 630 interacts with the second device 640, the first device 630 can run the service program 611a to analyze the UI description information, and determine the size of each component in the UI description information according to the screen information of the virtual screen 631. The method for parsing the UI description information by the first device may refer to the method for parsing the UI description information by the second device in the method flowchart shown in FIG. 5.

The first device 630 may parse the UI description information, and use the component library to obtain instantiated components. The first device 630 may draw the instantiated component on the virtual screen 631 and generate a drawing command. The drawing command can be used to draw components that are adapted to the virtual screen.

The first device 630 may send the aforementioned drawing command to the second device 640.

The second device 640 may include a service program 611a and a virtual machine/script engine 611b. When receiving drawing commands from the first device 630, the second device 640 can run the service program 611a to execute these drawing commands. In this way, the second device 640 can display each component in the UI description information on the user interface.

In addition, when an input event acting on the second device 640 is detected, the second device 640 may feed back the input event to the first device 630. When an input event fed back by the second device 640 is received, the first device 630 can actually process the service corresponding to the input event, analyze the UI description information, and generate a drawing command. The UI description information parsed by the first device 630 is UI description information corresponding to the user interface that needs to be updated on the second device 640. When receiving drawing commands, the second device 640 can run the service program 611a to execute these drawing commands. In this way, the second device 640 can update the user interface.

According to the system shown in FIG. 13, it can be seen that the first device parses the UI description information and generates drawing commands according to the screen information of the second device, and the second device can directly execute the drawing commands to draw each component. This not only solves the problem of adaptation of the components displayed on the user interface of the second device, but also reduces the processing burden of the second device.

In some embodiments, the first device 630 may pre-acquire the screen information of the second device 640 or the window information of a specific window in the second device 640 (such as the shape, size, and resolution of the window), and set the The screen information or window information is set as the screen information of the virtual screen 631. When the first device 630 interacts with the second device 640, the first device 630 can parse the UI description information, and combine with the screen information of the virtual screen 631 to determine that each component in the UI description information is displayed on the virtual screen 631, that is, the first device 630 Second, the attributes used on the screen of the device 640. For example, the type, size, location, color, etc. of the component. Then, the first device 630 may send to the second device 640 the UI description information that determines the attribute used by the component to be displayed on the screen of the second device 640. In this way, the second device 640 can instantiate the aforementioned UI description information according to the attributes and component libraries used on the display of the components in the aforementioned UI description information on the second device 640, so as to display the aforementioned UI on the user interface of the second device 640. Describe the components in the information. These components can respond to user actions.

In the foregoing embodiment, the first device may parse the UI description information to determine the attributes used by each component in the UI description information to be displayed on the screen of the second device. Furthermore, the second device may only need to instantiate the components in the UI description information and display the instantiated components. In this way, it is also possible to solve the problem of adaptation of the components displayed on the user interface of the second device and reduce the processing burden of the second device.

The following introduces a schematic diagram of the structure of a device involved in the process of multi-device interaction.

Both the first device and the second device can be devices such as mobile phones, computers, smart watches, smart TVs, and automobiles. The embodiment of the present application does not limit the types of the first device and the second device.

FIG. 14 shows a schematic structural diagram of a device 500. The schematic structural diagram may be a schematic structural diagram of the first device or a schematic structural diagram of the second device.

As shown in FIG. 14, the device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, and a battery 142, Antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone interface 170D, sensor module 180, display screen 190, camera 191, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a distance sensor, a touch sensor, etc.

It can be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the device 100. In other embodiments of the present application, the device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Among them, the different processing units may be independent devices or integrated in one or more processors.

The controller may be the nerve center and command center of the device 100. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 190, the camera 191, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be provided in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.

The wireless communication function of the device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

The wireless communication module 160 can provide applications on the device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), and global navigation satellite systems. (Global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be sent from the processor 110, perform frequency modulation, amplify it, and convert it into electromagnetic waves to radiate through the antenna 2.

Exemplarily, in the embodiment of the present application, when multiple devices interact, a wireless communication connection, such as BT, NFC, etc., may be established through the wireless communication module 160. In this way, devices can transmit UI description information and related instructions, data, etc. through wireless communication channels.

The mobile communication module 150 may provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform processing such as filtering, amplifying and transmitting the received electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic wave radiation via the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.

Exemplarily, in the application scenario of receiving notification of incoming calls, the device 100 (such as a mobile phone) may use the mobile communication module 150 to communicate with the base station to realize the functions of answering and hanging up calls.

The display screen 190 is used to display images, videos, and the like. The display screen 190 includes a display panel. The display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). AMOLED, flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc. In some embodiments, the device 100 may include 1 or N display screens 190, and N is a positive integer greater than 1.

The camera 191 is used to capture still images or videos. The object generates an optical image through the lens and is projected to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the device 100 may include 1 or N cameras 191, and N is a positive integer greater than 1.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, save music, video and other files in an external memory card.

The internal memory 121 may be used to store computer executable program code, where the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the device 100 by running instructions stored in the internal memory 121. The internal memory 121 may include a storage program area and a storage data area. Among them, the storage program area can store an operating system, at least one application program (such as a sound playback function, an image playback function, etc.) required by at least one function. The data storage area can store data (such as audio data, phone book, etc.) created during the use of the device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

In the embodiment of the present application, the internal memory 121 of the device may be used to store service programs, component libraries, and the like. The above service program can be used to parse the UI description information. The aforementioned component library contains executable low-level code corresponding to each component. When the device 100 receives the UI description information, the processor 110 may run a service program stored in the internal memory 121 to parse the UI description information. When determining the type of each component in the UI description information, the processor 110 may load the component library stored in the internal memory 121, execute the component library to obtain executable low-level code corresponding to the component, and instantiate the component.

The device 100 can implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, and an application processor. For example, music playback, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110, or part of the functional modules of the audio module 170 may be provided in the processor 110.

The speaker 170A, also called "speaker", is used to convert audio electrical signals into sound signals. The device 100 can listen to music through the speaker 170A, or listen to a hands-free call.

The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the device 100 answers a call or voice message, it can receive the voice by bringing the receiver 170B close to the human ear.

The microphone 170C, also called "microphone", "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 170C through the human mouth, and input the sound signal into the microphone 170C. The device 100 may be provided with at least one microphone 170C. In other embodiments, the device 100 may be provided with two microphones 170C, which can implement noise reduction functions in addition to collecting sound signals. In other embodiments, the device 100 may also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions.

The earphone interface 170D is used to connect wired earphones. The earphone interface 170D may be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

Touch sensor, also called "touch panel". The touch sensor 180K may be disposed on the display screen 190, and the touch screen is composed of the touch sensor and the display screen 190, which is also called a “touch screen”. The touch sensor is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. The visual output related to the touch operation may be provided through the display screen 190. In other embodiments, the touch sensor may also be disposed on the surface of the device 100, which is different from the position of the display screen 190.

In the embodiment of the present application, the processor 110 may determine the screen coordinates of the touch operation on the display screen 190 according to the position of the touch sensor that detects the touch operation, and then convert the screen coordinates into the component coordinates of the user interface to determine the Touch the associated component.

Fig. 15 exemplarily shows a schematic structural diagram of another device 100.

As shown in FIG. 15, the device 100 may include a display screen 910, a communication device 920, a memory 930, and a processor 940 that are coupled to each other through a bus. in:

The communication device 920 can be used for the device 100 to exchange data with other devices. For example, the device 100 may receive UI description information from another device, and the device 100 may send an event for indicating a user operation on the device 100 to the other device. Among them, the device 100 establishes a communication connection with other devices. The memory 930 may be used to store one or more programs.

The processor 940 may be used to execute one or more programs stored in the memory 930, so that the device 100 can perform the following operations:

The device 100 may receive the first UI description information from other devices. The first UI description information can be used to describe the first UI component of the first application. The first UI component may be determined according to the first user interface of the first application. The first UI component can be displayed on the first user interface with the first component attribute. The first application is an application installed on other devices.

The device 100 may instantiate the aforementioned first UI component according to the second component attribute and the executable underlying code of the first component. The second component attribute can be determined according to the first UI description information. The executable underlying code of the first UI component may be determined from the component library according to the component type of the first UI component. The component library may contain executable low-level codes for drawing different types of components.

The device 100 may use the display screen 910 to display the second user interface. The second user interface may include a first UI component. The first UI component may be displayed with the second component attribute in the second user interface.

Among them, the first component attribute and the second component attribute may include component type, component size, component location, and component appearance.

In the embodiment of the present application, the first UI component may be a component in the first application that can be displayed on user interfaces of different devices. The first UI component may be displayed with the first component attribute on the first user interface of the first device. The first UI component may be displayed with the second component attribute on the second user interface of the second device. Wherein, the first UI component displayed on the above-mentioned second user interface may be a component instantiated according to the second component attribute and the executable underlying code of the first UI component. That is, the components mentioned in the foregoing embodiments.

Exemplarily, as shown in FIG. 6A and FIG. 6C, the first application program may be a dial-up application. The dial-up application can include a contact component, a hang-up component, and an answer component. These components may represent the aforementioned first UI component. These components can be displayed on the user interface of the mobile phone as shown in FIG. 6A and the user interface of the smart watch as shown in FIG. 6C, respectively, with the first component attribute and the second component attribute. These components are displayed on the user interface of the mobile phone, and can be a contact component 213, a hang-up component 213, and an answer component 214, respectively. These components are displayed on the user interface of the smart watch, and may be a contact component 311, a hang-up component 312, and an answer component 313, respectively. The position, size, color and other parameters of these components displayed on the user interface of the smart watch may be different from those displayed on the user interface of the mobile phone.

In addition, the first UI component can be divided into the first component or the second component mentioned in the foregoing embodiment according to whether it contains sub-components. In other words, when the first UI component contains subcomponents, the first UI component may be the aforementioned first component. When the first UI component does not contain subcomponents, the first UI component may be the aforementioned second component. For the definition of the second UI component, please refer to the above description of the first UI component.

In the embodiment of the present application, both the first application and the second application may be a system application, a home screen application, or a third-party application. The system application program may include the dialing application, the information application, and the contact application mentioned in the foregoing embodiments. The main screen application may be the main interface application mentioned in the foregoing embodiment. The third-party application program may include the WeChat application, the sports health application, and the Google Maps application mentioned in the foregoing embodiment.

In the embodiment of the present application, the first component attribute may include the component type of the first UI component and the component size, component position, and component appearance displayed on the user interface of the first UI component. The aforementioned component type may be the type of the first UI component. For example, buttons, sliders, check boxes, text boxes, icons, etc. The above-mentioned component size, the above-mentioned component position, and the above-mentioned component appearance may be the size, position, and appearance of the first UI component. The appearance is, for example, background, foreground, border line. For the meaning of the second component attribute, the third component attribute, and the fourth component attribute, please refer to the description of the first component attribute.

In the embodiment of the present application, the component object of the component type of the first UI component may be a type of component corresponding to the component type of the first UI component. For example, buttons, sliders, check boxes, text boxes, and icons. In other words, if the component type of the first UI component is a button, the second device uses the executable underlying code of the first UI component to obtain the component object of the component type of the first UI component, which can indicate that the second device obtains A button. However, the size, position, and appearance of the button displayed on the user interface of the second device need to be determined by the second device according to the attributes of the button displayed on the user interface of the second device.

In the embodiment of the present application, the first application program is installed in the first device. When an event is detected, the first device may send the first UI description information of the first user interface to the second device. Exemplarily, the first application program may be a dial-up application. The above event refers to incoming call notification. Then the above-mentioned first user interface may be an incoming call notification interface. That is, the first device sends the UI description information of the call notification interface to the second device, so that the second device can notify the user to answer the call.

In the embodiment of the present application, the second device displays the second user interface. The second user interface may be a user interface displayed by the second device according to the UI description information from the first device. For example, the first device sends the UI description information of the call notification interface to the second device. Then, the foregoing second user interface may be an incoming call notification interface displayed on the second device, which includes the components described in the UI description information.

As used in the above embodiments, depending on the context, the term "when" can be interpreted as meaning "if..." or "after" or "in response to determining..." or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if detected (statement or event)" can be interpreted as meaning "if determined..." or "in response to determining..." or "when detected (Condition or event stated)" or "in response to detection of (condition or event stated)".

In the foregoing embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center. Transmission to another website site, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrated with one or more available media. The usable medium may be a magnetic medium, (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state hard disk).

A person of ordinary skill in the art can understand that all or part of the process in the above-mentioned embodiment method can be realized. The process can be completed by a computer program instructing relevant hardware. The program can be stored in a computer readable storage medium. , May include the processes of the foregoing method embodiments. The aforementioned storage media include: ROM or random storage RAM, magnetic disks or optical discs and other media that can store program codes.

Claims

A multi-device interaction method, characterized in that the method includes:

The second device receives the first user interface UI description information from the first device; the first UI description information is used to describe the first UI component of the first application, and the first UI component is based on the first application The first user interface of the program is determined; the first UI component is displayed on the first user interface as a first component attribute; the first application is an application installed on the first device;

The second device instantiates the first UI component according to the second component attribute and the executable underlying code of the first UI component; the second component attribute is determined according to the first UI description information, The executable low-level code of the first UI component is determined from a component library according to the component type of the first UI component, and the component library contains the executable low-level code for drawing different types of components;

The second device displays a second user interface, the second user interface includes the first UI component, and the first UI component is displayed in the second user interface with the second component attribute;

Wherein, the first component attribute and the second component attribute both include a component type and one or more of the following parameters: component size, component position, component appearance.
The method according to claim 1, wherein the method further comprises:

The second device detects a first user operation acting on the first UI component in the second user interface;

The second device changes the component size and/or component appearance of the first UI component in the second user interface; the component appearance includes one or more of the following: color, foreground, and background.
The method according to claim 2, wherein the method further comprises:

After detecting the first user operation acting on the first UI component in the second user interface, the second device sends a first event to the first device; wherein, the first event is used for Indicating the occurrence of the first user operation;

The second device receives the second UI description information from the first device; the second UI description information is used to describe the second UI component of the first application, and the second UI component is based on the Determined by the third user interface of the first application; the second UI component is displayed on the third user interface as a third component attribute;

The second device instantiates the second UI component according to the fourth component attribute and the executable underlying code of the second UI component; the fourth component attribute is determined according to the second UI description information, The executable underlying code of the second UI component is determined from the component library according to the component type of the second UI component

The second device displays a fourth user interface, the fourth user interface includes the second UI component, and the second UI component is displayed in the fourth user interface with the fourth component attribute;

Wherein, the third component attribute and the fourth component attribute both include a component type and one or more of the following parameters: component size, component position, and component appearance.
The method according to any one of claims 1-3, wherein the method further comprises:

The second device searches for the executable underlying code of the first UI component from the component library according to the component type of the first UI component;

The second device obtains the component object of the component type of the first UI component by using the executable underlying code of the first UI component;

The second device uses the second component attribute to set one or more of the following parameters of the component object of the component type of the first UI component in the second user interface: component size, component position, component appearance To obtain the instantiated first UI component.
The method according to any one of claims 1 to 4, wherein the first UI component includes at least one sub-component; the method further comprises:

The second device searches for the executable underlying code of the first UI component from the component library according to the component type of the first UI component, and obtains the executable underlying code of the first UI component from the component library according to the component type of the sub-component Find the executable low-level code of the sub-component;

The second device uses the executable low-level code of the first UI component to obtain the component object of the component type of the first UI component, and uses the executable low-level code of the sub-component to obtain the sub-component Component object of component type;

The second device uses the second component attribute to set one or more of the following parameters of the component object of the component type of the first UI component in the second user interface: component size, component position, component appearance To obtain the instantiated first UI component, and use the component properties of the subcomponent to set the component object of the component type of the subcomponent in the second user interface with one or more of the following parameters : Component size, component position, component appearance, to obtain the instantiated subcomponent.
The method according to any one of claims 1-5, wherein the first UI description information includes the first component attribute; and the method includes:

The second device determines the second component attribute according to the first component attribute and the screen information of the second device; the screen information includes one or more of the following: screen size, screen shape, screen resolution .
The method according to any one of claims 1-5, wherein the first UI description information includes the second component attribute; and the method includes:

The second device sends the screen information of the second device to the first device; the screen information includes one or more of the following: screen size, screen shape, and screen resolution;

The second component attribute is determined by the first device according to the first component attribute and screen information of the second device.
The method according to any one of claims 1-7, wherein the first UI component is determined according to some or all of the components in the first user interface.
The method according to any one of claims 1-8, wherein before the second device displays the second user interface, the method further comprises:

The second device is in a screen off state;

or,

The second device displays a third user interface, the third user interface is a user interface of a second application, and the second application is an application installed on the second device.
The method according to any one of claims 1-9, wherein the first application includes: a system application, a home screen application, and a third-party application; the third-party application is a slave Download and install applications from the third-party application market.
The method according to any one of claims 1-10, characterized by comprising:

The second device obtains the component library from the first device;

or,

The second device obtains the component library from the cloud;

or,

The component library is preset in the second device.
A multi-device interaction method, characterized in that the method includes:

The first device detects the first event and sends the first user interface UI description information to the second device; the first UI description information is used to describe the first UI component of the first application, and the first UI component is based on Determined by the first user interface of the first application; the first UI component is displayed on the first user interface as a first component attribute; the first application is installed on the first device s application;

The second device instantiates the first UI component according to the second component attribute and the executable underlying code of the first UI component; the second component attribute is determined according to the first UI description information, The executable low-level code of the first UI component is determined from a component library according to the component type of the first UI component, and the component library contains the executable low-level code for drawing different types of components;

The second device displays a second user interface; the second user interface includes the first UI component, and the first UI component is displayed in the second user interface with the second component attribute;

Wherein, the first component attribute and the second component attribute both include a component type and one or more of the following parameters: component size, component position, component appearance.
The method according to any one of claims 12, wherein the method further comprises:

The first device determines the second device from multiple devices according to a first preset rule; the multiple devices have established a communication connection with the first device;

or,

The first device detects a first user operation, and the first user operation is used to select the second device; the first device determines that the device selected by the first user operation is the second device.
The method according to claim 12 or 13, wherein the method further comprises:

The second device detects a second user operation acting on the first UI component in the second user interface;

The second device changes the component size and/or component appearance of the first UI component in the second user interface; the component appearance includes one or more of the following: color, foreground, and background.
The method according to claim 14, wherein the method further comprises:

After detecting a second user operation acting on the first UI component in the second user interface, the second device sends a second event to the first device; wherein, the second event is used for Indicating the occurrence of the second user operation;

The first device sends second UI description information to the second device; the second UI description information is used to describe the second UI component of the first application, and the second UI component is based on the Determined by the third user interface of the first application; the second UI component is displayed on the third user interface as a third component attribute;

The second device instantiates the second UI component according to the fourth component attribute and the executable underlying code of the second UI component; the fourth component attribute is determined according to the second UI description information, The executable underlying code of the second UI component is determined from the component library according to the component type of the second UI component;

The second device displays a fourth user interface; the fourth user interface includes the second UI component, and the second UI component is displayed in the fourth user interface with the fourth component attribute;

Wherein, the third component attribute and the fourth component attribute both include a component type and one or more of the following parameters: component size, component position, and component appearance.
The method according to any one of claims 12-15, wherein the first UI description information includes the first component attribute; the method includes:

The second device determines the second component attribute according to the first component attribute and the screen information of the second device; the screen information includes one or more of the following: screen size, screen shape, screen resolution .
The method according to any one of claims 12-15, wherein the first UI description information includes the second component attribute; and the method includes:

The second device sends the screen information of the second device to the first device; the screen information includes one or more of the following: screen size, screen shape, and screen resolution;

The second component attribute is determined by the first device according to the first component attribute and screen information of the second device.
A device including a display screen, a communication device, a memory, and a processor, wherein the display screen is used to display a user interface; the memory is used to store one or more programs; the processor is used to execute all The one or more programs enable the device to implement the method according to any one of claims 1 to 11.
A computer program product containing instructions, characterized in that, when the computer program product runs on a device, the device is caused to execute the method according to any one of claims 1 to 11.
A computer-readable storage medium, comprising instructions, characterized in that, when the instructions run on a device, the device executes the method according to any one of claims 1 to 11.