WO2023010976A1

WO2023010976A1 - Terminal device, content service relay method, and content service relay interruption method

Info

Publication number: WO2023010976A1
Application number: PCT/CN2022/096018
Authority: WO
Inventors: 王光强; 杨绍栋; 肖兵; 刘文静; 公磊
Original assignee: 聚好看科技股份有限公司
Priority date: 2021-08-05
Filing date: 2022-05-30
Publication date: 2023-02-09

Abstract

The present disclosure relates to a terminal device, comprising a controller, configured to: send a service relay instruction to a relay terminal device, wherein the service relay instruction carries content service relay data, so that the relay terminal device executes a content service relay operation corresponding to the content service relay data; and receive a relay end feedback result sent by the relay terminal device, wherein the relay end feedback result is a result generated after the relay terminal device ends the content service relay operation, and the relay end feedback result comprises the execution progress data of the relay terminal device executing the content service relay operation. When the terminal device is currently in a first state, the terminal device can continue to execute the content service relay operation, and when the terminal device is currently in a second state, the terminal device cannot continue to execute the content service relay operation.

Description

Terminal device, content service relay method, and content service relay interruption method

This application requires that the application number submitted on August 5, 2021 is 202110896120.4, the application number submitted on September 24, 2021 is 202111121095.9, the application number submitted on September 24, 2021 is 202111120885.5, and the application number submitted on September 24, 2021 Priority of Chinese patent applications with application number 202111120453.4, application number 202111122128.1 filed on September 24, 2021, application number 202111120881.7 filed on September 24, 2021, and application number 202111126832.4 filed on September 26, 2021 rights, the entire contents of which are incorporated by reference in this disclosure.

technical field

The present application relates to the technical field of terminal equipment, and in particular to a terminal equipment, a content service relay method, and a content service relay interruption method.

Background technique

With the development of terminal products and the improvement of people's family living standards, more and more families have multiple household terminal devices. Home terminal devices can carry out content service relay among each other. For example, the video playback service on the TV can be relayed to the display screen of the refrigerator, and the video playback service on the refrigerator display screen can also be relayed to the TV.

The existing screen projection function is a cross-terminal video control function, which requires the projection terminal and the screen terminal to support specific protocols, such as DLNA, Miracast, etc. There are two main projection methods, one is represented by DLNA , which is mostly used for online video playback. After the user selects the screen projection function on the playback page, the screen projection terminal will find the screen projection terminal in the LAN that supports this protocol according to the predetermined protocol. By directly passing the video playback address to the projected terminal through the LAN, the projected terminal plays the video according to the video playback address. At this time, the projected terminal no longer performs video data analysis, and can sleep or shut down without affecting the projected terminal. screen playback. The other is represented by Miracast, which belongs to the operation of screen mirroring, that is, the screen projection terminal still displays the image after performing the screen projection, and mirrors the displayed image to the screen projection terminal through the LAN, and the dormancy or Closing will affect the playback of the projected terminal.

Contents of the invention

According to some embodiments of the present application, there is provided a terminal device, including a controller, and the controller is configured to: send a service relay instruction to the relay terminal device, and the service relay instruction carries content service relay data, so that the relay terminal device Execute the content service relay operation corresponding to the content service relay data; receive the relay end feedback result sent by the relay end device, wherein the relay end feedback result is that the relay end device ends the content service relay operation The generated result, the relay end feedback result includes the execution progress data of the content service relay operation performed by the relay terminal device; wherein, when the terminal device is currently in the first state, the terminal device is able to continue to execute the content service Relay operation, wherein, when the terminal device is in the first state, the terminal device can continue to execute content service relay operation according to the execution progress data; wherein, when the terminal device is currently in the second state , the terminal device cannot continue to execute the content service relay operation, wherein, when the terminal device is in the second state, the terminal device cannot continue to execute the content service relay operation according to the execution progress data.

According to some embodiments of the present application, there is provided a terminal device, which includes a controller, and the controller is configured to: receive a service relay instruction from the initiator device, wherein the service relay instruction carries content service relay data, and the The terminal device may execute the content service relay operation corresponding to the content service relay data; receive a relay interruption instruction, and according to the relay interruption instruction, not execute the content service relay operation corresponding to the content service relay data.

According to some embodiments of the present application, a content service relay method is provided, including: the terminal device sends a service relay command to the relay terminal device, and the service relay command carries content service relay data, so that the relay terminal device executes the content corresponding to the content service relay data Service relay operation: the terminal device receives the relay end feedback result sent by the relay end device, wherein the relay end feedback result is the result generated by the relay end device after the content service relay operation is completed, and the relay end feedback result includes the content service relay performed by the relay end device The execution progress data of the operation; wherein, when the terminal device is currently in the first state, the terminal device can continue to execute the content service relay operation, wherein, when the terminal device is in the first state, the terminal device can continue to execute according to the execution progress data Content service relay operation; wherein, when the terminal device is currently in the second state, the terminal device cannot continue to execute the content service relay operation, wherein, when the terminal device is in the second state, the terminal device cannot continue according to the execution progress data Perform content service relay operations.

According to some embodiments of the present application, there is provided a content service relay interruption method, including: a terminal device receives a service relay command from an initiator device, wherein the service relay command carries content service relay data, and the terminal device can execute content service relay data corresponding The content service relay operation; receiving the relay interruption instruction, and according to the relay interruption instruction, not executing the content service relay operation corresponding to the content service relay data.

Description of drawings

FIG. 1 is a usage scenario of a display device according to some embodiments;

FIG. 2 is a block diagram of a hardware configuration of a control device 100 according to some embodiments;

FIG. 3 is a block diagram of a hardware configuration of a display device 200 according to some embodiments;

FIG. 4 is a software configuration diagram in a display device 200 according to some embodiments;

FIG. 5 is a display diagram of an icon control interface of an application program in the display device 200 according to some embodiments;

Fig. 6 is a schematic diagram of a framework of a terminal device content service relay system according to some embodiments;

7 is a schematic diagram of a user interface of an initiator device according to some embodiments;

Fig. 8 is a schematic diagram of a user interface of another initiator device according to some embodiments;

FIG. 9 is a schematic diagram of another user interface of an initiator device according to some embodiments;

Fig. 10 is another application scenario diagram of content service relay according to some embodiments;

Fig. 11 is another application scenario diagram of content service relay according to some embodiments;

Fig. 12 is a schematic diagram of a user interface of another initiator device according to some embodiments;

Figure 13 is a schematic diagram of a user interface of a relay device according to some embodiments;

Fig. 14 is a schematic diagram of a user interface of another initiator device according to some embodiments;

Fig. 15 is a schematic diagram of a user interface of another relay device according to some embodiments;

Fig. 16 is a schematic diagram of a service relay business process according to some embodiments;

Fig. 17 is a schematic diagram of a content service relay application scenario of a terminal device according to some embodiments;

Fig. 18 is a scene diagram of matching dimensions of a content service relay device according to some embodiments;

Fig. 19 is a schematic diagram of a timing sequence of content service relay signaling processing according to some embodiments;

Fig. 20 is a schematic diagram of another content service relay signaling processing sequence according to some embodiments;

Fig. 21 is another scene diagram of content service relay device matching dimension according to some embodiments;

Fig. 22 is a scene diagram of another dimension matching content service relay device according to some embodiments;

Fig. 23 is a scene diagram of another content service relay device matching dimension according to some embodiments;

Fig. 24 is a schematic diagram of a user interface of another initiator device according to some embodiments;

Fig. 25 is a schematic diagram of another content service relay signaling processing sequence according to some embodiments;

Fig. 26 is a schematic diagram of a user interface of another initiator device according to some embodiments;

Fig. 27 is a schematic diagram of a user interface of another initiator device according to some embodiments;

Fig. 28 is a schematic diagram of a user interface of another initiator device according to some embodiments;

Fig. 29 is a schematic diagram of a business process for displaying a relay button control on the user interface of the initiator device according to some embodiments;

Fig. 30 is a schematic diagram of a business process for implementing service relay interruption according to some embodiments;

FIG. 31 is a signaling sequence diagram of a baton return according to some embodiments;

Fig. 32 is a flowchart of a content service relay return method according to some embodiments;

Fig. 33 is a flowchart of a content display method of a display device according to some embodiments;

Fig. 34 is a flowchart of a matching method of a terminal device according to some embodiments;

Fig. 35 is a flow chart of a method for relay interruption of content services according to some embodiments;

Fig. 36 is a signaling diagram of a method for content service relay interruption according to some embodiments;

Fig. 37 is a signaling diagram of a method for reporting service capability information of a terminal device according to some embodiments;

Fig. 38 is a signaling diagram of a content service relay method for a terminal device according to some embodiments;

Fig. 39 is a flowchart of a content service relay method for a terminal device according to some embodiments;

Fig. 40 is a schematic diagram of the overall internal structure of a local area network for content service relay according to some embodiments;

Fig. 41 is a diagram of interaction signaling between components during the capability information registration process of content service relay according to some embodiments;

Fig. 42 is a diagram of interaction signaling between components in the process of obtaining a relay terminal device for content service relay according to some embodiments;

Fig. 43 is an interactive signaling diagram of various components during video relay playback of content service relay according to some embodiments.

Detailed ways

In order to make the purpose and implementation of the application clearer, the following will describe the exemplary implementation of the application in conjunction with the accompanying drawings in the exemplary embodiment of the application. Obviously, the described exemplary embodiment is only a part of the implementation of the application , but not all examples.

The brief description of the terms in this application is only for the convenience of understanding the implementations described below, and is not intended to limit the implementations of this application. These terms are to be understood according to their ordinary and usual meaning unless otherwise stated.

The display device provided in the embodiments of the present application may have various implementation forms, for example, it may be a TV, a smart TV, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Fig. 1 and Fig. 2 are a specific implementation manner of the display device of the present application.

Fig. 1 is a schematic diagram of an operation scene between a display device and a control device according to an embodiment. As shown in FIG. 1 , the display device 200 also performs data communication with the server 400 , and the user can operate the display device 200 through the smart device 300 or the control device 100 .

In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes at least one of infrared protocol communication, Bluetooth protocol communication, and other short-distance communication methods, and the display device is controlled wirelessly or wiredly. Device 200. The user can control the display device 200 by inputting user instructions through at least one of buttons on the remote control, voice input, and control panel input.

In some embodiments, the smart device 300 (such as a mobile terminal, a tablet computer, a computer, a notebook computer, an AR/VR device, etc.) can also be used to control the display device 200 . For example, the display device 200 is controlled using an application program running on the smart device.

In some embodiments, the smart device 300 and the display device may also be used for data communication.

In some embodiments, the smart device 300 may also be other terminals capable of communication, such as air conditioners, refrigerators, smart central control, access control and other devices.

In some embodiments, the display device may not use the aforementioned smart device or control device to receive instructions, but may receive user control through touch or gesture.

In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the module for obtaining voice commands configured inside the display device 200 can directly receive the user's voice command control , the user's voice command control can also be received through the voice control device provided outside the display device 200 .

In some embodiments, the display device 200 also performs data communication with the server 400 . The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 . The server 400 may be one cluster, or multiple clusters, and may include one or more types of servers.

FIG. 2 is a block diagram of a configuration of the control device 100 according to the exemplary embodiment. As shown in FIG. 2 , the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply. The control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and play an intermediary role between the user and the display device 200 .

As shown in Figure 3, the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user input interface 280. at least one.

In some embodiments, the external device interface 240 may include, but is not limited to, the following: High Definition Multimedia Interface Interface (HDMI), Analog or Data High Definition Component Input Interface (Component), Composite Video Input Interface (CVBS), USB Input Interface (USB) , RGB port, etc. any one or more interfaces.

In some embodiments, the user can input user commands through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives user input commands through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

In some embodiments, the user input interface 280 is an interface that can be used to receive control input (such as: physical buttons on the display device body, or others).

Referring to Fig. 4, in some embodiments, the system is divided into four layers, from top to bottom are respectively the application (Applications) layer (abbreviated as "application layer"), application framework (Application Framework) layer (abbreviated as "framework layer") "), Android runtime (Android runtime) and system library layer (referred to as "system runtime layer"), and the kernel layer.

As shown in Figure 4, in some embodiments of the present application, the application framework layer includes managers (Managers), content providers (Content Provider) etc., wherein the manager includes at least one of the following modules: Activity Manager (Activity Manager) Use to interact with all activities running in the system; the Location Manager is used to provide system services or applications with access to the system location service; the Package Manager is used to retrieve the information currently installed on the device Various information related to the application package on the Internet; Notification Manager (Notification Manager) is used to control the display and clearing of notification messages; Window Manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers on the user interface and desktop widgets.

In some embodiments, after the display device is started, it can directly enter the interface of the preset video-on-demand program. The video-on-demand program is the default system program of the system, and its interface can be as shown in FIG. The content display area below the bar 510, the content displayed in the content display area will change with the change of the selected control in the navigation bar. The program in the application layer can be integrated in the video-on-demand program and displayed through a control in the navigation bar, or can be further displayed after the application control in the navigation bar is selected.

In some embodiments, after the display device is started, it can directly enter the display interface of the signal source selected last time, or the signal source selection interface, wherein the signal source can be a preset video-on-demand program, or an HDMI interface, and the live display device At least one of the interfaces, etc., after the user selects different signal sources, the display can display the content obtained from different signal sources.

With the development of terminal products and the improvement of people's family living standards, more and more families have multiple household terminal devices. Home devices can relay content services to each other. For example, the video playback service on the TV can be relayed to the display screen of the refrigerator, and the video playback service on the refrigerator display screen can also be relayed to the TV. The present application provides a terminal device content service relay system, as shown in FIG. 6 , a schematic diagram of a framework of the terminal device content service relay system. The system includes at least two terminal devices 100 . The terminal devices 100 are connected by communication via a local area network. Data transfer between the terminal devices 100 can be performed through the server 200, or data can be directly sent and received between the terminal devices. In the content service relay system of the present application, all terminal devices 100 can be used as initiator devices or relay device. In the present application, the solution is described by taking direct data transmission and reception between terminal devices as an example. The initiator device is a terminal device that initiates the content service relay process, and the relay end device is a target terminal device in the content service relay process. The server in the embodiment of the present application may be an AIOT (Artificial Intelligence & Internet of Things, Artificial Intelligence Internet of Things) server.

The content service relay service scenario in the embodiment of the present application includes two types: one is to trigger the relay service through a button control, and the other is to wake up the relay service by voice.

In some embodiments, while the foreground of a certain terminal device in the system (that is, the initiator device) is running the first application, it searches for a relayable device according to the application data of the first application. The relayable device is a terminal device capable of performing operations corresponding to the application data of the first application. If there is a relayable device, a relay prompt is generated. The relay prompt is used to prompt the user that there is another terminal device in the same local area network that can execute the operation corresponding to the application data of the first application.

Further, the user may input a service relay instruction according to the relay prompt. The initiator device receives the service relay instruction input by the user, and generates a relay request, and the relay request carries the application data of the first application. The initiator device sends a relay request to the found device that can be relayed. The relay-capable device executes an operation corresponding to the application data of the first application according to the relay request.

If there is no relayable device, no relay prompt will be generated, that is, no relay operation will be performed.

In some embodiments, as shown in the schematic diagram of the user interface of the initiating device in FIG. 7 , based on the service application currently running on the foreground of the initiating device, if there is a relay device, a relay button control is displayed on the user interface of the initiating device. . The following will explain how to determine whether there is a relay device. Here, it is judged whether there is a relay device, and it is not necessary to query all relay devices.

The business scenario in the example shown in FIG. 7 is a business scenario in which the relay service is triggered through a button control. After the user clicks the relay button in the user interface, the initiator device receives the user's operation and generates a service relay command. The generated service relay command carries the device information of the initiator device and the service information of the current service of the initiator device. According to the service relay instruction, the initiator device will query the service of the current initiator device, and then query the relayable device based on the service information of the current service. In the voice wake-up relay service business scenario, the user can directly input voice, and the initiator device receives the voice input by the user, and can also generate a service relay command. The initiating device queries the relayable device according to the generated service relay command, and the user interface shown in Figure 8 displays the prompt "Looking for an executable device". At this time, the running service of the initiator device can be stopped or continued. After the device that can be relayed is found, the information of the device that can be relayed will be displayed on the initiator device. In some embodiments, if multiple devices that can be relayed are found, a list of devices that can be relayed is displayed on the user interface, such as the user interface shown in FIG. 9 . In the user interface shown in FIG. 9, there are also a "delay 5 seconds to play" option bar and a "original device to continue playing" option bar.

When the user selects a relayable device in the relayable device list, for example, selects "Bedroom TV", and at the same time turns on the switch of the "Delay 5-second playback" option bar, as shown in Figure 12, the user on the initiator device The interface displays the prompt "soon to be pushed to the bedroom TV" and a countdown dialog box. The countdown dialog includes a back button by which the user can cancel the relay before the 5-second countdown.

In addition, on the user interface of the relay device as shown in FIG. 13 , that is, the user interface of the pushed device "bedroom TV" also displays a prompt saying "receiving push from XX device soon" and a countdown dialog box. Among them, XX device is the initiator device. The countdown dialog box displayed on the user interface of the relay device includes a return button and an OK button. If the user clicks the back button on the user interface of the relay device, the relay is cancelled. If the user clicks a confirmation button on the user interface of the relay terminal device, the relay is confirmed.

After the relay is determined, as shown in Figure 14, the user interface of the initiating device displays the prompt "relayed to the bedroom TV", and at the same time, the user interface of the initiating device jumps back to the interface before the relay business content. As shown in FIG. 15 , the user interface of the relay device displays the prompt "The content of XX device has been relayed", and at the same time, the user interface of the relay device starts relaying business content, such as playing a video.

In the service relay business scenario where the relay service is triggered by a button control, as shown in FIG. 16 , the business process of the above embodiment includes:

A1: The business application of the initiator device registers the relay capability of the business application with the unified relay service application, and the unified relay service application controls the transfer of scenarios and data during the relay process. After the initiating device system enters the relay business scene (entering the relay business scene can be the user interface enters a specific page, for example, the video playback application enters the video playback page from the main page. A relay button control is set in the video playback page.) The user clicks the user After the relay button in the interface, the initiator device receives the user operation and generates a service relay command. The unified relay service application can judge the current relay business scenario according to the operation input by the user.

A2: After the unified relay service application receives the service relay command, it carries the business information (the information of the business application currently running on the foreground of the initiator device) to the AIOT server to query the current account (it can be the account registered by the user on the relay service platform in advance, and then The user can then bind the terminal device under the same account), and can relay the relay terminal device of the current business.

A3: According to the current service and the status of all devices under the current account (only online devices are returned, not offline devices), the AIOT server returns a list of relay end devices under the current account that support current business relay.

The formation process of the relay terminal equipment list is as follows: the equipment capabilities of all terminal equipment are stored in the AIOT server in advance. When the AIOT server receives the service information of the current service of the initiator device, it traverses the stored device capabilities to find out the device capabilities that match the service information of the current service, and if it matches, it is determined to be a relayable device. List all the devices that can be relayed in the form of a list, which is the list of devices on the relay side. If there is only one relay device, there is only one terminal device listed in the relay device list. If there are multiple devices that can be relayed, there are also multiple terminal devices listed in the relay device list.

A4: After the unified relay service application receives the returned data (including device ID, device name and other data), it will display the list of relay end devices according to the returned data for users to choose.

A5: After the user selects a device, the device information of the selected device will be transmitted to the unified relay service application. (The device information mainly transmitted is the device ID, which is used to indicate the relay terminal device selected by the user)

A6: The unified relay service collects the business information of the current business and the equipment information of the selected equipment, assembles all the information, and transmits it to the AIOT server.

A7: After receiving the assembled information above, the AIOT server analyzes the information. Then, according to the device information, the service information (content information that needs to be relayed) is transmitted to the target device, that is, to the relay terminal device selected by the user.

A8: After receiving the content information of the relay, the relay device executes the corresponding relay action according to the content information. The initiator of the relay action is the unified relay service application of the relay device, and the implementer of the relay action is the application.

A9: After the relay end device performs the relay action, the unified relay service application of the relay end device will feed back the relay execution result to the initiator device through the AIOT server.

A10: The initiator device receives the relay execution result, and displays different relay states to the user according to the relay execution result. For example, if the relay end device successfully performs the relay action, it will display the relay success status. If the relay end device has not successfully performed the relay action, but is waiting for the relay countdown, it will display the state of starting the relay countdown.

In the business scenario of waking up the relay service by voice, as shown in FIG. 16 , the business process of the above embodiment includes:

B1: This step is a speech recognition process. After the voice application collects the voice signal input by the user, it sends the voice signal to the voice server, and the voice server performs semantic analysis on the voice signal, that is, recognizes the user's intention from the voice signal.

B2: After the voice server recognizes the user's intention as a relay service from the voice signal, it calls the AIOT server to query the relayable equipment (when calling the AIOT server, the voice server sends a call request to the AIOT server, and the call request carries business information for querying the relayable device ).

B3: After receiving the call request, the AIOT server queries the relay end device under the current account according to the business information that can relay the current business. And feed back the equipment information of the equipment that can be relayed to the voice server.

The voice server can only be used to identify the user's intention from the voice signal, and then feed back the user's intention to the unified service relay application. Afterwards, the unified service relay application interacts with the AIOT server. Some embodiments of the present application forward data through the voice server, which can shorten the link and improve the system response speed.

B4: The voice server returns the device information queried from the AIOT server to the voice application.

B5: The voice application calls the interface of the unified relay service application, and transmits the information recognized by the voice and the obtained device information to the unified relay service application. The voice recognition information may include a relay service instruction, and the device information includes a device ID.

B6: After receiving the device information of the relay terminal device, the unified relay service application combines the business information and transmits the above information (including business information, device information, relay service instructions, etc.) to the AIOT server. It may be that the relay service instruction carries service information.

B7: The AIOT server sends the business information and service relay instructions to the relay end device corresponding to the device information.

B8: After receiving the relay service instruction, the unified relay service application of the relay terminal device executes the corresponding relay action according to the business information carried in the relay service command, wherein the relay action is consistent with the content of the current service application of the initiator device.

B9: After the relay end device performs the relay action, the unified relay service application of the relay end device will feed back the relay execution result to the initiator device through the AIOT server.

B10: The initiator device receives the relay execution result, and displays different relay states to the user according to the relay execution result. For example, if the relay end device successfully performs the relay action, it will display the relay success status. If the relay end device has not successfully performed the relay action, but is waiting for the relay countdown, it will display the state of starting the relay countdown.

The content service relay process is usually that the user selects the target device on the initiator device, that is, the relay device. After the relay is determined to be triggered, the initiator device sends a service relay command to the relay device. The relay terminal device then executes the relay operation according to the service relay command.

Before the user selects the relay device on the initiator device, the initiator device and the relay device need to be matched. Device matching is based on the service capabilities of each terminal device. In some embodiments, the service capability corresponds to a service parameter of an application installed on the terminal device. For example, if a video playback application is installed on a display device, the service capability it has is a video playback capability. If the speaker is installed with an audio playback application, the service capability it has is the audio playback capability. If the tablet computer is installed with a video call application, the business capability it has is the video call capability.

In some embodiments, all terminal devices store service capability information of other terminal devices in the same local area network. It may be a capability information registration request automatically sent to other terminal devices after the terminal device is turned on, and the capability information registration request carries the service capability information of the terminal device. Other terminal devices store the service capability information of the terminal device according to the capability information registration request.

For example, the same local area network includes three devices, terminal devices A, B, and C. After the process of capability information registration, the service capability information list (b1, b2...bn) of terminal device B and terminal device A are stored in terminal device A. C's service capability information list (c1, c2...cn). The service capability information list (a1, a2...an) of the terminal device A and the service capability information list (c1, c2...cn) of the terminal device C are stored in the terminal device B. The service capability information list (a1, a2...an) of the terminal device A and the service capability information list (b1, b2...bn) of the terminal device B are stored in the terminal device C.

Each terminal device can report its own business capability information to the server, or directly send it to other terminal devices. In some embodiments of the present application, the report of the service capability information of the terminal device is described in detail by taking the terminal device reporting the service capability information of the terminal device to the server as an example.

The terminal device can report service capability information to the server, and the server can be an AIOT system or a capability management system. It may be that the terminal device automatically sends a service capability information registration request to the server when the system is started, and the capability information registration request carries the service capability information of the terminal device, so as to realize the reporting of the service capability information. It may also be that the terminal devices report the service capability information to each other, and the process of the terminal devices reporting the service capability information to each other will not be described in detail in this application.

For example, the three terminal devices A, B, and C are respectively installed with applications (A1, A2...An), (B1, B2...Bn) and applications (C1, C2...Cn), after the process of reporting service capability information , the service capability information list (a1, a2...an) of terminal device A, the service capability information list (b1, b2...bn) of terminal device B and the service capability information list (c1, c2...bn) of terminal device C are saved on the server. ...cn). Among them, the business capability information (a1, a2...an) in the business capability information list corresponds to the application (A1, A2...An), and the business capability information (b1, b2...bn) corresponds to the application (B1, B2...Bn ), the service capability information (c1, c2...cn) corresponds to the application (C1, C2...Cn) respectively.

In some embodiments, when the A1 application on the terminal device A is running in the foreground, the service capability information matching the service capability information a1 corresponding to the A1 application is searched according to the service parameters of the A1 application. If other service capability information matching the service capability information a1 is found, the corresponding relay terminal device is matched. If no other service capability information matching the service capability information a1 is found, the relay terminal device is not matched. For example, when the service capability information a1 matches b1, the terminal device B is determined to be the relay terminal device. When the service capability information a1 matches c2, the terminal device C is determined to be the relay terminal device.

In some embodiments, if the video playback application A1 is running on the terminal device, it can be determined that the terminal device A is performing a video playback function, and the video playback capability can be marked as a1 in the service capability information list of the terminal device A. After determining When the device can be relayed, it is found that b1 and a1 match, that is, b1 indicates that terminal device B can play video, and B is regarded as the device that can be relayed. For another example, if the video playback application A1 is running on the terminal device, it can be determined that the terminal device A is playing video. The video playback application A1 corresponds to the first authorization information, and the video playback capability corresponding to the first authorization information can be played on the terminal The business capability information list of device A is marked as a1. When determining the device that can be relayed, it is found that b1 and a1 match, that is, b1 indicates that terminal device B can play the video corresponding to the first authorization information, and then B will be used as For the relayable device, it is found that c1 indicates that terminal device C can play video, but the played video corresponds to the second authorization information, then C cannot be regarded as the relayable device.

The service parameters of applications installed on terminal devices will change as users use them. For example, as applications are installed or uninstalled, service capabilities supported by the terminal device will also increase or decrease. Therefore, in the content service relay process of some embodiments of the present application, it is necessary to monitor whether the service parameters of the applications installed on the terminal device change, and report according to the changed result. In order to avoid that the content service relay process is not carried out according to the actual service capability of the terminal equipment. If the content service relay process is not carried out according to the actual service capability of the terminal device, it may happen that the terminal device has a certain service capability, but the system judges that the terminal device is not a relay terminal device. Or, the terminal device does not have a certain service capability, but because the service capability information list has not been updated, the system misjudges that the terminal device is the relay terminal device.

In some embodiments, if it is determined that there is a change in the service parameter of the application installed on the terminal device, a service capability information update request is generated according to the change. The update request carries the identification information of the newly added capability or the newly reduced capability of the terminal in the table and the identifier of the terminal device, and sends the service capability information update request to the server, and the server updates the corresponding A list of service capability information of device identifiers. The updated service capability information list is used to record the service capability information of the terminal device, and the recorded service capability information corresponds to the service parameters of the applications installed on the terminal device.

If it is determined that there is no change in the service parameters of the applications installed on the terminal device, no service capability information update request is generated. Therefore, there is no need to update the service capability information list on the server. In this way, content service relay between terminal devices is carried out according to the actual service capabilities of the terminal devices, so as to avoid confusion in the process of content service relay.

In some embodiments, when there is a change in the service parameter of the application installed on the terminal device, there are several possible situations: application installation, application version change, and application uninstallation. Application installation, application version changes, and application downloads will all cause changes in application business parameters. Every time the application installed on the terminal device changes, the changed data can be sent to other terminal devices, so that the service capability information of the terminal device stored on other terminal devices is also updated. In the local area network, the business capabilities of each terminal device are recorded as a record on other terminal devices. When looking for relayable devices based on application data, traverse these records and match the application data with the service capability information in the records.

For example, if applications A1 and A2 are installed on terminal device A, service capability information a1 and a2 of terminal device A are stored on other terminal devices. When the application A1 is uninstalled from the terminal device A, the terminal device A no longer has the service capability represented by the service capability information a1. Synchronously delete the service capability information a1 stored on other terminal devices. When the application A3 is newly installed on the terminal device A, the terminal device A has the service capability represented by the service capability information a3. Synchronously add new service capability information a3 of terminal device A to other terminal devices.

After the terminal device reports the service capability information to the server, the server stores the service capability information list of the terminal device. For example, in the above example, after the three terminal devices A, B, and C report the service capability information, the server stores a list of service capability information corresponding to the information of the terminal device: the list of service capability information of terminal device A (a1, a2...an), the service capability information list (b1, b2...bn) of the terminal device B, and the service capability information list (c1, c2...cn) of the terminal device C. If at least one of application installation, application version change, and application uninstallation occurs to the application installed on the terminal device, the service parameters of the application will be changed accordingly. After the above service capability information update request, the records in the service capability information list will also be changed accordingly.

If the service parameter change is application installation, the service capability information update request is a service capability information increase request. The corresponding server adds service capability information to the service capability information list according to the service capability information addition request. The added service capability information corresponds to service parameters of newly installed applications.

If the service parameter change is an application version change, the service capability information update request is a service capability information modification request. The corresponding server modifies the service capability information in the service capability information list according to the service capability information modification request. The modified service capability information corresponds to the service parameter of the application whose version changes.

If the service parameter change is application uninstallation, the service capability information update request is a service capability information deletion request. The corresponding server elaborates the service capability information in the service capability information list according to the service capability information deletion request. The deleted service capability information corresponds to the service parameter of the uninstalled application.

In some embodiments, when a new application A3 is installed on the terminal device A, the service parameter of the terminal device A changes to application installation, and the terminal device A newly has the service capability represented by the service capability information a3. At the same time, terminal device A sends a service capability information addition request to the server, and the server adds service capability information a3 to the service capability information list of terminal device A after receiving the service capability information addition request.

For example, if there is no broadcasting application in the terminal device A, it does not have the service capability of playing broadcasting. The service capability information list of the terminal device A stored in the server also does not include the service capability information for broadcasting. If the application for broadcasting is newly installed in terminal device A, terminal device A has the service capability of playing broadcasting. At the same time, the terminal device A sends a service capability information adding request to the server, and the server adds service capability information for broadcasting to the service capability information list of the terminal device A according to the request. In this way, when the content service is relayed between terminal devices, it can be performed according to the actual service capability of terminal device A.

When the version of the application B1 on the terminal device B is upgraded from version 1.0 to version 2.0, the change of the service parameter of the terminal device B is a version change. Application B1 of version 1.0 only supports video services, and application B1 of version 2.0 supports both video services and audio services, and other applications installed on terminal device B do not support audio services. Therefore, the service capability information of the terminal device B is newly added with audio playback service capability information. Similarly, the server receives the request to modify the service capability information, and modifies the service capability information list of the terminal device B to include the audio playback service capability information.

When the application C1 is uninstalled on the terminal device C, the service parameter of the terminal device C changes to application uninstallation, and the terminal device C loses the service capability c1. Similarly, the server receives the request to delete the service capability information, and deletes the service capability information c1 from the service capability information list of the terminal device C.

In some embodiments, when the system of the current terminal device is started, the controller automatically inquires whether there is a difference between the current service parameters of the applications installed on the terminal device and the service parameters saved last time. If there is a difference between the current service parameter and the last saved service parameter, it is determined that there is a change in the service parameter applied on the terminal. When the terminal device starts the system, it not only collects the service parameter information of the boot-up self-starting application, but also collects the service parameter information of the non-boot-up self-starting application.

For example, an application A1 and an application A2 are installed on the terminal device A, the application A1 is an application that starts automatically when booting, and the application A2 is an application that does not start automatically when booting. When the terminal device A is turned on, the controller automatically collects the current service parameters of the application A1 and the application A2, and compares them with the service parameters saved last time. If there is a difference between the current service parameters of the application A1 and the last saved service parameters, the difference or the current service parameters are reported to the server, so that the service capability information related to the terminal device A and the application A1 is updated in the server. If there is no difference between the current service parameters of the application A2 and the last saved service parameters, no relevant information will be reported to the server.

In some embodiments, after the current terminal device is started, if the application installed on the terminal device changes, the updated relevant service parameters will be automatically sent to the server. For example, the application A1 installed on the terminal device A may be an application running in the foreground, an application running in the background, or an application that has not been started. No matter which state the application is in, if the service parameters of the application A1 change, the application A1 will automatically report the changed service parameters to the server. If the service parameters of the application A1 do not change, no relevant information is reported to the server. The implementation process of the above-mentioned business capability information reporting method is as follows:

After each application on the terminal device is installed on the terminal device, the service parameters of the current version will be written into the ANDROIDMANIFEST.XML file of the APK, and put into the META-DATA attribute. The reporting of business capability information is completed by the unified relay service application. When the terminal device is turned on for the first time, the unified relay service application collects the business capability information of the system-wide business applications. After the collection is completed, the collected business capability information of all applications is reported to the server, and a record of the business capability information of each application is saved locally. , the business capability information record is consistent with the business capability information list stored in the server.

The service capability information of the terminal equipment changes as the service parameters of the service application change. Changes in service parameters of a service application refer to changes such as application installation, version update, and uninstallation. When these changes occur in the application, an update broadcast will be sent to the unified relay service application. After receiving the update broadcast, the unified relay service application re-collects the service parameters of the application that sent the update broadcast, and generates a service capability information update request according to the re-collected service parameters. The unified relay service application reports the service capability information update request to the server. The server updates the service capability information list of the terminal device according to the service capability information update request. Since not all applications are self-starting applications, the unified relay service application is used to collect the business parameters of each application, and uniformly report the business capability information to the server according to the business parameters, so as to monitor the updates and changes of all applications on the terminal device.

In some embodiments, the unified relay service application also collects service parameters of applications that have been uninstalled but whose data has not been cleared. For example, although the application A1 on the terminal device A has been uninstalled, but the data has not been cleared, then when the terminal device B is the initiator device, the application A1 can be queried, indicating that the terminal device A can have the service capability represented by the service capability information a1, If the service capability information a1 matches the service parameters of the application B1 running in the foreground of the terminal device B, a prompt can be popped up on the terminal device A to prompt the user to install the application A1, so that the terminal device A can serve as the relay end device and complete the relay. In this way, the success rate of content service relay can be improved, and user experience can be further improved.

Some embodiments of the present application use a unified relay service application to uniformly collect parameters of each application. After the system is started, the unified relay service application queries the information of all applications in the system, and stores the information of all applications after querying. If the system is just started, it will restart the process of comprehensively querying the application information. If the application information queried this time is different from the application information saved last time, it means that the application has changed. If the system is kept on, when the application changes, the changed application will automatically send a corresponding broadcast to the unified relay service application.

In some embodiments, after the service capability information list of the terminal device is changed, the server determines whether there is any terminal device that can be relayed back to the terminal that initiates the relay within the preset time period before the change, and if so, then Re-feedback, otherwise no re-feedback. This situation is mainly due to the addition or deletion of the capabilities supported by the relayable device within the preset time period after the relay is triggered last time, and it may be that the user intends to use or not use the relayable device to perform the current relay operation. Therefore, it is convenient for the user to choose by proactively prompting the updated relayable device. After the preset time period after the last triggering of the relay, the user's operation to change the capability supported by the relay device is weakly related to the last relay operation, and the device that can be relayed after the change may not be reminded.

In some embodiments, before feeding back the updated device that can be relayed, the server needs to determine whether the updated application is the one that the user relayed last time, and if so, feedback is given; otherwise, no feedback is given. In this case, the application used in the last relay is a specific application. If the changed application is irrelevant, it is weakly related to the implementation of the last relay, and there is no need to remind the change. If the application being used has been upgraded or downgraded, it is strongly related to the implementation of the last relay, and a reminder of the change is required. In some embodiments, if the V1 version of the A application in the terminal device A does not support the first capability, the V2 version of the A application in the terminal device B supports the first capability, so in the last relay process, the terminal device B was As a relay-able device, it is fed back to the terminal that initiates the relay. When the A application in terminal device A is upgraded from version V1 to version V2, terminal device A can also be used as a relayable device. This enables the resend of the updated relayable device. When the application C is added to the terminal device A, it is weakly related to the realization of the last relay, and the reminder change may not be performed.

In some embodiments, before feeding back the updated relay-capable device, the server needs to determine whether the updated capability is related to the capability of the user's last relay use, and if so, feedback is given; otherwise, no feedback is given.

In some embodiments, before feeding back the updated device that can be relayed, the server needs to determine whether the capability changed after the update is the capability used by the user last time. If so, feedback is given; otherwise, no feedback is given. In this case, the application used in the last relay is a specific ability, and if the changed irrelevant ability is weakly related to the realization of the last relay, it is not necessary to remind the change. If there is an increase or decrease of relevant capabilities, it is strongly related to the realization of the last relay, and a reminder of the change is required. In some embodiments, if the A application in the terminal device A does not support the first capability, the B application in the terminal device B supports the first capability. Therefore, in the last relay process, the terminal device B was used as a relay capable device, and the feedback To the terminal that initiated the relay. The application B is installed on the terminal device A to add the first capability, and the terminal device A can also be used as a relayable device. This enables the resend of the updated relayable device. If terminal device A installs application C and adds a second capability, the terminal device still cannot be used as a relayable device. In this way, it is possible not to resend the updated relayable device.

In some embodiments, the terminal receiving the updated relay capable device may be the first terminal that initiates the first relay. It is convenient for the user to determine whether to switch the relay device through the first terminal.

In some embodiments, the terminal receiving the updated relay capable device may be the second terminal receiving the first relay. This is considering that the first terminal may have been out of use by the user, such as dormant, powered off, etc., and the second terminal is being used by the user. Therefore, it is selected to be displayed on the second terminal, so that the user can determine whether to switch the relay device through the second terminal.

Unlike DLNA, since DLNA discovers terminals after the user initiates them, device discovery and user operation time are strongly correlated, and there is no need to pre-set a list that supports screen projection.

In some embodiments of the present application, since the reminder of the device that can be relayed is an automatic reminder, it is reminded before the user performs screen projection, and the device that can be relayed is not necessarily in the same local area network, so through the business capability information list management terminal supported Capabilities will facilitate the realization of functions.

After the above process of reporting service capability information, the server stores service capability information of each terminal device. This service capability information can be used to display a list of devices that can respond to the relay for the initiator after receiving the relay request.

In some embodiments of the present application, the application with the relay service function runs in the foreground, and when the current page enters a specific page requiring relay service operations, it will automatically query the relay-capable devices. If the application running in the foreground does not have the relay service function, or, although the application running in the foreground has the relay service function, but the current page does not enter a specific page that may need to perform relay service operations, then the device that can be relayed will not be automatically queried.

In some embodiments, while the foreground of a terminal device in the system (that is, the initiator device) is running an application, it searches for the relay device according to the application data of the application, which is the device matching process. In the process of relay service operation, although relay can be performed between terminal devices, some embodiments of the present application use a server to perform relay service operation.

In one scenario, as shown in the application scenario diagram of FIG. 17 , the initiator device is a display device 100 - 1 , and the application running in the foreground is a video playback application. Then the current scene is that the video playing application of the display device is playing the video, and the application data of the application is the data related to playing the video. There are also terminal devices such as a display device 100-2, a speaker 100-3, and an alarm clock 100-4 in the system. Service capability information of terminal devices such as the display device 100-2, the speaker 100-3, and the alarm clock 100-4 are stored in advance in the display 100-1.

Among them, the service capability of the display device 100-2 is to play video and play audio, the service capability of the speaker 100-3 is to play audio, and the service capability of the alarm clock 100-4 is an alarm clock reminder. After the video playing application on the display device 100-1 runs in the foreground, it searches the server for a terminal device capable of playing video. At this time, the application data is playing video, and the stored service capability information is traversed according to the application data playing video. Finally, if it is found that the service capability of the display device 100-2 is to play video and matches the current application data, the display device 100-2 is determined as the relay terminal device.

In some embodiments, when the initiating device searches for the relayable device, it not only needs to determine whether the service capability information of other terminal devices matches the application data of the first application. It is also necessary to receive online information of other terminal devices. If the service capability information of a certain terminal device matches the application data of the first application, but the state of the terminal device is offline, the terminal device cannot be determined as a relay-capable device.

The relay service can have two business scenarios: the scenario where the button triggers the relay service and the scenario where the relay service is invoked by voice. In the scenario of voice-activated relay service, if the voice signal input by the user includes a specific relay device, the content service may be relayed to the relay device. If the voice signal input by the user does not include a specific relay device, the system can only return a list of devices that can be relayed, and the user needs to manually select a relay device in the list of devices that can be relayed to achieve the purpose of relaying content services, resulting in poor user experience .

Device capability refers to the functions supported by the device, such as audio and video playback, audio playback, temperature control, lighting switch, washing machine operation, and so on. In this application, the server side performs screening and feedback of relayable devices according to device capabilities. It is different from the voice control in the related art. In the current voice control, it can correspond to a single device, but when it corresponds to multiple devices, the device needs to be explicitly controlled by the user, for example: increase the volume of the display device, turn off the air conditioner, and so on. Take speakers as an example. The speaker will parse out the specified device name, and then control the device according to the device name. When corresponding to multiple devices, if you only say volume up, it will be difficult to determine the object of adjustment, or only adjust the speaker (default setting) itself.

In some embodiments of the present application, in a voice-activated relay service scenario, the terminal device may include a sound collector, and the sound collector is configured to collect a voice signal input by a user. After the voice signal input by the user is collected, the voice signal is sent to the voice recognition server, so that the voice text can be recognized from the voice signal. Conventional technology may be used to recognize the speech text from the speech signal, and the process of speech recognition will not be described in detail in this application.

In some embodiments, if the voice text includes device information, the relay device is determined according to the device information. The device information points to the only relay device. A unique relay device can thus be determined from the voice text. The device information may be any of device location information, device ID, device name, device type, device capability, and the like. After the unique relay device is determined, a relay relationship is established between the terminal device and the relay device, so as to complete the device matching process and further carry out the content service relay process.

In some embodiments, the above process may be implemented by a server, or by the terminal device itself. In some embodiments, the voice server mentioned above can be a part of the server, or it can be independent.

In some embodiments, the server may analyze the capability required by the voice command according to the voice text, and then determine the relay device according to the determined capability. For example. The voice command of "supply air" can analyze the ability of "supply air", and the corresponding "supply air" capability can be air conditioner, fresh air fan, fan, etc. "Start rinsing function" can analyze the "rinsing" ability, and the corresponding "rinsing" ability can include washing machines, dishwashers, etc. For "volume plus", the ability of "sound adjustment" can be analyzed, and the corresponding "sound adjustment" ability can include TV sets, speakers, refrigerator screens, etc. For "start sweeping the floor", the "sweeping" ability can be analyzed, and the corresponding "sweeping" ability can only be the sweeping robot.

If only the device type information is included in the voice text, the only relay device cannot be determined based on the device type information, only at least two relay end devices can be determined. Then, the final relay device is determined from the relay devices according to the relationship between the relay dimension information of the relay devices and the originating dimension information of the initiator device. For example, the voice text only includes "television", and multiple relay devices may be determined according to the voice text. Then, according to the relay dimension information of multiple televisions and the originating dimension information of the initiating device, it is determined that the final relayed TV is a TV having the same initiating dimension information of the initiating device. In this way, even if the voice signal input by the user does not include a clear relay device, the system can automatically determine the final relay device to achieve the purpose of content service relay, without requiring the user to manually select the relay device, thereby improving user experience.

In some scenarios, there may be only one relay device, so there is no need to compare the relay dimension information of the relay device with the initiation dimension information of the initiator device, and directly determine the relay device as the final relay device. In other scenarios, it is possible to determine multiple relay devices according to the relay dimension information of the relay device and the initiation dimension information of the initiator device. At this time, according to the frequency of use of these relay devices by the user, the relay device with the highest frequency of use may be determined as the final relay device. Alternatively, a dialog box is popped up for the user to select the desired final relay device from these relay devices. Alternatively, the desired final relay device is selected from these relay devices according to the user's location, where the location may refer to the current location, or the location to which the user will move determined according to the user's moving path. In some embodiments, the user's location information may be determined by sensors on one or more devices.

The relay device is a terminal device that finally relays the content service from the current terminal device, and the relay device is a device that matches the voice text. At the same time, the default relay terminal device here is a terminal device that can perform operations corresponding to the application data of the application running in the foreground of the current terminal device.

In some embodiments, the dimension information of the device includes a first dimension, a second dimension and a third dimension. The first dimension includes the second dimension, that is, the second dimension is within the range of the first dimension. The second dimension includes the third dimension, that is, the third dimension is within the scope of the second dimension, and at the same time, the third dimension is also within the scope of the first dimension. The relay device can be determined according to the relationship between the relay dimension information of the relay end device and the initiation dimension information of the initiator device, which can be divided into the following situations:

If there is a relay device whose first dimension, second dimension and third dimension respectively match the first dimension, second dimension and third dimension of the initiator device, that is, the relay terminal dimension information of the relay device is consistent with the originator device's If the originating dimension information completely matches, the relay end device is determined as the final relay device.

If there is no relay device whose first dimension, second dimension, and third dimension match the first dimension, second dimension, and third dimension of the initiating device, but there are The relay device whose first dimension and second dimension match, that is, the relay device and the initiator device are only within the same second dimension range, but different third dimension ranges. However, if there is no relay device within the same third-dimensional range as the initiator device, the relay device is determined as the final relay device.

If there is no relay device whose first dimension, second dimension, and third dimension respectively match the first dimension, second dimension, and third dimension of the initiator device, and there is no first dimension and second dimension The first dimension of the end device matches the second dimension of the relay end device, but there is a relay end device whose first dimension matches the first dimension of the initiator device, that is, there are only relay end devices in the same first dimension range, and the The relay device is determined as the final relay device.

In some embodiments, as shown in Figure 18, Figure 21, Figure 22, and Figure 23, the first dimension is the account number, and the second dimension is the room attribute identifier, such as the user's house in a different region. The three dimensions identify room areas, for example, different rooms in different houses of the user. The account is the account ID registered by the user in the unified relay service application. Users can create multiple homes under each account dimension. For example, a user owns multiple houses, which are located in A, B, and C respectively. The houses on land A, land B, and land C are the second dimension in the frame diagram. The third dimension is a plurality of physical spaces established based on the home dimension, such as master bedroom, living room, second bedroom, kitchen and other spaces.

In the scene shown in Figure 18, TV A and entrance screen A are located in the master bedroom of home A, entrance screen B is located in the living room of home A, and refrigerator with screen A is located in the kitchen of home A. TV B is located in the living room of home B. The entrance screen C is located in the living room of home C, and the refrigerator with screen B is located in the kitchen of home C. Home A, home B, and home C all belong to account A.

If the voice text includes device information, such as "relay to TV B in the living room of home B", and the voice text includes clear device information, no matter which terminal device is the current device, the device information points to the only relay device, and the final The relay devices are TV B in the living room of home B.

The signaling processing sequence of the above embodiment is, as shown in the signaling diagram of FIG. 19 , the initiator device receives the voice signal input by the user. The initiator device sends the voice signal and business information to the voice server, and the voice server recognizes the voice text from the voice signal, including device information, and the device information points to the only relay device, TV B in the living room of home B. The voice server sends the device information and service information to the AIOT server. The AIOT server searches for the device ID based on the device information, and sends service information to TV B in the living room of home B according to the device ID, so that TV B in the living room of home B performs corresponding relay operations according to the service information.

In the scenario shown in Figure 18, if the voice text only includes device type information, for example, the user inputs a voice signal to TV A, and the voice text recognized from the voice signal includes "relay to the entrance screen", at this time the voice text does not Include explicit device information, including only the device type information "entrance screen". The device type information "entrance screen" points to multiple entrance screens. In the scene shown in Figure 18, there is an entrance screen A in the master bedroom of home A, and the entrance screen A and TV A are located in the same account, the same house, and the same physical space, that is, the first dimension, the second dimension and the third dimension of the entrance screen A are respectively Matches the first dimension, second dimension and third dimension of TV A. Therefore, the porch screen A is determined as the final relay device, and finally the purpose of content service relay is realized, and the content service relay can be relay playback of videos.

The signaling processing sequence of the above embodiment is, as shown in the signaling diagram of FIG. 20 , the initiating relay terminal receives the voice signal input by the user. The initiator device sends the voice signal to the voice server, and the voice server recognizes the voice text from the voice signal including device type information, and the device type information points to multiple relay devices. The voice server sends the device information and business information of multiple devices to the AIOT server. The AIOT server matches the dimension information of multiple devices with the dimension information of the originating device, and after matching, the entrance screen is obtained as the final relay device. After the AIOT server finds the device ID of the entrance screen A, it sends service information to the entrance screen A according to the device ID, so that the entrance screen A performs the corresponding relay operation according to the service information.

In the scene shown in Figure 21, TV A is located in the master bedroom of home A, entrance screen B is located in the living room of home A, and refrigerator with screen A is located in the kitchen of home A. TV B is located in the living room of home B. The entrance screen C is located in the living room of home C, and the refrigerator with screen B is located in the kitchen of home C.

If the voice text only includes device type information, for example, the user inputs a voice signal to TV A, and the voice text recognized from the voice signal includes "relay to the entrance screen". At this time, the voice text does not include clear device information, only includes Type information "entrance screen". The device type information "entrance screen" points to multiple entrance screens. TV A is located in the master bedroom of home A, and in the scene shown in Figure 21, there is no entrance screen in the master bedroom of home A. Therefore, in the current scene, there is no relay device in the same account, the same house, and the same physical space as TV A. In the current scene, it is searched in the upward dimension whether there is a relay terminal device in the same account as TV A but not in the same physical space. In the current scene, it is found that there is an entrance screen B in the living room of home A. The entrance screen B and TV A are in the same account and the same house but not in the same physical space. Therefore, in the current scene, the entrance screen B is determined as the final relay device.

The signaling processing sequence of the foregoing embodiment is similar to the signaling processing sequence of the embodiment shown in FIG. 20 , and the signaling processing sequence according to some implementation manners of the present application will not be repeated here.

In the scene shown in FIG. 22 , TV A is located in the master bedroom of home A, and refrigerator A with a screen is located in the kitchen of home A. TV B is located in the living room of home B. The entrance screen C is located in the living room of home C, and the refrigerator with screen B is located in the kitchen of home C.

If the voice text only includes device type information, for example, the user inputs a voice signal to TV A, and the voice text recognized from the voice signal includes "relay to the entrance screen". At this time, the voice text does not include clear device information, only includes Type information "entrance screen". The device type information "entrance screen" points to multiple entrance screens. TV A is located in the master bedroom of home A, and in the scene shown in Figure 22, there is no entrance screen in the master bedroom of home A. Therefore, in the current scene, there is no relay device in the same account, the same house, and the same physical space as TV A. In the current scene, check whether there is a relay terminal device in the same account as TV A but not in the same physical space in the upward dimension. In the current scene, there is no entrance screen in the master bedroom, living room and kitchen of Home A. Therefore, in the current scene, there is no relay terminal device that is in the same account as TV A but not in the same physical space. In the current scene, check whether there is a relay terminal device that has the same account as TV A but is not in the same family. In the current scene, it is found that there is an entrance screen C in home C under the same account. The entrance screen C and TV A are under the same account, but not in the same family, nor in the same physical space. Therefore, in the current scene, the entrance screen C is determined as the final relay device.

In some embodiments, if there is no relay device whose first dimension matches the first dimension of the current terminal device, but there is also a relay device whose second dimension matches the second dimension of the current device, or there is a third dimension A relay device that matches the third dimension of the current device generates prompt information, so that the user can determine the relay device as the relay device after adding the relay device to the first dimension according to the prompt information.

For example, in the scene shown in FIG. 23 , TV A is located in the master bedroom of home A, and refrigerator A with a screen is located in the kitchen of home A. TV B is located in the living room of home B. Refrigerator B with screen is located in the kitchen of home C. The entrance screen D is located in the living room of home A, but the TV A is under the account A, and the entrance screen D is under the account B or the entrance screen D has not been added to any account. In the current scenario, the first dimension of the entrance screen D does not match the first dimension of the TV A, but the entrance screen D matches the voice text and belongs to the relay terminal device. At this time, a prompt message indicating to add the entrance screen D to the account A may pop up on the display of the current terminal device. The user can operate according to the prompt information, and finally add the entrance screen D to the account A, so that the first dimension of the entrance screen D matches the first dimension of TV A. At the same time, since the entrance screen D is located in the living room of home A, after the entrance screen D is added to the account A, the entrance screen D and TV A are in the same account and the same home. That is to say, finally, the first dimension and the second dimension of the entrance screen D match the first dimension and the second dimension of the TV A respectively.

After the above-mentioned device matching process, if there is a relay device, the display device 100-1 generates a relay prompt, which is used to prompt the user that there is currently a matching relay device. If there is no relay terminal device, no relay prompt may be generated. If the initiator device has a display, the relay button control may be displayed on the display, for example, a schematic diagram of a user interface as shown in FIG. 7 . The user can trigger the content service relay process by clicking the relay button control. The relay prompt can also be in the form of voice broadcast.

In some embodiments, if it is determined that there is a relay device, a relay button is displayed on the user interface. When the initiator device receives an operation of clicking the relay button input by the user, it generates a query instruction. The initiator device sends the query command to the AIOT server, and the AIOT server queries all relayable devices in the system according to the query command, and then feeds back all relayable devices to the initiator device. Then, the information of all relay-capable devices is displayed on the user interface of the relay device, that is, the list of relay devices is displayed.

In some embodiments, the user can set to add a relay button on a specific page of a specific application. For example, on the video playback page of the video playback application, if there is a relay device, a relay button will be displayed. On the teaching-related pages of the learning application, if there is a relay terminal device, the relay button will be displayed, and on the self-learning related pages of the learning application, it will not be set to display the relay button if there is a relay terminal device.

In some embodiments, if relay equipment is not present, a relay guidance button control may be displayed on the display. The user can trigger the display of content service relay guidance information on the display by clicking the relay guidance button control. Here, the content service relay guidance information may be introduction information for the content service relay, or information that guides the user to perform a next step related operation of the content service relay. In this way, if there is no relay terminal device, even if the relay button control is not displayed, the content related to the content service relay can be displayed on the user interface of the first application, thereby improving user experience.

In the application scenario shown in FIG. 8 , the user inputs a relay operation by selecting the target device display device 100 - 2 , that is, the relay terminal device. After receiving the relay operation input by the user, the display device 100-1 generates a service relay command. The service relay command carries the application data and media data of the video playback application, and the service relay command also carries the device ID of the relay terminal device. The initiator device sends the service relay instruction to the display device 100-2 according to the device ID through the AIOT server. After receiving the service relay instruction, the display device 100-2 continues to play the video according to the application data and media asset data of the video playing application.

In one scenario, as shown in the application scenario diagram of FIG. 10 , the initiating device is a tablet computer 200 - 1 , and the application currently running in the foreground of the initiating device (hereinafter referred to as the first application) is a video call application. Then the current scenario is to use the video call application of the tablet computer to make a video call, and the application data of the first application is the video call data. In the same local area network, there are also terminal devices such as the display device 200-2 and the mobile terminal 200-3. The business capability information of the display device 200-2 and the mobile terminal 200-3 is stored in advance in the tablet computer 200-1.

Among them, the service capabilities of the display device 200-2 are video playback and audio playback, and the service capabilities of the mobile terminal 200-3 are video playback, audio playback, and video calls. The video call application on the tablet computer 200-1 runs in the foreground, and searches for a terminal device capable of performing video call operations in the local area network. At this time, the application data is a video call, and the stored service capability information is traversed according to the application data video call. Finally, it is found that the service capability of the mobile terminal 200-3 is a video call, which matches the current application data. Then the mobile terminal 200-3 is determined as a relay-able device.

After the device that can be relayed is determined, the tablet computer 200-1 generates a relay prompt, and the user is prompted that there is currently a relayable device, and the user can input a service relay instruction according to the relay prompt. Since the tablet computer 200-1 has a display, the generated relay prompt may be a relay button displayed on the display. The user can call to select the relay button to input relay execution. Here, the relay prompt can also be a voice prompt. For example, after it is determined that there is a device that can be relayed, the user will be prompted to trigger the relay through voice input through the voice broadcast "Please enter the relay voice to start the relay". It is also possible to broadcast "Please shake the device to start the relay" through the voice broadcast to prompt the user to trigger the relay through gestures. These prompts can also be displayed on the display at the same time to facilitate the user to perform related operations.

The relay is triggered in the above manner, that is, after the user inputs the service relay command, the tablet computer 200-1 receives the service relay command, generates a relay request according to the service relay command, and sends the relay request to the mobile terminal 200-3. Here, the relay request may carry instructions for the video call and data related to the video call. After receiving the relay request, the mobile terminal 200-3 continues the video call process according to the video call instruction and video call data.

In one scenario, as shown in the application scenario diagram of FIG. 11 , the initiator device is an alarm clock 300 - 1 , and the first application is an alarm clock reminder application. Then the current scene is that the alarm clock notification application of the alarm clock is prompting, and the application data of the first application is related data of the alarm clock notification. In the same local area network, there are also terminal devices such as the display device 300-2 and the display device 300-3. The service capability information of the display device 300-2 and the display device 300-3 is stored in advance in the alarm clock 300-1.

Wherein, the service capability of the display device 300-2 is to play video and play audio, and the service capability of the display device 300-3 is to play video and play audio. After the alarm clock application on the alarm clock 300-1 runs in the foreground, it searches the local area network memory for a terminal device that can perform the alarm clock prompt operation. At this time, the application data is an alarm clock prompt, and the stored service capability information is traversed according to the application data alarm clock prompt. Since there is no other terminal device in the current same local area network whose business capability can match the current application data alarm notification, there is no other terminal device that can perform the alarm clock notification operation in the same current local area network, that is, there is no relayable device. In this scenario, no relay notification is generated on the alarm clock, and the final operation of the alarm clock notification is still performed on the alarm clock 300-1.

In some embodiments, after receiving the relay request, the relay-capable device executes an operation corresponding to the application data according to the application data carried in the relay request. After the relay, the relay capable device also feeds back the relay result to the initiating device. The relay result is used to indicate that the device that can be relayed has started to perform corresponding operations according to the application data.

In the scenario shown in FIG. 10 , after the video call on the tablet computer 200-1 is relayed to the mobile terminal 200-3, the video call process continues on the mobile terminal 200-3. At the same time, the mobile terminal 200-3 feeds back the relay result to the tablet computer 200-1. After the tablet computer 200-1 receives the relay result feedback, it can display prompts related to the relay result on the display. For example, a prompt saying "Video passing has been continued on the mobile terminal 200-3" is displayed.

In some embodiments, after receiving the relay result fed back by the relay device, the initiating device may suspend the application process of the first application, or directly close the first application.

In the scenario shown in FIG. 17 , after the video playback of the first application of the display device 100-1 is relayed to the display device 100-2, the video playback process continues on the display device 100-2. At the same time, the display device 100-2 feeds back the relay result to the display device 100-1. After receiving the feedback relay result, the display device 100-2 may suspend the video playing process of the video playing application. The display device 100-2 may also directly close the video playing application according to the relay result.

In some embodiments, if the first application initiator device is a video application, the corresponding relay button control is a video relay button control. If the first application is an audio application, the corresponding relay button control is an audio relay button control. If the first application is a picture display application, the corresponding relay button control is a picture display relay button control. The video relay button control, the audio relay button control and the picture display relay button control may be displayed in different forms. In this way, it is convenient for the user to view the application types that currently need to be relayed.

For example, as shown in FIG. 24 in the user interface of the audio application, if there is a relay device, an audio relay button control is displayed on the user interface of the audio application. Audio relay button controls can be displayed with musical notation icons. The same video relay button control can be displayed as a video icon, and the picture display relay button control can be displayed as a picture symbol icon.

The signaling processing sequence of the above embodiment is shown in the signaling diagram shown in Figure 25. When the initiator device is on the audio application user interface, it sends a relay device query command to the AIOT server through the unified relay service application. The query command carries application data of the audio application. The AIOT server queries whether there is a terminal device that can execute the same or similar application data according to the application data of the audio application, and feeds back the query result to the initiator device. If the query result shows that there is a terminal device capable of executing the same or similar application data, an audio relay button control is displayed on the audio application user interface of the initiator device. Conversely, the audio relay button controls are not displayed. The audio on the relay device may be applied to different applications on the initiator device, but it can be judged that they are applications with the same function according to the application data.

In some embodiments, the name of the second application on the relay device may also be displayed on the user interface of the first application. The second application is an application that executes operations corresponding to the application data of the first application. For example, in the user interface shown in FIG. 26 , if the first application on the currently displayed device is YYY music, then the device installed with the audio application is the relay device. If the audio application installed on the relay device is WW Music, the words "WW Music" will be displayed near the relay button control. If the audio application installed on the relay terminal device is MM Music, the word "MM Music" will be displayed near the relay button control. If both WW music and MM music are installed on the relay device, the words "WW music" and "MM music" can be displayed at the same time. If the user selects "WW Music", the corresponding relay operation will be performed by WW Music on the relay device. If the user selects "MM Music", the relay operation will be performed by MM Music on the relay device.

In some embodiments, if there are multiple relay devices that can perform the relay operation, a "plus sign" may be superimposed on the relay button control to prompt the user that there are currently multiple relay devices. When the user clicks on the relay button control, a relay end device list may be displayed on one side of the relay button control. And the device name is included in the relay terminal device list, and the name of the application that can perform the relay operation is displayed on one side of each device name.

For example, in the user interface shown in FIG. 27 , when the first application on the currently displayed device is YYY music, the devices installed with the audio application include multiple devices: device A, device B, and device C. On the side of the device name, the application names that can perform the relay operation are displayed respectively: KK music, MM music and WW music. Furthermore, if there are multiple applications that can perform relay operations on one relay device, all of them will be displayed after the device name.

In some embodiments, a thumbnail of the user interface of the second application may also be displayed on the user interface of the first application. In the user interface shown in Figure 28, the first application on the current display device is WWW music, and the application installed on the relay device is WW music, then a thumbnail of the WW music user interface is displayed on one side of the relay button control. In this way, the user can view the user interface of the second application after relaying, which is convenient for the user to decide whether to click the relaying button control, so as to relay the content service, and further improve the user experience.

If the audio application on the initiator device and the audio application on the relay device are different application software, the relay operation can be transmitted according to the name, singer, time and other information of the audio media data. For example, the audio application on the initiator device is YYY Music, and the played song is "XXXXX". When relaying to the relay device, you can search for the corresponding song on the audio application WW Music of the relay device based on the song name, singer name, time "XXXX" year and other information. And continue to play the song according to the current playing time of the song on the initiator device.

In terms of displaying the thumbnail of the user interface of the second application, the AIOT server may automatically send a screenshot request to the relay device after querying the application on the relay device. After receiving the screenshot request, the relay device sends the previously saved screenshot of the second application interface to the AIOT server. The AIOT server feeds back the screenshot of the second application interface to the initiator device. After receiving the screenshot of the second application interface, the initiator device displays a thumbnail of the screenshot of the second application interface on the first application interface.

In some embodiments, the display device and the determined relay device need to be under the same account. Adding terminal devices to the same account is a related technology, and this application will not elaborate on it. During the content service relay process, only the devices under the same account can carry out the content service relay process with each other. This can avoid interference from devices that are not under the same account.

In the above embodiment, the implementation process of displaying the relay button control on the user interface of the first application is the process shown in Figure 29, including: C1, the service application running on the foreground of the terminal device enters the relay service scene (entering the relay service scene can be It is the user interface that enters a specific page, for example, after the video playback application enters the video playback page from the main page), the business application carries its relay scene identifier and requests to bind the unified relay service application, so that the unified relay service application can know that it is currently in the relay business scenario business applications. C2. After receiving the binding request of the business application, the unified relay service application sends a query request (carrying business application data) to the AIOT server (which may be a device capability storage system). The AIOT server searches for terminal devices supporting the same service under the same account according to the service application data carried in the query request, that is, queries the relay terminal device. C3. The server feeds back the found device information of the relay end device and the application information that can execute the relay to the unified relay service application. C4. The unified relay service application returns the queried device information of the relay end device and application information that can perform relay to the business application. C5. The business application displays different UIs on the user interface of the business application according to the device information of the relay terminal device and the application information that can execute the relay.

For example, if the service application in the relay service scenario on the initiator device is a video application, the searched application information of the service application of the relay device is also a video application, and the video relay control is displayed. If the service application on the initiator device in the relay service scenario is an audio application, then the searched application information of the service application of the relay device is also an audio application, and the audio relay control is displayed.

If the user clicks the relay button control in the above embodiment, the user interface shown in FIG. 7 can also be jumped to the user interface shown in FIG. 9 . In some embodiments, if there are at least two relayable devices in the local area network, and the initiator device has a display, a list of relayable devices is displayed on the display. The user interface shown in Figure 9 lists the names of at least two relay devices (if there is only one device that can be relayed, the list of relay devices only displays the information of one device that can be relayed), and the user can select one of them One relay device, that is, determine the relay device from multiple relay devices. The initiating relay terminal generates a service relay instruction according to the user's operation. At the same time, the service relay command is sent to the determined relay end device. The service relay instruction carries content service relay data (corresponding to the application data of the first application running in the foreground of the initiator device). After receiving the content service relay data, the relay terminal may execute the content service relay operation corresponding to the content service relay data.

If the current service relay instruction is an instruction sent by the user’s misoperation, for example, the user mistakenly clicked the relay button or clicked a certain relay terminal device by mistake in Figure 9, the initiator device will send the service relay instruction to the selected relay terminal device , the current relay device is the relay device. The name of the relayable device displayed in the relayable device list may include the current location of the device. After receiving the service relay instruction, the relay terminal device may execute the content service relay operation corresponding to the content service relay data carried in the service relay instruction.

In some embodiments, the master bedroom sound box is an initiator device, and the first application is an audio playback application. The relayable devices found in the home LAN include: display devices on the second floor, speakers in the second bedroom, and other devices. If the user currently intends to go to the second bedroom, then by selecting the second bedroom speaker in the relayable device list, input the service relay instruction, and send the service relay instruction to the second bedroom speaker. After the speaker in the second bedroom receives the service relay instruction, it will relay the audio.

If the initiator device has a display, it can display function option selection bars such as "delay 5s playback" and "continue playback on the original device" on the user interface. For example, in the user interface shown in Figure 9, if the user selects the "Delay 5s Play" option, the initiator device needs to wait for 5 seconds after receiving the service relay command input by the user before sending the service relay command to the relayable device . During the 5s waiting process, the user can cancel the relay operation at any time. For example, when the initiator device is a display device, the user may input a command to cancel service relay by pressing a return button on the control device. In this way, the situation that the user sends the service relay instruction by mistake can be avoided. After a waiting process of 5 seconds, if the cancel service relay instruction input by the user is not received, the service relay instruction is sent to the selected relay capable device.

If the user selects "the original device continues to play", the selected device that can be relayed receives the service relay command and starts to perform the relay operation. At the same time, the initiator device continues the operation before the relay. For example, in the application scenario shown in FIG. 17 , after the display device 100-2 performs a relay operation to play the video, the display device 100-1 continues to play the video. In this way, the same video can be played simultaneously on different terminal devices. At this time, the process and effect of playing the same video on different terminal devices do not affect each other.

In some embodiments, after the initiator device sends the service relay instruction to the relay device, the relay device performs the relay operation. The relay end device can also serve as an initiator device, and send a service relay command to other relay end devices, so that other relay end devices can also perform relay operations. The initiator device can also send a service relay command to multiple relay devices at the same time, so that multiple relay devices can perform relay operations.

In some embodiments, the home local area network includes multiple display devices: a display device in the living room, a display device in the master bedroom, a display device in the second bedroom, and a display device in the guest bedroom. The display device in the living room is used as the initiator device. When using the video playback application to play video, it can send service relay commands to the other three display devices at the same time, so that the other three display devices will perform the relay operation after receiving the service relay command, that is, perform the video playback operation. . It can also be that the display device in the living room first initiates a receiving command to the display device in the master bedroom, and after the display device in the master bedroom executes the video playback operation, the display device in the master bedroom sends a service relay command to the display device in the second bedroom, so that the display device in the second bedroom executes the video playback operation. operate. Finally, the display device in the second bedroom sends a service relay command to the display device in the guest bedroom, so that the display device in the guest bedroom executes the operation of playing video. The user can set each terminal device to automatically send the service relay command, or the user can input the service relay command, and then send a relay request to other terminal devices according to the service relay command.

When the user finds that the current service relay command is sent by mistake, he can continue to input the relay interrupt operation, and the device generates and receives the relay interrupt command according to the interrupt operation. The relay interrupt command carries business information and the device ID of the relay device, and the AIOT server sends the interrupt command and business information to the corresponding relay device according to the device ID. The relay end device does not execute the content service relay operation corresponding to the content service relay data according to the relay interruption instruction. The relay terminal device needs to receive the relay interruption command before the relay terminal device executes the content service relay operation corresponding to the content service relay data. In addition, the application installed on the relay end device for the user to perform the relay operation may be in an open state or not in an open state. If the interruption instruction is not received after receiving the service relay instruction, the relay end device starts the unopened relay application according to the relay service instruction, so as to perform the corresponding relay operation.

In some embodiments, the relay interruption instruction may be an instruction generated according to an interruption operation input by the user on the relay end device. After the relay device receives the relay service instruction, it can display a relay dialog box on the relay device, and the relay dialog box includes a relay confirmation button control and a relay cancel button control. If the user clicks the confirmation relay button control, the content service relay operation corresponding to the content service relay data is executed. If the user clicks the cancel relay button control, a relay interruption instruction is generated according to the user operation, and the relay device does not execute the content service relay operation corresponding to the content service relay data according to the relay interruption instruction.

The relay countdown dialog box can also be displayed on the relay end device, and the relay countdown time confirmation button control and the relay button control cancel button control are displayed on the relay countdown dialog box. The user clicks the confirm relay button control or the cancel relay button control, and inputs the operation of confirming relay or canceling relay, and the relay end device generates a confirmation service relay command or a relay interrupt command correspondingly. If the time exceeds the countdown time and the user does not click any button control, a relay service instruction can be automatically generated.

In some embodiments, the relay interruption instruction may also be an instruction generated according to an interruption operation input by the user on the initiator device. After the relay device receives the relay service instruction, before executing the content service relay operation corresponding to the content service relay data, the relay dialog box is displayed on the initiator device (the trigger condition for displaying the relay dialog box can be that the relay terminal device receives the relay service After the instruction, the result of receiving the relay service instruction is fed back to the initiator device, and the trigger condition for displaying the relay dialog box can also be that the relay service instruction is issued by itself). A Cancel Relay button control and a Confirm Relay button control are displayed on the relay dialog. If the user clicks the button control to cancel the relay, a relay interruption instruction is generated according to the user operation. And send the relay interruption command to the relay end device. If the user clicks the confirmation relay button control, a confirmation service relay instruction is generated according to the user operation. The relay dialog can display a countdown. For example, if the user does not click any button control within 5 seconds, the relay dialog will be canceled. The interruption operation in the embodiment of the present application is assumed to be performed before the relay operation is performed by the relay terminal device.

A sound collector can be set on the initiator device, and the sound collector receives voice signals input by the user. For example, the voice text "Cancel Handoff" is recognized from the voice signal input by the user. The initiating device generates a relay interrupt command based on the voice text, which carries business information and the device ID of the relay device, and the AIOT server sends the interrupt command and business information to the corresponding relay device according to the device ID. The relay end device does not execute the content service relay operation corresponding to the content service relay data according to the relay interruption instruction.

In some embodiments, after the relay end device does not perform the content service relay operation corresponding to the content service relay data, it also feeds back the relay result to the initiator device. The initiator device may continue to perform the content service relay operation corresponding to the content service relay data according to the relay result fed back. That is, the operation before the relay is continued. After the initiating device sends a service relay instruction to the relaying device, the initiating device may or may not interrupt the previous operation.

For example, the display device is playing a video before the relay, and when sending a service relay command to the relay terminal device, the operation of playing the video may be interrupted, or the operation of playing the video may not be interrupted. If the operation of playing the video is interrupted, the operation of playing the video can be continued after receiving the relay result fed back by the relay end device (the content service relay operation corresponding to the content service relay data has not been executed). A dialog box may also pop up on the display of the initiator device to prompt the user whether to continue playing the video on the initiator device. A "Yes" button control and a "No" button control can be set on the pop-up dialog box. If the user selects the "Yes" button control, the operation of playing the video will continue according to the operation input by the user. If the user selects the "No" button control, according to the operation entered by the user, the operation of playing the video will not continue.

The following describes the process of service relay interruption in combination with the implementation process diagram of service relay interruption as shown in Figure 30: First, after the system is started, the unified relay service application compiles the relay capabilities of each business application in the entire system, and The relay capability of each business application is reported to the AIOT server.

D1 to D8 are similar to the processes of B1 to B8 in the foregoing embodiments, and the process of D1 to D8 will not be repeated in this example.

In the embodiment of the present application, the relay activity may be interrupted before the relay device performs a corresponding action. As shown in D9-D10 in Figure 31, it is the process of interrupting the service relay at the relay end device.

D9: Before the D8 process, the user inputs an interrupt command on the relay device, and the relay device does not execute the corresponding action, that is, the D8 process is not executed. And the unified relay service application of the relay end device sends the interrupted assembly data to the AIOT server. Wherein, the assembly data includes interruption result information, service information, and device information of the initiator device.

D10: AIOT will interrupt the result information and business information, and send them to the unified relay service application of the initiator device according to the device information. Then, according to the interruption result information and the service information, the prompt information of the interruption of the relay service is displayed on the initiator device. If other applications have been switched on the current initiating device, the relay service interruption prompt information related to the interruption result information and service information may also be displayed on the initiating device.

As shown in D11-D12 in Figure 21, it is the process of interrupting the service relay at the initiator device.

D11: The user interrupts the relay through the interrupt relay button or voice input on the initiator device to trigger the relay interruption, and sends the relay interruption command to the AIOT server. The relay interrupt command carries business information (application service ID) and device information of the relay end device.

D12: After the AIOT server finds the relay end device according to the device information, it sends the relay interrupt command to the relay end device. The unified relay service application of the relay end device queries the corresponding application service according to the application service relay ID, and cancels the activation of the application service, so that the relay end device does not perform the corresponding relay action.

After the relay end device starts to perform corresponding operations according to the content service relay data, the user can end the relay at any time. For example, the user may input an instruction to end service relay on the relay device, and the relay device will stop performing corresponding operations according to the content service relay data after receiving the end service relay instruction. Alternatively, the user inputs an end service relay command on the initiating device, and the initiating device sends the end service relay command to the relay device. After the relay end device receives the end service relay instruction word from the initiator device, it stops performing corresponding operations according to the content service relay data. The way for the user to input the command to end the service relay may be voice input, button selection input, gesture input, etc., which is not limited in this application.

In the application scenario shown in FIG. 17 , the display device 100 - 2 plays the video in relay after receiving the relay request. At this time, a relay interruption button may be displayed on the display device 100-1 or the display device 100-2. When the user selects the relay interruption button on the display device 100-1 or the display device 100-2, a relay interruption instruction is input. If the relay interruption button is selected on the display device 100-1, a relay interruption instruction is input on the display device 100-1. The display device 100-1 then sends the relay interruption instruction to the display device 100-2. The display device 100-2 ends the relay play video according to the relay interruption instruction. If the relay interruption button is selected on the display device 100-2, the relay interruption instruction is directly input on the display device 100-2. Finally, the display device 100-2 ends the relay play video according to the relay interruption instruction.

After the relay end device finishes performing the corresponding operation according to the content service relay data, that is, after the content service relay operation ends, the relay end device can generate a relay end feedback result and send the relay end feedback result to the initiator device to realize the content service Relay operation. The relay end feedback result includes the execution progress data of the content service relay operation performed by the relay end device, and the initiator device can continue to execute the content service relay operation according to the execution progress data. For example, the media asset content can be played continuously on the initiator device according to the progress of the media asset content on the relay device.

In the application scenario shown in FIG. 17 , after the display device 100 - 2 ends the relay play of the video according to the relay interruption instruction, it feeds back the playback progress of the current video to the display device 100 - 1 . For example, when the relay is over, the playing progress of the video M on the display device 100-2 is 1 hour, 10 minutes and 6 seconds. The display device 100-2 feeds back the video playback progress of 52% to the display device 100-1. After the display device 100-1 receives the feedback of the playback progress, it continues to play the video M from 1 hour, 10 minutes and 6 seconds.

In some embodiments, if the originator device is currently in the first state, the originator device can continue to execute the content service relay operation according to the execution progress data, and then the originator device continues to execute the content service relay operation according to the execution progress data.

In some embodiments, if the initiator device is currently in the second state, the initiator device cannot continue to execute the content service relay operation according to the execution progress data, then the originator device does not continue to execute the content service relay operation according to the execution progress data.

The signaling processing sequence of the above embodiment is as shown in the signaling diagram in Figure 31:

Step 1. The initiator device sends a relay service command (carrying the service information of the application currently running in the foreground of the initiator device and the device information of the relay device) to the AIOT server.

Step 2: The AIOT server sends the business information and relay service instructions to the corresponding relay terminal device according to the device information.

Step 3: The relay terminal device receives the service information, and performs a corresponding relay operation according to the service information. The service information may include the basic information of the service and the execution progress data of the service on the relay device.

For example, if the application currently running in the foreground of the initiator device is an application related to media asset playback, the service information may include basic information such as the name of the media asset data, as well as the time when the media asset is played on the initiator device during relay. The relay device continues to play the media asset according to the above basic information and playing time.

Step 4: After receiving the instruction to interrupt the relay operation, the relay terminal device does not continue to perform the corresponding relay operation. After interrupting the execution of the relay operation, the device at the relay end generates a relay end feedback result. The feedback result of the completion of the relay may include basic information of the business and execution progress data of the business when the relay terminal device interrupts the execution of the relay operation.

For example, the feedback result of the relay end includes service information and device information of the initiator device. The service information at this time may also include the name of the media asset data, and also include the playing time of the media asset on the relay end device when the relay end device interrupts the relay operation.

Step 5: The AIOT server receives the relay end feedback result sent by the relay end device, and sends the relay end feedback result to the initiator device.

Step 6. If the initiator device is in the first state when receiving the relay end feedback result, continue to execute the relay operation according to the execution progress data, wherein, when the initiator device is in the first state, the initiator device can continue according to the execution progress data Perform content service relay operations.

Step 7. If the initiator device is in the second state when receiving the relay end feedback result, it does not continue to execute the relay operation according to the execution progress data. Proceed to the content service relay operation.

In this way, in the content service relay return scenario, the current state of the initiator device can be fully considered to avoid affecting the normal use of the initiator device, thereby improving user experience. The initiator device and the relay device can directly transmit instructions and data, or forward instructions and data through the server, which is not limited in this application.

When the initiator device is in the first state, it may be in the following different situations:

The first situation is that when receiving the feedback result of the relay end, the initiating device is currently in the first service content scene, wherein the first service content scene is the service content scene in which the initiating device was in before the relay.

For example, before the relay, the application running in the foreground on the initiator device is a video application. When receiving the relay end feedback result, the application running in the foreground on the initiating device is still a video application. Then the initiator device can continue to execute the content service relay operation according to the execution progress data, that is, continue to play the media asset content.

If the application running in the foreground on the initiator device is a video application, the media content played by the video application on the initiator device is not the media content played before the relay. After the initiating device finishes playing the currently played media content, the previous media content can be played according to the execution progress data.

The second situation is that when receiving the relay end feedback result, the initiating device is not currently in any service content scene. For example, there may be no applications running in the foreground on the initiator device. Then the initiator device can continue to execute the content service relay operation according to the execution progress data.

When the initiator device is in the second state, it may be in the following different situations:

The first situation is that when receiving the feedback result of the end of the relay, the initiator device is currently in the second business content scene, the second business content scene is different from the first business content scene, and the first business content scene is the originating end device before the relay. The business content scenario in which the device is located. For example, before the relay, the application running in the foreground on the initiator device is a video application. When receiving the relay end feedback result, the application running in the foreground on the initiator device is an audio application. Then the initiator device cannot continue to execute the content service relay operation according to the execution progress data, that is, the initiator device cannot continue to play the media asset content.

The second situation is that the relay initiating terminal is currently offline. At this time, the initiating relay terminal cannot continue to execute the content service relay operation according to the execution progress data, that is, the initiating device cannot continue to play the media content. The feedback result of the relay end can be saved on the server, and when the initiator device enters the online state from the offline state, it can send a state change request to the server. After receiving the status change request of the initiator device, the server determines that the status of the initiator device is online, and then sends the relay end feedback result to the initiator device. Further, if the initiator device is not in any business content scene or in the first business content scene after entering the online state, it can continue to execute the content service relay operation according to the execution progress data.

In some embodiments, when receiving the relay end feedback result, if the initiating device is currently in the first state in the above embodiment, it is determined according to the execution progress data that the relay device has completed the corresponding execution based on the content service relay data. Content service relay operation. Then the initiator device also does not continue to execute the content service relay operation according to the execution progress data.

In some embodiments, if the initiator device is in the first state and receives the feedback result of the completion of the relay, the initiator device may continue to execute the content service relay operation according to the execution progress data. Then the user can perceive the operation change on the initiator device. For example, if the media asset content continues to be played according to the execution progress data, the user can watch the media asset content that continues to be played on the initiator device. However, if the initiating device is in the second state, after receiving the relay end feedback result, the initiating device cannot continue to execute the content service relay operation according to the execution progress data. Then the user cannot perceive the operation change on the initiator device.

At this point, a return failure prompt may be generated on the initiator device to prompt the user that the relay return of the current content service fails. For example, a dialog box may be popped up on the relay device and/or the initiator device, and the dialog box includes a prompt saying "relay return of content service failed". The relay end device and/or the originator device can also voice broadcast the return failure prompt, for example, the voice broadcast prompt "content service relay return failure", so as to further improve the user's interactive experience.

This application also provides a content service relay return method, as shown in Figure 32, the flow chart of the content service relay return method, the method includes the following steps:

Step S101 , sending a service relay command to the relay device. The service relay command carries content service relay data, so that the relay device executes a content service relay operation corresponding to the content service relay data.

Step S102: Receive the relay end feedback result sent by the relay end device, wherein the relay end feedback result is the result generated by the relay end device after the content service relay operation is completed, and the relay end feedback result includes the execution of the content service relay operation by the relay end device progress data.

Step S103, when the initiator device is currently in the first state, continue to execute the content service relay operation according to the execution progress data, wherein, when the originator device is in the first state, the initiator device can continue to execute the content service relay operation according to the execution progress data operate.

Step S104, when the initiator device is currently in the second state, continue to execute the content service relay operation according to the execution progress data, wherein, when the initiator device is in the second state, the originator device cannot continue to execute the content service according to the execution progress data Relay operation.

In some embodiments, when the initiator device is currently in the first state, the initiator device is currently in the first service content scenario, or the initiator device is not currently in any service content scenario. Wherein, the first service content scene is the service content scene where the initiator device is in before the relay.

In some embodiments, when the initiator device is currently in the second state, the initiator device is currently in the second service content scenario, or the initiator device is currently in an offline state. Wherein, the second business content scenario is different from the first business content scenario.

The present application also provides a content display method of a display device, as shown in the flow chart of the method in FIG. 33 , the method includes the following steps:

Step S201 , run the first application in the foreground on the initiator device, and search for a relayable device according to the application data of the first application, wherein the relayable device is a terminal device that can perform operations corresponding to the application data of the first application.

Step S202 , if there is a relayable device, display a relay button control on the user interface of the first application, wherein the relay button control is used to trigger a content service relay process.

Step S203 , if there is no relay-capable device, display a relay guide button control on the user interface of the first application, wherein the relay guide button control is used to trigger display of content service relay guide information.

In this way, when there is no terminal device that can relay the content service of the current smart device, the relay guidance button control can be displayed on the user interface, and the user can be guided to perform the next step related to the content service relay, thereby improving the user experience.

In some embodiments, different relay button controls may be displayed depending on the type of the first application. For example, if the first application is a video application, a video relay button control is displayed on the user interface of the first application. If the first application is an audio application, an audio relay button control is displayed on the user interface of the first application. If the first application is a picture display application, a picture display relay button control is displayed on the user interface of the first application.

In some embodiments, the name of the second application on the relay-capable device may also be displayed on the user interface of the first application, and the second application is an application that can perform operations corresponding to the application data of the first application.

In some embodiments, a thumbnail of a user interface of a second application on the relayable device may also be displayed on the user interface of the first application, and the second application is an application that can perform operations corresponding to the application data of the first application.

The present application also provides a terminal device matching method, which is applied in the content service relay scenario, as shown in the method flow chart in Figure 34, and the method includes the following steps:

Step S301, the current terminal device (initiating device) receives the voice signal from the sound collector, and recognizes the voice text from the voice signal (recognition of the voice text through the voice server).

Step S302, if the voice text includes device information, determine the relay device according to the device information, where the device information points to a unique relay device.

Step S303, if the voice text includes device type information, determine the relayable device according to the device type information, and determine the relayable device from among the relayable devices according to the relationship between the relay dimension information of the relayable device and the initiation dimension information of the current terminal device , where the device type information points to at least two relayable devices.

Step S304, establishing a relay relationship between the determined relay device and the current terminal device.

In some embodiments, the dimension information of the device includes a first dimension, a second dimension, and a third dimension, the first dimension includes the second dimension, and the second dimension includes the third dimension. The first dimension is the account number, the second dimension is the room attribute identification, for example, it can be the user's house in different regions, and the third dimension is the room area identification, for example, it can be the user's different rooms in different houses. The account is the account ID registered by the user in the unified relay service application. Users can create multiple homes under each account dimension. For example, a user owns multiple houses, which are located in A, B, and C respectively. The houses on land A, land B, and land C are the second dimension in the frame diagram. The third dimension is a plurality of physical spaces established based on the home dimension, such as master bedroom, living room, second bedroom, kitchen and other spaces. Then, the step of determining the relay device according to the relationship between the relay dimension information of the relayable device and the second space of the terminal device may be:

If there is a relayable device whose first dimension, second dimension, and third dimension respectively match the first dimension, second dimension, and third dimension of the terminal device, determine the relayable device as the relaying device;

Or, if there is no relayable device whose first dimension, second dimension, and third dimension respectively match the first dimension, second dimension, and third dimension of the terminal device, and there is a first dimension and second dimension that are respectively compatible with the terminal A relayable device whose first dimension and second dimension match the relayable device is determined as the relayable device.

Or, if there is no relayable device whose first dimension, second dimension, and third dimension respectively match the first dimension, second dimension, and third dimension of the terminal device, and there is no first dimension, second dimension, and A relayable device whose first dimension matches the second dimension of the terminal device, and a relayable device whose first dimension matches the first dimension of the terminal device exists simultaneously, and the relayable device is determined as the relaying device.

That is to say, when determining the relay device, the relay-capable devices whose first dimension, second dimension and third dimension respectively match the first dimension, second dimension and third dimension of the current terminal device are considered first. If there is no relayable device that matches all three dimensions, further consider a relayable device whose first dimension and second dimension respectively match the first dimension and the second dimension of the current terminal device. If there is no first dimension and second dimension matching the first dimension and the second dimension of the current terminal device respectively, finally consider the relayable device whose first dimension matches the first dimension of the current terminal device. If there is no relay capable device whose first dimension matches the first dimension of the current terminal device, it means that there is no relay capable terminal device in the system. In this way, a relay device with the same room area identifier or the same room attribute identifier as the current device can be matched as much as possible.

The present application also provides a content service relay interruption method, as shown in the method flowchart shown in FIG. 35 and the method signaling diagram shown in FIG. 36. The method includes the following steps:

Step S401, the relay end device receives a relay service instruction from the initiator device. The relay service command carries content service relay data, and the relay end device can perform the content service relay operation corresponding to the content service relay data.

Step S402. Before the relay end device executes the content service relay operation corresponding to the content service relay data, if the relay end device receives the relay interruption instruction, the relay end device will not execute the corresponding content service relay data according to the relay interruption instruction. Content service relay operation.

In this way, even if the relay terminal device has received the service relay command, it can not execute the relay operation according to the relay interruption command, that is, if the user accidentally triggers the content service relay process, the content service relay will be interrupted in time, thereby improving the user experience. In the method embodiment of the present application, between terminal devices, instructions and data may be transferred through a server, or may not be transferred through a server, but the instructions and data are directly transferred between terminal devices.

The relay interruption instruction may be an instruction generated according to an interruption operation input by the user on the relay end device. For example, after the relay device receives the relay service instruction, it may display a relay dialog box on the relay device, and the relay dialog box includes a relay confirmation button control and a relay relay button control. If the user clicks the confirmation relay button control, the content service relay operation corresponding to the content service relay data is executed. If the user clicks the button control to cancel the relay, a relay interruption instruction is generated according to the user operation.

The relay interruption instruction may also be an instruction generated according to an interruption operation input by the user on the initiator device. For example, after receiving the relay service instruction, the relay device displays a relay dialog box on the initiator device before executing the content service relay operation corresponding to the content service relay data. A Cancel Relay button control is displayed on the relay dialog. If the user clicks the button control to cancel the relay, a relay interruption instruction is generated according to the user operation. And send the relay interruption command to the relay end device.

The present application also provides a method for reporting service capability information of a terminal device, as shown in the signaling diagram in FIG. 37 , the method includes the following steps:

Step 1, the terminal device monitors whether there is a change in the service parameters of the applications installed on its system.

Step 2: If there is a change in the service parameter, a service capability information update request is generated according to the change, and the service capability information update request is sent to the server. The server updates the service capability information list stored on the server according to the service capability information update request. The service capability information list is used to record the service capability information of the terminal equipment, and the service capability information corresponds to service parameters.

Step 3: If there is no change in the service parameters, no service capability information update request is generated. Then the business capability information list saved on the server does not change.

In this way, content service relay between terminal devices can be performed according to the actual service capabilities of the terminal devices, thereby improving user experience.

The present application also provides a content service relay method for a terminal device, such as the signaling diagram shown in FIG. 38 and the method flowchart shown in FIG. 39 , the method includes the following steps:

Step S501 , run the first application in the foreground on the initiator device, and search for a relayable device according to the application data of the first application, wherein the relayable device is a terminal device that can perform operations corresponding to the application data of the first application.

Step S502, if there is a relayable device, generate a relay prompt, so that the user can input a relay operation according to the relay prompt, generate a service relay command according to the relay operation, generate a relay request according to the service relay command, and send the relay request to the relayable device, so that The relay-capable device executes operations corresponding to the application data according to the application data carried in the relay request.

Step S503, if there is no relay capable device, no relay prompt is generated.

The above-mentioned embodiment is a solution implemented based on a local area network, and the internal overall structure of the local area network of the present application is shown in FIG. 40 . The schematic diagram of the internal structure of the local area network shown in FIG. 40 is only an example of two terminal devices, and the two terminal devices can be used as the initiator device or the relay device.

Among them, the relay component provides the following capabilities for the application:

Register the service capability information of this terminal device to other terminals, and obtain service capability information from other terminal devices. When an application of a certain terminal device is not started, the capability parameter of the unstarted application may also be sent to other terminal devices through the hardware support component. It is also possible to feed back the basic online and offline information of the terminal device to other terminal devices through the hardware support component.

Figure 41 to Figure 43 are the signaling diagrams of the interaction between the components of the initiator device and the components of the relay device, in which Figure 41 is the process of registering capability information, Figure 42 is the process of obtaining the relay device, and Figure 43 is the video relay The process of playing. After the application of the terminal device is started (some applications are not self-started after booting, and after the whole machine is started, the application service capabilities installed on the current device will be collected for unified registration), register the device capabilities with the underlying service system through the relay component. And send the registration result to other terminal devices. At the same time, the relay component also subscribes to the application changes on the device.

Taking the video relay process as an example, the interaction process is: through the relay component, query the terminal devices that can be relayed in the LAN, and display the list of terminal devices that can be relayed through the application. After determining the relay device, the application generates a video playback request to request the identification and playback progress of the currently playing video from the relay component, and generates playback data according to the video identification and playback progress, and then passes it to the peer through the relay component and the underlying service system . Due to the query of the relay end device through the relay component and the underlying system service. The relay terminal device has been queried, so after receiving the input operation to trigger the relay, the video playback data is forwarded. Finally, after the relay device receives the video playback data sent by the initiator device, the application in the relay device analyzes and plays it, and feeds back the relay result to the application in the initiator device.

In some embodiments, searching for the relayable device based on the application data of the application can start after the application is started, which can improve system efficiency. The terminal device may contain multiple different applications. If the application is not started, the relayable device will be searched. Query, the data interaction generated during the query process will increase the operating pressure of the terminal device and/or server and/or the relay device, so after the application is started, searching for the relay device based on the application data of the application can alleviate this problem.

In some embodiments, the application data represents the current function. Some applications may not directly start the function after starting. For example, a video application will generally display a list of media resources after the application is started, and start only after the user selects it. Video playback and audio applications also have the same problem. Therefore, in some embodiments, after the application is started, the relay-able device is not directly searched according to the application data of the application. Instead, when the application data corresponding to the function of the application is detected to be invoked, the relay-able device is searched according to the application data of the application. For example, when a video application detects the application data representing the activation of the playback function, it searches for relayable devices based on the application data of the application; when an audio application detects the application data representing the activation of the playback function, it searches for the relayable device based on the application data of the application ; When the alarm clock application detects the application data representing the activation of the alarm clock, it searches for the relayable device according to the application data of the application. In this way, after the application is started, the user can operate normally without invalid query. When the user activates the corresponding function, the device that can be relayed is searched according to the application data of the application, so that the timing of the pop-up of the relay prompt is more accurate, which improves the user experience. experience.

In some embodiments, after the application is started, the device that can be relayed can be found directly according to the application data of the application.

Claims

A terminal device comprising:

Controller, configured as:

Sending a service relay command to the relay terminal device, the service relay command carrying content service relay data, so that the relay terminal device performs a content service relay operation corresponding to the content service relay data;

receiving the relay end feedback result sent by the relay end device, wherein the relay end feedback result is a result generated by the relay end device after the content service relay operation is completed, and the relay end feedback result includes the relay end Execution progress data of the content service relay operation performed by the end device;

Wherein, when the terminal device is currently in the first state, the terminal device can continue to perform content service relay operations, wherein, when the terminal device is in the first state, the terminal device can Continue to execute the content service relay operation according to the above execution progress data;

Wherein, when the terminal device is currently in the second state, the terminal device cannot continue to perform content service relay operations, wherein, when the terminal device is in the second state, the terminal device cannot Continue to execute the content service relay operation according to the execution progress data.
According to the terminal device according to claim 1, when the terminal device is currently in the first state, the terminal device is currently in the first service content scene, wherein the first service content scene is the terminal before the relay The business content scenario in which the device is located.
According to the terminal device according to claim 1, when the terminal device is currently in the first state, the terminal device is not currently in a service content scene.
According to the terminal device according to claim 1, when the terminal device is currently in the second state, the terminal device is currently in the second service content scene, wherein the second service content scene is different from the first service content scene Not the same, the first service content scene is the service content scene that the terminal device is in before the relay.
According to the terminal device according to claim 1, when the terminal device is currently in the second state, the terminal device is currently in an offline state.
The terminal device according to claim 1, when the terminal device is currently in the first state, and the execution progress data indicates that the content service relay operation corresponding to the content service relay data has been executed, the execution progress is not performed according to the execution progress. Data continues to perform content service relay operations.
The terminal device according to claim 1, the controller is further configured to: when the terminal device is currently in the second state,

A return failure prompt is generated, wherein the return failure prompt is used to prompt the user that the current content service relay fails to return.
The terminal device according to claim 1, the controller is further configured to:

Determine relayable equipment.
According to the terminal device according to claim 8, the controller is further configured to:

Finding the relayable device according to the application data of the first application, wherein the first application is an application running in the foreground of the terminal device, and the relayable device is a terminal device capable of performing operations corresponding to the application data.
According to the terminal device according to claim 8, the controller is further configured to:

Receive voice signals from the sound collector;

recognizing speech text from said speech signal;

When the voice text includes device type information, determine the relayable device according to the device type information.
According to the terminal device according to claim 8, the controller is further configured to:

receiving service capability information from other terminal devices, the service capability information corresponding to parameters of applications installed in other terminal devices;

matching the business capability information with the application data;

When the service capability information matches the application data, it is determined that another terminal device is the relayable device.
According to the terminal device according to claim 8, the controller is further configured to:

When there is the relayable device, displaying a relay button control on the user interface of the first application, wherein the relay button control is used to trigger a content service relay process;

When there is no relay capable device, a relay guide button control is displayed on the user interface of the first application, wherein the relay guide button control is used to trigger display of content service relay guide information.
A terminal device according to claim 9,

When the first application is a video application, the relay button control is a video relay button control;

When the first application is an audio application, the relay button control is an audio relay button control;

When the first application is a picture display application, the relay button control is a picture display relay button control.
According to the terminal device according to claim 9, the controller is further configured to: when there is the relayable device,

Displaying the name of the second application on the relay capable device on the user interface of the first application, wherein the second application is an application that can execute the operation corresponding to the application data.
According to the terminal device according to claim 9, the controller is further configured to: when there is the relayable device,

Displaying a thumbnail image of a user interface of a second application on the relayable device on the user interface of the first application, wherein the second application is an application capable of executing an operation corresponding to the application data.
According to the terminal device according to claim 8, the terminal device and the relayable device are under the same account.
According to the terminal device according to claim 8, the controller is further configured to:

When the relayable device exists in the same local area network, a relay prompt is generated;

When the relay capable device does not exist in the same local area network, the relay prompt is not generated.
The terminal device according to claim 1, the controller is further configured to:

Generate an end service relay instruction according to the input instruction to end the relay, and send the end service relay instruction to the relay end device, so that the relay end device ends according to the application data according to the end service relay instruction Perform corresponding operations.
According to the terminal device according to claim 18, the controller is further configured to: after the relay terminal device finishes performing corresponding operations according to the application data,

receiving the execution progress data sent by the relay device, wherein the execution progress data is used to indicate the execution progress of the operation corresponding to the application data when the relay device finishes executing the corresponding operation according to the application data;

Continue to execute the operation corresponding to the application data on the terminal device according to the execution progress data.
According to the terminal device according to claim 8, when the terminal device further includes a display, and there are at least two relayable devices, a list of relayable devices is displayed on the display, and the list of relayable devices includes at least Two of said relayable devices.
According to the terminal device according to claim 8, the controller is further configured to: determine the relay terminal device.
According to the terminal device according to claim 21, the service relay instruction is an instruction input by the user by selecting one of the relayable devices in the relayable device list, and the selected relayable device is to receive the relay request terminal equipment.
According to the terminal device according to claim 21, the terminal device further stores the frequency of use of the relayable devices, and the relayable device with the highest frequency of use is the relay terminal device that receives the relay request.
The terminal device according to claim 21, the controller is further configured to:

receiving the speech signal from the sound collector, and recognizing speech text from the speech signal;

When the voice text includes device information, the relay device is determined according to the device information, wherein the device information points to the unique relay device.
The terminal device according to claim 22, the controller is further configured to:

determining the relay end device from the relay capable devices according to the relationship between the relay dimension information of the relay capable device and the dimension information initiated by the terminal device,

Wherein, the dimension information is used to characterize the location area of the device, including the first dimension, the second dimension, and the third dimension respectively used to characterize the first location area, the second location area, and the third location area of the device, wherein, Determining a first location area according to a first dimension, determining a corresponding second location area from said first location area determined according to a second dimension, and determining from said second location area determined according to a third dimension corresponding third location area.
The terminal device according to claim 25, wherein if there is a relayable device whose first dimension, second dimension and third dimension all match the initiated first dimension, second dimension and third dimension, then the relayable A relay device is determined as the relay terminal device.
The terminal device according to claim 25, wherein when the first dimension, the second dimension and the third dimension do not exist, they are related to the initiated first dimension, the second dimension and the When the relayable device matches the third dimension, if there is the relayable device whose first dimension and the second dimension both match the initiated first dimension and the second dimension, then the The relay capable device is determined as the relay terminal device.
The terminal device according to claim 27, wherein, when there is no said first dimension and said second dimension matching said first dimension and said second dimension of said terminal device, said relayable When connecting a device, if there is a relayable device whose first dimension matches the initiated first dimension, the relayable device is determined to be the relay terminal device.
The terminal device according to claim 28, wherein when there is no relayable device that matches the first dimension of the terminal device, if there is the second dimension and the The relayable device whose second dimension matches the terminal device, or if there is the relayable device whose third dimension matches the third dimension of the terminal device, generating prompt information to After the user is made to add the relayable device to the first dimension according to the prompt information, the relayable device is determined as the relay terminal device.
A terminal device comprising:

Controller, configured as:

receiving a service relay instruction from the initiator device, wherein the service relay instruction carries content service relay data, and the terminal device can execute a content service relay operation corresponding to the content service relay data;

Receiving a relay interruption instruction, and according to the relay interruption instruction, not executing a content service relay operation corresponding to the content service relay data.
According to the terminal device according to claim 30, the relay interruption instruction is an instruction generated by the initiator device according to an input interruption operation.
According to the terminal device according to claim 30, the relay interruption instruction is an instruction generated by the terminal device according to an input interruption operation.
The terminal device according to claim 32, the terminal device further comprising a display, and the controller is configured to:

Displaying a confirmation relay dialog box on the display, wherein the confirmation relay dialog box includes a confirmation relay button control and a cancel relay button control;

When the relay cancel button control is selected, the relay interruption instruction is generated.
According to the terminal device according to claim 30, the controller is further configured to:

After not executing the content service relay operation corresponding to the content service relay data, feed back the relay result to the initiator device, so that the initiator device continues to execute the content service relay operation corresponding to the content service relay data according to the relay result. Content service relay operation.
According to the terminal device according to claim 30, the controller is further configured to:

When there is a change in the service parameter of the application installed on the terminal device, generate a service capability information update request according to the change, and send the service capability information update request to the server, so that the server can use the service capability information according to the change. The update request updates the service capability information list, where the service capability information list is used to record the service capability of the terminal device, and the service capability information corresponds to the service parameter;

When there is no change in the service parameters of the applications installed on the terminal device, the service capability information update request is not generated.
According to the terminal device according to claim 35, the controller is configured to: when determining whether there is an update of the service parameters of the application installed on the terminal device,

When the system of the terminal device is started, query the current service parameters of the applications installed on the terminal device, and determine the There are changes in the service parameters of the applications installed on the terminal.
The terminal device according to claim 36, the controller is configured to:

When the system of the terminal device is started, a start-up broadcast sent by the system is received, and the current service parameters of the applications installed on the terminal device are inquired according to the start-up broadcast.
According to the terminal device according to claim 36 or 37, when the system of the terminal device is started, the startup mode of at least one application installed on the terminal device is not the boot-up self-startup mode.
According to the terminal device according to claim 35, the controller is configured to: when determining whether there is a change in a service parameter of an application installed on the terminal device,

After the system of the terminal device is started, when an update broadcast sent by the application installed on the terminal device is received, it is determined that there is a change in the service parameter of the application installed on the terminal device.
The terminal device according to claim 35, the controller is configured to:

When the change is application installation, the service capability information update request is a service capability information addition request, so that the server adds service capability information to the service capability information list according to the service capability information addition request, wherein, The added service capability information corresponds to the service parameter of the installed application.
The terminal device according to claim 35, the controller is configured to:

When the change is an application version change, the service capability information update request is a service capability information modification request, so that the server modifies the service capability information in the service capability information list according to the service capability information modification request, Wherein, the modified service capability information corresponds to the service parameter of the application whose version changes.
The terminal device according to claim 35, the controller is configured to:

When the change is application uninstallation, the service capability information update request is a service capability information deletion request, so that the server deletes the service capability information from the service capability information list according to the service capability information deletion request, wherein , the deleted service capability information corresponds to the service parameter of the uninstalled application.
A content service relay method, comprising:

The terminal device sends a service relay command to the relay terminal device, the service relay command carries content service relay data, so that the relay terminal device performs a content service relay operation corresponding to the content service relay data;

The terminal device receives the relay end feedback result sent by the relay end device, wherein the relay end feedback result is a result generated by the relay end device after the content service relay operation is completed, and the relay end feedback result Including the execution progress data of the content service relay operation performed by the relay terminal device;

Wherein, when the terminal device is currently in the first state, the terminal device can continue to perform content service relay operations, wherein, when the terminal device is in the first state, the terminal device can Continue to execute the content service relay operation according to the above execution progress data;

Wherein, when the terminal device is currently in the second state, the terminal device cannot continue to perform content service relay operations, wherein, when the terminal device is in the second state, the terminal device cannot Continue to execute the content service relay operation according to the execution progress data.
A content service relay interruption method, comprising:

The terminal device receives a service relay instruction from the initiator device, wherein the service relay instruction carries content service relay data, and the terminal device can perform a content service relay operation corresponding to the content service relay data;

Receiving a relay interruption instruction, and according to the relay interruption instruction, not executing a content service relay operation corresponding to the content service relay data.