CN115964011A

CN115964011A - Method and related device for displaying application interface based on multi-screen cooperation

Info

Publication number: CN115964011A
Application number: CN202310250876.0A
Authority: CN
Inventors: 金国春; 石彬
Original assignee: Shenzhen Xfanic Technology Co Ltd
Current assignee: Shenzhen Xfanic Technology Co Ltd
Priority date: 2023-03-16
Filing date: 2023-03-16
Publication date: 2023-04-14
Anticipated expiration: 2043-03-16
Also published as: CN115964011B

Abstract

The application provides a method and a related device for displaying an application interface based on multi-screen cooperation, wherein the method comprises the following steps: according to a first screen projection request of first terminal equipment, first data of the first terminal equipment are sent to second terminal equipment, the second terminal equipment displays an interface of the first terminal equipment according to the first data, a first operation event generated according to operation on a first function module of an adapter is obtained, a first control instruction corresponding to the first operation event and a first application identifier corresponding to the first function module are sent to the first terminal equipment, second data generated by the first terminal equipment are sent to the second terminal equipment, and therefore the second terminal equipment displays the interface generated by the first terminal equipment executing the first control instruction. In the application, the function module is included on the adapter for realizing screen projection between the first terminal device and the second terminal device, the first terminal device is quickly controlled through the function module, and the operation efficiency is improved while the screen is stably projected.

Description

Method and related device for displaying application interface based on multi-screen cooperation

Technical Field

The application belongs to the technical field of communication, and particularly relates to a method for displaying an application interface based on multi-screen cooperation and a related device.

Background

At present, interaction between terminal devices is more and more frequent, and scenes used in association between terminal devices are more and more, for example, a plurality of terminals are mutually matched to realize a cross-device multi-screen cooperation scene. In an existing multi-screen cooperative use scene, a first terminal device often projects contents displayed in a display screen of the first terminal device to a second terminal device through a network, however, the quality of the projected screen is often influenced by the network, which results in unstable projected screen quality and low safety.

Disclosure of Invention

The application provides a method and a related device for displaying an application interface based on multi-screen cooperation, wherein a plurality of terminal devices are connected through an adapter provided with a plurality of functional modules, so that multi-screen cooperation among the plurality of terminal devices is realized, the stability and the safety of a screen projection process are improved, and the intelligence of the adapter is improved by quickly controlling applications in the connected terminal devices through the functional modules.

In a first aspect, the present application provides a method for displaying an application interface based on multi-screen collaboration, which is applied to an adaptor of a multi-screen collaboration system, where the multi-screen collaboration system includes at least one first terminal device and at least one second terminal device, each first terminal device in the at least one first terminal device is connected to a first interface aggregator of the adaptor, each second terminal device in the at least one second terminal device is connected to a second interface aggregator of the adaptor, the adaptor is configured to transmit screen projection display data of the first terminal device to the second terminal device in real time, the adaptor includes at least one function module, and the at least one function module represents a module for controlling applications in the first terminal device, and the method includes:

responding to a first screen projection request of the first terminal device, and acquiring first data of the first terminal device, wherein the first data represents interface display data of the first terminal device at the current time point;

sending the first data to the second terminal device, so that the second terminal device displays an interface of the first terminal device at the current time point;

if the fact that a user operates on a first functional module of the at least one functional module is detected, generating a first operation event according to the operation of the user on the first functional module;

acquiring a first control instruction set corresponding to the first function module, and searching a first control instruction corresponding to the first operation event from the first control instruction set, where the first control instruction set includes a correspondence between different first operation events and different control instructions, and the control instruction represents an instruction for controlling an application of the first terminal device to execute a preset operation;

searching a first application identifier corresponding to the first function module from a preset application and module corresponding relation table, wherein the application and module corresponding relation table comprises corresponding relations between different function modules and different application identifiers;

sending the first application identifier and the first control instruction to the first terminal device, so that the first terminal device executes the first control instruction for a first application corresponding to the first application identifier;

receiving second data of the first terminal device, wherein the second data represent interface display data generated by the first terminal device according to the first control instruction for controlling the first application;

and sending the second data to the second terminal equipment so that the second terminal equipment displays an interface generated by the first terminal equipment executing the first control instruction according to the second data.

In a second aspect, the present application provides an apparatus for displaying an application interface based on multi-screen coordination, where the apparatus includes an adaptor, where the adaptor includes at least one function module, where the function module is used to control terminal devices connected to a first interface aggregator and a second interface aggregator of the adaptor, and the adaptor is used to project an interface of at least one first terminal device connected to the first interface aggregator onto a screen of at least one second terminal device connected to the second interface aggregator in real time, and the apparatus includes:

a first obtaining unit, configured to obtain first data of a first terminal device in response to a first screen projection request of the first terminal device, where the first data represents interface display data of the first terminal device at a current time point, the first terminal device is a device of the at least one first terminal device, and the second terminal device is a device of the at least one second terminal device;

the first sending unit is used for sending the first data to the second terminal device so that the second terminal device can display an interface of the first terminal device at the current time point;

the processing unit is used for generating a first operation event according to the operation of the user on a first functional module in the at least one functional module if the operation of the user on the first functional module is detected;

a second obtaining unit, configured to obtain a first control instruction set corresponding to the first function module, and search for a first control instruction corresponding to the first operation event from the first control instruction set, where the first control instruction set includes a correspondence between different first operation events and different control instructions, and the control instruction represents an instruction for controlling an application of the first terminal device to execute a preset operation;

the searching unit is used for searching a first application identifier corresponding to the first function module from a preset application and module corresponding relation table, and the application and module corresponding relation table comprises corresponding relations between different function modules and different application identifiers;

an execution unit, configured to send the first application identifier and the first control instruction to the first terminal device, so that the first terminal device executes the first control instruction for a first application corresponding to the first application identifier;

the receiving unit is used for receiving second data of the first terminal device, wherein the second data represent interface display data generated by the first terminal device according to the first control instruction;

and the second sending unit is used for sending the second data to the second terminal equipment so as to enable the second terminal equipment to display an interface generated by the first terminal equipment executing the first control instruction according to the second data.

In a third aspect, the present application provides an adaptor, the adaptor includes a micro control unit, a first interface aggregator, a second interface aggregator, at least one function module, a first conversion module, and a second conversion module, the first interface aggregator includes at least one interface, the second interface aggregator includes at least one interface, the first interface aggregator is connected to a first conversion module, the first conversion module is connected to the second conversion module, the second conversion module is connected to the second interface aggregator, the micro control unit is connected to each of the first conversion module, the second conversion module, and the at least one function module, and is configured to control each of the first conversion module, the second conversion module, and the at least one function module, the adaptor is connected to at least one first terminal device through the first interface aggregator, the first conversion module is configured to convert first data of the at least one first terminal device received by the first interface aggregator into second data, the second conversion module is configured to convert the second data of the at least one first terminal device into second data, and the second conversion module is configured to obtain second data, and the second conversion module is configured to perform the third conversion step of the second conversion module.

In a fourth aspect, the present application provides an electronic device, comprising: one or more processors;

one or more memories for storing programs,

the one or more memories and the program are configured to control the electronic device, by the one or more processors, to execute the instructions of the steps in any of the methods of the first aspect of the embodiments of the present application.

In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, and wherein the computer program causes a computer to perform some or all of the steps as described in any one of the methods of the first aspect of the embodiments of the present application.

In a sixth aspect, the present application provides a computer program, wherein the computer program is operable to cause a computer to perform some or all of the steps as described in any of the methods of the first aspect of the embodiments of the present application. The computer program may be a software installation package.

The beneficial effects brought by the technical scheme provided by some embodiments of the application at least comprise:

in this embodiment, at least one first terminal device and at least one second terminal device are connected through an adapter. Responding to a first screen projection request of the first terminal device, the adapter acquires first data of the first terminal device, the first data represents interface display data of the first terminal device at the current time point, and the first data is sent to the second terminal device, so that the second terminal device displays an interface of the first terminal device at the current time point, and screen projection is achieved. If it is detected that the user operates on a first functional module of the at least one functional module, a first operation event is generated according to the operation of the user on the first functional module, a first control instruction corresponding to the first event is obtained from the first control instruction set, a first application identifier corresponding to the first functional module is searched from the application-module correspondence table, and the first application identifier and the first control instruction are sent to the first terminal device, so that the first terminal device can execute the first control instruction for the first application corresponding to the first application identifier, and the control accuracy is improved. And receiving second data of the first terminal equipment, and sending the second data to the second terminal equipment, so that the second terminal equipment displays an interface generated by the first terminal equipment executing the first control instruction according to the second data. In the embodiment of the application, the transmission of the screen projection data between the first terminal device and the second terminal device is realized through the adapter, and the accuracy and the stability of screen projection are improved. In addition, the adapter comprises a function module, and the function module can be associated with the application in the first terminal device, so that the efficiency of controlling the application in the first terminal device is improved, and the user experience is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic structural diagram of an adapter provided in an embodiment of the present application;

fig. 2 is a schematic diagram of a multi-screen collaboration system according to an embodiment of the present application;

fig. 3 is a flowchart of a method for displaying an application interface based on multi-screen collaboration according to an embodiment of the present application;

fig. 4 is a block diagram illustrating functional units of an apparatus for displaying an application interface based on multi-screen coordination according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a sub-embodiments of the present application, and not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may alternatively include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

Referring to fig. 1, fig. 1 is a schematic structural diagram of an adapter according to an embodiment of the present disclosure.

An adapter, the adapter comprising a Micro Control Unit (MCU), a first interface aggregator, a second interface aggregator, at least one function module, a first conversion module and a second conversion module, the first interface aggregator comprising at least one interface, the second interface aggregator comprising at least one interface, the first interface aggregator being connected to a first conversion module, the first conversion module being connected to the second conversion module, the second conversion module being connected to the second interface aggregator, the microcontroller being connected to each of the first conversion module, the second conversion module and the at least one function module for controlling each of the first conversion module, the second conversion module and the at least one function module, the adapter being connected to at least one first terminal device through the first interface aggregator, the first conversion module being configured to convert first data received by the first interface aggregator into second data, the second conversion module being configured to convert second data received by the first interface aggregator into a third data, the second conversion module being configured to convert the second data received by the first interface aggregator, and the third conversion module being configured to implement the second conversion module. As shown in fig. 1, wherein at least one functional module may include a functional module a, a functional module b, \8230;, a functional module n. It will be appreciated that the micro-control unit may alternatively be capable of implementing or executing the various exemplary logic blocks, units and circuits described in connection with the present disclosure, and is not limited thereto in particular.

In one possible example, the first interface aggregator is an aggregator including at least one of a display interface, a universal serial bus interface, and a network interface; the second interface aggregator is an aggregator comprising at least one of a display interface, a universal serial bus interface and a network interface.

Specifically, the first interface Hub may further include a universal serial bus Hub (USB Hub), which is a device that can extend one USB interface to multiple USB interfaces and enable the USB interfaces to be used simultaneously. The display interface, the universal serial bus interface and the network interface of the first interface aggregator may be aggregated by a serial bus aggregator. Similarly, the second interface aggregator may further include a universal serial bus hub, and the display interface, the universal serial bus interface, and the network interface of the second interface aggregator may be aggregated by the serial bus hub.

In one possible example, the adapter may also include memory, which may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of example, but not limitation, many forms of Random Access Memory (RAM) are available, such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct bus RAM (DR RAM).

In order to better understand the technical solution of the embodiment of the present application, a multi-screen coordination system that may be involved in the embodiment of the present application is introduced first.

Referring to fig. 2, fig. 2 is a schematic view of a multi-screen coordination system according to an embodiment of the present disclosure.

The multi-screen coordination system comprises an adapter, at least one first terminal device and at least one second terminal device. Each first terminal device in the at least one first terminal device is connected with the first interface collector of the adapter, each second terminal device in the at least one second terminal device is connected with the second interface collector of the adapter, the adapter is used for transmitting screen projection display data of the application of the first terminal device to the second terminal device in real time, the adapter comprises at least one function module, and the at least one function module comprises a module used for controlling the application in the first terminal device. As shown in fig. 2, in particular, the at least one first terminal device may include a first terminal device a, a first terminal device b, \8230; \ 8230;, a first terminal device m. The at least one second terminal device may include a second terminal device a, a second terminal device b, \8230;, and a second terminal device b.

Each terminal device in the at least one first terminal device may be a terminal device such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a wearable device, a head-mounted device, and a vehicle-mounted terminal, and the type of the terminal device is not limited herein. Similarly, each terminal device in the at least one second terminal device may be a terminal device such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a wearable device, a head-mounted device, or a vehicle-mounted terminal, and the type of the terminal device is not limited herein.

Referring to fig. 3, fig. 3 is a flowchart illustrating a method for displaying an application interface based on multi-screen coordination according to an embodiment of the present disclosure. Hereinafter, a method for displaying an application interface based on multi-screen coordination according to an embodiment of the present application will be described in detail with reference to the accompanying drawings. As shown in fig. 3, a method for displaying an application interface based on multi-screen coordination is applied to an adaptor of a multi-screen coordination system, where the multi-screen coordination system includes at least one first terminal device and at least one second terminal device, each first terminal device in the at least one first terminal device is connected to a first interface aggregator of the adaptor, each second terminal device in the at least one second terminal device is connected to a second interface aggregator of the adaptor, the adaptor is configured to transmit screen projection display data of the first terminal device to the second terminal device in real time, the adaptor includes at least one function module, and the at least one function module represents a module for controlling applications in the first terminal device, and the method includes the following steps:

step 301, responding to a first screen projection request of the first terminal device, and acquiring first data of the first terminal device.

The first data represent interface display data of the first terminal device at the current time point. Specifically, each first terminal device in the at least one first terminal device is connected to each second terminal device in the at least one second terminal device through the adapter. The adapter acquires first data of the first terminal equipment after receiving a first screen projection request in the first terminal equipment, and provides data support for subsequent screen projection.

Step 302, sending the first data to the second terminal device, so that the second terminal device displays an interface of the first terminal device at the current time point.

After receiving the first data, the adapter sends the first data to the second terminal device, so that the second terminal device displays an interface of the first terminal device at the current time point. The interface of the first terminal equipment is projected to the interface of the second terminal through the adapter, so that the screen projection is more stable and safer, and the screen projection efficiency is improved.

Step 303, if it is detected that the user performs an operation on a first functional module of the at least one functional module, generating a first operation event according to the operation of the user on the first functional module.

The adapter comprises at least one functional module, and the functional module can be used for controlling applications in the first terminal equipment. For example, the functional module may be a music input module, the music input module includes piano keys for playing music, music can be quickly input through the piano keys, and then when the music input module is associated with a music application, a music score can be quickly input through the music input module, so that the efficiency of controlling the application is improved, and the use experience of a user is improved. And if the operation of the user on the first functional module is detected, generating a first operation event according to the operation, thereby providing data support for subsequent control of the corresponding application.

Step 304, obtaining a first control instruction set corresponding to the first function module, and searching a first control instruction corresponding to the first operation event from the first control instruction set.

The first control instruction set comprises corresponding relations between different first operation events and different control instructions, and the control instructions represent instructions for controlling the application of the first terminal device to execute preset operations. Specifically, different control instruction sets are preset for different functional modules, so that the efficiency of searching for instructions corresponding to operation events is improved. After the first operation event is generated, a first control instruction set corresponding to the first function module is obtained, a first control instruction corresponding to the first operation event is searched from the first control instruction set, and data support is provided for subsequent control applications.

Step 305, searching a first application identifier corresponding to the first function module from a preset application and module corresponding relation table.

The application-module corresponding relation table comprises corresponding relations of different functional modules and different application identifications. And searching a first application identifier corresponding to the first functional module from the application and module corresponding relation table, and providing data support for subsequent searching of corresponding applications.

Step 306, sending the first application identifier and the first control instruction to the first terminal device.

And sending the first application identifier and the first control instruction to the first terminal device, so that the first terminal device executes the first control instruction for the first application corresponding to the first application identifier. Specifically, the adapter sends the first application identifier and the first control instruction to the first terminal device, and the first terminal device determines the first application corresponding to the first function module according to the first application identifier and executes the first control instruction to control the first application.

Step 307, receiving second data of the first terminal device.

And the second data represent interface display data generated by the first terminal equipment according to the first control instruction to control the first application. Specifically, after executing the first control instruction, the first terminal device generates interface display data, where the interface display data is the second data, and the adapter acquires the second data to provide data support for screen projection to the second terminal device.

Step 308, sending the second data to the second terminal device, so that the second terminal device displays an interface generated by the first terminal device executing the first control instruction according to the second data.

The adapter sends the second data to the second terminal device, the second terminal device analyzes the second data after receiving the second data, and an interface generated by the first terminal device executing the first control instruction is displayed through the display screen.

Therefore, in this example, the first terminal device and the second terminal device are connected through the adapter, and the screen projection data of the first terminal device are transmitted to the second terminal device, so that the stability and the safety of data transmission are improved, and the stability and the safety of screen projection are further improved. And the application in the first terminal equipment is controlled through the functional module, so that the operation efficiency is further improved, and the use experience of a user is improved.

In a possible example, the searching, from a preset application-to-module correspondence table, a first application identifier corresponding to the first functional module includes: when determining that the first functional module corresponds to a plurality of reference application identifiers from a preset application-module correspondence table, acquiring a set of times of association of an application corresponding to each reference application identifier in the plurality of reference application identifiers and the first functional module, and determining the maximum times in the set; and setting the application identifier corresponding to the maximum times as the first application identifier.

In a specific example, if the first function module is set to correspond to multiple reference application identifiers, that is, the first function module may associate multiple applications in the first terminal device, the multiple reference application identifiers corresponding to the first function module are determined from a preset application-to-module correspondence table, the number of times that the application corresponding to each reference application identifier is associated with the first function module is obtained, and a set of the number of times that the application corresponding to the reference application identifier is associated with the first function module is generated. And determining the maximum times in the set, and setting the application identifier corresponding to the maximum times as the first application identifier. For example, if the first functional module is an input keyboard, the input keyboard can be used to control inputting text information, so the input keyboard can be associated with applications such as a text editing application and a chat application requiring text input in the first terminal device. And if the input keyboard is associated with the text editing application and the chat application in the first terminal equipment. When the text editing application and the chat application are started in the first terminal device at the same time, the user starts the first function module, and the adapter needs to determine which application in the first terminal device the first function module is associated with. The specific judgment process is as follows: determining a plurality of reference application identifications corresponding to the input keyboard, and if the plurality of reference application identifications are found to comprise the application identification of the text editing application and the application identification of the chat application, acquiring a first frequency of the text editing application associated by the first functional module and a second frequency of the chat application associated by the first functional module. And if the numerical value of the first time is 3 and the numerical value of the second time is 4, setting the application identifier of the chat application as the first application identifier.

Therefore, in this example, the function module may set a plurality of corresponding reference application identifiers, and the function module is enabled to determine the associated application according to the historical association times, so that the intelligence and the convenience of use are improved while the practicability of the function module is ensured.

In a possible example, the searching, from a preset application-to-module correspondence table, a first application identifier corresponding to the first functional module includes: when determining that the first functional module corresponds to a plurality of reference application identifiers from a preset application-module correspondence table, acquiring the number of times that the application corresponding to each reference application identifier in the plurality of reference application identifiers is associated with the first functional module; sorting the reference application identifiers in the multiple reference application identifiers according to the sequence of the times from large to small to obtain a sorting table; sending the sorting list and a preset selection prompt message to the first terminal device, wherein the selection prompt message represents a message for prompting the user to select a reference application identifier from the sorting list; receiving a feedback message of the first terminal device, wherein the feedback message comprises a reference application identifier selected by the user; and setting the reference application identifier selected by the user as the first application identifier.

In a specific example, if the first function module is set to correspond to multiple reference application identifiers, determining multiple reference application identifiers corresponding to the first function module from a preset application-to-module correspondence table, obtaining the number of times that the application corresponding to each reference application identifier is associated with the first function module, and sorting the reference application identifiers in the multiple reference application identifiers according to the number of times of each reference application identifier and in a descending order to obtain a sorting table. And sending the sorting list and a preset selection prompt message to the first terminal device to prompt a user to select the associated application. It is to be understood that the sorting table may also be sent to a second terminal device, through which the user may select an application of the first terminal device associated with the first function module.

It can be seen that, in this example, when the first functional module is set to correspond to the multiple reference application identifiers, the sorting table arranged according to the number of times of each reference application identifier is sent to the first terminal device, so that the user can select the first functional module, the autonomy of selection is improved, and the use experience of the user is further improved.

In one possible example, if an operation of the user on the first functional module is detected, an application identifier corresponding to the first functional module is obtained from the application-module correspondence table. The method comprises the steps of sending a preset starting instruction to a first terminal device according to an application identifier when detecting that the application corresponding to the application identifier is not started in the first terminal device, so as to start the application corresponding to the application identifier, generating an operation event according to the operation of a user on a first function module after the application is started, obtaining a control instruction corresponding to the operation event, and sending the control instruction and the application identifier to the first terminal device, so that the first terminal device controls the application corresponding to the application instruction, and interface display data are generated. The efficiency of starting the application is improved, and then the user experience is improved.

In one possible example, after the sending the second data to the second terminal device, the method further includes: if the user is detected to perform a second operation on a first operation module of the second terminal device, generating a second operation event according to the second operation; if an operation event which is the same as the second operation event is found from a preset association relation table, acquiring an association instruction corresponding to the same operation event, wherein the association relation table comprises at least one operation event corresponding to the association instruction, and the association instruction represents an association instruction for associating an operation module of the second terminal device with the first terminal device; associating the operation module of the second terminal equipment with the first terminal equipment according to the association instruction; acquiring a third operation performed by the user on a second operation module of the second terminal device, and generating a third operation event according to the third operation; searching a second application identifier corresponding to the second operation module from the application and module corresponding relation table; acquiring a second control instruction set corresponding to the second operation module, and searching a second control instruction corresponding to the third operation event from the second control instruction set, wherein the second control instruction set comprises corresponding relations between different third operation events and different control instructions; sending the second application identifier and the second control instruction to the first terminal device, so that the first terminal device is just executing the second control instruction on a second application corresponding to the second application identifier; acquiring third data of the first terminal device, wherein the third data represent interface display data generated by the first terminal device according to the second control instruction to control the second application; and sending the third data to the second terminal device, so that the second terminal device displays an interface generated by the first terminal device executing the second control instruction according to the third data.

In a specific example, the second terminal device may include a first operation module, and if the user performs a second operation on the first operation module of the second terminal device, a second operation event is generated according to the second operation. In order to ensure the accuracy of the control, the terminal device corresponding to the second operation event needs to be judged. Firstly, whether an operation event identical to the second operation event exists is searched from a preset association relation table, if yes, an association instruction corresponding to the identical operation event is obtained, and the association instruction represents an association instruction for associating the operation module of the second terminal device with the first terminal device. For example, if the operation module of the second terminal device is a touch screen module, and the operation event corresponding to the association instruction included in the association relation table is a circle drawn on the touch screen module. And detecting a second operation of the user for drawing a circle on the first operation module, generating a second operation event according to the second operation of the user, and acquiring the associated instruction corresponding to the same operation event if the second operation event is the same as the operation event corresponding to the associated instruction. And associating the operation module of the second terminal equipment with the first terminal equipment according to the association instruction. And if the third operation of the user on the second operation module is detected, generating a third operation event according to the third operation. And searching a second application identifier corresponding to the second operation module from the application and module corresponding relation table, acquiring a second control instruction set corresponding to the second operation module, and searching a second control instruction corresponding to the third operation event from the second control instruction set. And sending the second application identifier and the second control instruction to the first terminal device, so that the first terminal device executes the second control instruction for the application corresponding to the second application identifier, thereby acquiring third data of the first terminal device. And sending the third data to the second terminal equipment so that the second terminal equipment displays an interface generated by the first terminal equipment executing the second control instruction according to the third data, and screen projection is realized. It is to be understood that the first operating module and the second operating module may be the same operating module in the second terminal device, or may be different operating modules, and are not limited herein.

As can be seen, in this example, when it is detected that the user performs an operation on the operation module of the second terminal device, the operation event corresponding to the operation is detected to control the terminal device, and the association relation does not need to be manually indicated by the user, so that the efficiency of associating the operation module with the terminal device is improved.

In one possible example, after the sending the second data to the second terminal device, the method further includes: if it is detected that the user performs a fourth operation on a third operation module of the second terminal device, acquiring a third application identifier of an application started by the second terminal device at the current time point; searching a fourth application identifier corresponding to the third operation module from a preset application set, wherein the application set comprises corresponding relations between different operation modules of the second terminal device and different application identifiers of the second terminal device; if the third application identifier is different from the fourth application identifier, associating a third operation module of the second terminal device with the first terminal device; generating a fourth operation event according to the fourth operation; searching a fifth application identifier corresponding to the third operation module from the application and module corresponding relation table; acquiring a third control instruction set corresponding to the third operation module, and searching for a corresponding third control instruction from the third control instruction set according to the fourth operation event, wherein the third control instruction set comprises corresponding relations between different fourth operation events and different control instructions; sending the fifth application identifier and the third control instruction to the first terminal device, so that the first terminal device controls a third application corresponding to the third application identifier according to the third control instruction; acquiring fourth data of the first terminal device, wherein the fourth data represents interface display data generated by the first terminal device according to the third control instruction to control the third application; and sending the fourth data to the second terminal device, so that the second terminal device displays an interface generated by the first terminal device executing the third control instruction according to the fourth data.

In a specific example, the second terminal device may include a third operation module, and if the user performs a fourth operation on the third operation module, a fourth operation event is generated according to the fourth operation. In order to ensure the accuracy of the control, the terminal device corresponding to the fourth operation event needs to be judged. The method comprises the steps of firstly obtaining a third application identifier of an application started by second terminal equipment in a current time point, then obtaining a fourth application identifier corresponding to a third operation module from an application set, judging whether the third application identifier is the same as the fourth application identifier, and if the third application identifier is the same as the fourth application identifier, associating the third operation module with the application corresponding to the fourth application identifier in the second terminal equipment. And if not, associating the third operation module of the second terminal device with the first terminal device, searching a fifth application identifier corresponding to the third operation module from the application-module correspondence table, acquiring a third control instruction set corresponding to the third operation module, searching a corresponding third control instruction from the third control instruction set according to a fourth operation event generated by a fourth operation, and sending the fifth application identifier and the third control instruction to the first terminal device, so that the first terminal device controls a fifth application corresponding to the fifth application identifier according to the third control instruction, and thus fourth data of the first terminal device is acquired. And after the adapter acquires the fourth data, sending the fourth data to the second terminal device, so that the second terminal device displays an interface generated by the first terminal device executing the third control instruction according to the fourth data.

As can be seen, in this example, when it is detected that the user performs an operation on the operation module of the second terminal device, the operation event corresponding to the operation is detected to control the terminal device, and the association relationship does not need to be manually indicated by the user, thereby improving the efficiency of associating the operation module with the terminal device.

In one possible example, after the sending the first data to the second terminal device, the method further includes: if the fact that the at least one functional module comprises a second functional module of the display screen is detected, a fourth application identifier corresponding to the second functional module is searched from the application and module corresponding relation table; sending the sixth application identifier and a preset acquisition request to the first terminal device, where the acquisition request represents that the request acquires data of a fourth application corresponding to the sixth application identifier; receiving fifth data of the first terminal device, wherein the fifth data is data of the fourth application; and displaying the content corresponding to the fifth data on the second functional module.

In a specific example, if the second function module of the adaptor includes a display screen, in order to improve the display efficiency, the second function module may directly obtain the corresponding display data from the first terminal device. Specifically, if it is detected that the second function module includes the display screen, the adaptor searches a sixth application identifier corresponding to the second function module from the application-module correspondence table, and sends the sixth application identifier and a preset acquisition request to the first terminal device, so as to obtain data of a fourth application corresponding to the sixth application identifier. And after the adapter acquires the data of the fourth application, analyzing the data and displaying the data through a display screen of the second functional module.

It can be seen that, in this example, if the functional module of the adaptor includes the display screen, the data that needs to be displayed and corresponds to the application of the functional module can be directly obtained, so as to improve the display efficiency and further improve the user experience.

In one possible example, the at least one functional module may include a touch screen module for cutting a video, and may correspond to a video clip application, that is, may correspond to a video progress bar in the video clip application, and the progress bar is scaled up to a touch screen of the touch screen module, and the progress of the corresponding video may be controlled by controlling the touch screen. The at least one functional module may further include a sliding functional module for sliding a document page in the document application, a volume adjusting functional module for adjusting volume, and the like, so that the type of the functional module in the at least one functional module may be set according to actual requirements, and the specific details are not limited herein.

The embodiment of the application provides a device for displaying an application interface based on multi-screen cooperation, and the device can be an electronic device. Specifically, the device for displaying an application interface based on multi-screen coordination provided by the embodiment of the present application may include modules corresponding to the corresponding steps.

In the embodiment of the present application, according to the above method, the division of the function modules may be performed on the device for displaying an application interface based on multi-screen coordination, for example, each function module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The division of the modules in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

Referring to fig. 4, in a case that each function module is divided according to each function, fig. 4 is a block diagram of functional units of an apparatus for displaying an application interface based on multi-screen coordination according to an embodiment of the present application, where the apparatus includes:

an adaptor including at least one function module for controlling terminal devices connected to a first interface aggregator and a second interface aggregator of the adaptor, the adaptor being configured to project an interface of at least one first terminal device connected to the first interface aggregator on a screen of at least one second terminal device connected to the second interface aggregator in real time, the apparatus comprising:

a first obtaining unit 401, configured to obtain first data of a first terminal device in response to a first screen projection request of the first terminal device, where the first data represents interface display data of the first terminal device at a current time point, the first terminal device is a device of the at least one first terminal device, and the second terminal device is a device of the at least one second terminal device;

a first sending unit 402, configured to send the first data to the second terminal device, so that the second terminal device displays an interface of the first terminal device at a current time point;

a processing unit 403, configured to generate a first operation event according to an operation of a user on a first functional module of the at least one functional module, if it is detected that the user performs the operation on the first functional module;

a second obtaining unit 404, configured to obtain a first control instruction set corresponding to the first function module, and search for a first control instruction corresponding to the first operation event from the first control instruction set, where the first control instruction set includes a correspondence between different first operation events and different control instructions, and the control instruction represents an instruction for controlling an application of the first terminal device to execute a preset operation;

a searching unit 405, configured to search a preset application-module correspondence table for a first application identifier corresponding to the first function module, where the application-module correspondence table includes correspondence between different function modules and different application identifiers;

an executing unit 406, configured to send the first application identifier and the first control instruction to the first terminal device, so that the first terminal device executes the first control instruction;

a receiving unit 407, configured to receive second data of the first terminal device, where the second data represents interface display data generated by the first terminal device according to the first control instruction to control the first application;

a second sending unit 408, configured to send the second data to the second terminal device, so that the second terminal device displays, according to the second data, an interface generated by the first terminal device executing the first control instruction.

In one possible example, the lookup unit 405 is further configured to: when a plurality of reference application identifications corresponding to the first function module are determined from a preset application and module corresponding relation table, acquiring a set of times of association of the application corresponding to each reference application identification in the plurality of reference application identifications and the first function module, and determining the maximum times in the set; and setting the application identifier corresponding to the maximum times as the first application identifier.

In one possible example, the lookup unit 405 is further configured to: when determining that the first functional module corresponds to a plurality of reference application identifiers from a preset application-module correspondence table, acquiring the number of times that the application corresponding to each reference application identifier in the plurality of reference application identifiers is associated with the first functional module; sorting the reference application identifiers in the multiple reference application identifiers according to the sequence of the times from large to small to obtain a sorting table; sending the sorting list and a preset selection prompt message to the first terminal device, wherein the selection prompt message represents a message for prompting the user to select a reference application identifier from the sorting list; receiving a feedback message of the first terminal device, wherein the feedback message comprises the reference application identifier selected by the user; and setting the reference application identifier selected by the user as the first application identifier.

In a possible example, the apparatus further includes a generating unit, configured to generate a second operation event according to a second operation if it is detected that the user performs the second operation on a first operation module of the second terminal device; if the operation event identical to the second operation event is found from a preset association relation table, acquiring an association instruction corresponding to the identical operation event, wherein the association relation table comprises at least one operation event corresponding to the association instruction, and the association instruction represents an association instruction for associating the operation module of the second terminal device with the first terminal device; associating the operation module of the second terminal device with the first terminal device according to the association instruction; acquiring a third operation performed by the user on a second operation module of the second terminal device, and generating a third operation event according to the third operation; searching a second application identifier corresponding to the second operation module from the application and module corresponding relation table; acquiring a second control instruction set corresponding to the second operation module, and searching a second control instruction corresponding to the third operation event from the second control instruction set, wherein the second control instruction set comprises corresponding relations between different third operation events and different control instructions; sending the second application identifier and the second control instruction to the first terminal device, so that the first terminal device is just executing the second control instruction on a second application corresponding to the second application identifier; third data of the first terminal device are obtained, and the third data represent interface display data generated by the first terminal device according to the second control instruction for controlling the second application; and sending the third data to the second terminal device, so that the second terminal device displays an interface generated by the first terminal device executing the second control instruction according to the third data.

In a possible example, the apparatus further includes a third obtaining unit, configured to, if it is detected that the user performs a fourth operation on a third operation module of the second terminal device, obtain a third application identifier of an application that is enabled by the second terminal device at a current time point; searching a fourth application identifier corresponding to the third operation module from a preset application set, wherein the application set comprises corresponding relations between different operation modules of the second terminal device and different application identifiers of the second terminal device; if the third application identifier is different from the fourth application identifier, associating a third operation module of the second terminal device with the first terminal device; and generating a fourth operation event according to the fourth operation; searching a fifth application identifier corresponding to the third operation module from the application and module corresponding relation table; acquiring a third control instruction set corresponding to the third operation module, and searching for a corresponding third control instruction from the third control instruction set according to the fourth operation event, wherein the third control instruction set comprises corresponding relations between different fourth operation events and different control instructions; sending the fifth application identifier and the third control instruction to the first terminal device, so that the first terminal device controls a third application corresponding to the fifth application identifier according to the third control instruction; fourth data of the first terminal device are obtained, and the fourth data represent interface display data generated by the first terminal device controlling the third application according to the third control instruction; and sending the fourth data to the second terminal device, so that the second terminal device displays an interface generated by the first terminal device executing the third control instruction according to the fourth data.

The above-described embodiments may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. The procedures or functions according to the embodiments of the present application are wholly or partially generated when the computer instructions or the computer program are loaded or executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire or wirelessly. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, data center, etc., that contains one or more collections of available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium may be a solid state disk.

Embodiments of the present application also provide a computer storage medium, where the computer storage medium stores a computer program for electronic data exchange, the computer program enabling a computer to execute part or all of the steps of any one of the methods described in the above method embodiments, and the computer includes an electronic device.

Embodiments of the present application also provide a computer program product, which includes a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

The computer program product may be a software installation package, the computer comprising an electronic device.

It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

In the several embodiments provided in the present application, it should be understood that the disclosed method, apparatus and system may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative; for example, the division of the unit is only a logic function division, and there may be another division manner in actual implementation; for example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be physically included alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware, or in the form of hardware plus a software functional unit.

The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute some steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Although the present invention is disclosed above, the present invention is not limited thereto. Various changes and modifications can be easily made by those skilled in the art without departing from the spirit and scope of the present invention, and it is within the scope of the present invention to include different functions, combination of implementation steps, software and hardware implementations.

Claims

1. A method for displaying an application interface based on multi-screen collaboration is applied to an adapter of a multi-screen collaboration system, where the multi-screen collaboration system includes at least one first terminal device and at least one second terminal device, each first terminal device in the at least one first terminal device is connected to a first interface aggregator of the adapter, each second terminal device in the at least one second terminal device is connected to a second interface aggregator of the adapter, the adapter is configured to transmit screen projection display data of the first terminal device to the second terminal device in real time, the adapter includes at least one function module, and the at least one function module characterizes a module used for controlling applications in the first terminal device, and the method includes:

2. The method according to claim 1, wherein the searching for the first application identifier corresponding to the first functional module from a preset application-module correspondence table includes:

when determining that the first functional module corresponds to a plurality of reference application identifiers from a preset application-module correspondence table, acquiring a set of times of association of an application corresponding to each reference application identifier in the plurality of reference application identifiers and the first functional module, and determining the maximum times in the set;

and setting the application identifier corresponding to the maximum times as the first application identifier.

3. The method according to claim 1, wherein the searching for the first application identifier corresponding to the first functional module from a preset application-module correspondence table includes:

when determining that the first functional module corresponds to a plurality of reference application identifiers from a preset application-module correspondence table, acquiring the number of times that the application corresponding to each reference application identifier in the plurality of reference application identifiers is associated with the first functional module;

sorting the reference application identifiers in the multiple reference application identifiers according to the sequence of the times from large to small to obtain a sorting table;

sending the sorting list and a preset selection prompt message to the first terminal device, wherein the selection prompt message represents a message for prompting the user to select a reference application identifier from the sorting list;

receiving a feedback message of the first terminal device, wherein the feedback message comprises a reference application identifier selected by the user;

setting the reference application identifier selected by the user as the first application identifier.

4. The method according to any of claims 1-3, wherein after sending the second data to the second terminal device, the method further comprises:

if the user is detected to perform a second operation on a first operation module of the second terminal device, generating a second operation event according to the second operation;

if an operation event which is the same as the second operation event is found from a preset association relation table, acquiring an association instruction corresponding to the same operation event, wherein the association relation table comprises at least one operation event corresponding to the association instruction, and the association instruction represents an association instruction for associating an operation module of the second terminal device with the first terminal device;

associating the operation module of the second terminal equipment with the first terminal equipment according to the association instruction;

acquiring a third operation performed by the user on a second operation module of the second terminal device, and generating a third operation event according to the third operation;

searching a second application identifier corresponding to the second operation module from the application and module corresponding relation table;

acquiring a second control instruction set corresponding to the second operation module, and searching a second control instruction corresponding to the third operation event from the second control instruction set, wherein the second control instruction set comprises corresponding relations between different third operation events and different control instructions;

sending the second application identifier and the second control instruction to the first terminal device, so that the first terminal device is just executing the second control instruction on a second application corresponding to the second application identifier;

acquiring third data of the first terminal device, wherein the third data represent interface display data generated by the first terminal device according to the second control instruction to control the second application;

and sending the third data to the second terminal device, so that the second terminal device displays an interface generated by the first terminal device executing the second control instruction according to the third data.

5. The method according to any of claims 1-3, wherein after said sending the second data to the second terminal device, the method further comprises:

if it is detected that the user performs a fourth operation on a third operation module of the second terminal device, acquiring a third application identifier of an application started by the second terminal device at the current time point;

searching a fourth application identifier corresponding to the third operation module from a preset application set, wherein the application set comprises corresponding relations between different operation modules of the second terminal device and different application identifiers of the second terminal device;

if the third application identifier is different from the fourth application identifier, associating a third operation module of the second terminal device with the first terminal device;

generating a fourth operation event according to the fourth operation;

searching a fifth application identifier corresponding to the third operation module from the application and module corresponding relation table;

acquiring a third control instruction set corresponding to the third operation module, and searching for a corresponding third control instruction from the third control instruction set according to the fourth operation event, wherein the third control instruction set comprises corresponding relations between different fourth operation events and different control instructions;

sending the fifth application identifier and the third control instruction to the first terminal device, so that the first terminal device controls a third application corresponding to the fifth application identifier according to the third control instruction;

acquiring fourth data of the first terminal device, wherein the fourth data represents interface display data generated by the first terminal device controlling the third application according to the third control instruction;

and sending the fourth data to the second terminal device, so that the second terminal device displays an interface generated by the first terminal device executing the third control instruction according to the fourth data.

6. The method of claim 1, wherein after the transmitting the first data to the second terminal device, the method further comprises:

if the at least one functional module is detected to comprise a second functional module of the display screen, searching a sixth application identifier corresponding to the second functional module from the application and module corresponding relation table;

sending the sixth application identifier and a preset acquisition request to the first terminal device, where the acquisition request represents that the request acquires data of a fourth application corresponding to the sixth application identifier;

receiving fifth data of the first terminal device, wherein the fifth data is data of the fourth application;

and displaying the content corresponding to the fifth data on the second functional module.

7. An apparatus for displaying an application interface based on multi-screen coordination, the apparatus comprising an adaptor, the adaptor comprising at least one function module, the function module being configured to control terminal devices connected to a first interface aggregator and a second interface aggregator of the adaptor, the adaptor being configured to project an interface of at least one first terminal device connected to the first interface aggregator onto a screen of at least one second terminal device connected to the second interface aggregator in real time, the apparatus comprising:

the processing unit is used for generating a first operation event according to the operation of the user on a first functional module in the at least one functional module if the user is detected to operate on the first functional module;

8. An adaptor, said adaptor comprising a micro control unit, a first interface aggregator, a second interface aggregator, at least one function module, a first conversion module and a second conversion module, said first interface aggregator comprising at least one interface, said second interface aggregator comprising at least one interface, said first interface aggregator being connected to a first conversion module, said first conversion module being connected to said second conversion module, said second conversion module being connected to said second interface aggregator, said micro control unit being connected to each of said first conversion module, said second conversion module and said at least one function module for controlling each of said first conversion module, said second conversion module and said at least one function module, said adaptor being connected to at least one first terminal device through said first interface aggregator, said first conversion module being adapted to convert first data received by said first interface aggregator into second data, said second conversion module being adapted to convert said second conversion module into said second data, said second conversion module being adapted to perform said third conversion module, said third conversion module being adapted to perform the method as described in any claim 6.

9. The adapter according to claim 8, wherein the first interface aggregator is an aggregator including at least one of a display interface, a universal serial bus interface and a network interface;

the second interface aggregator is an aggregator comprising at least one of a display interface, a universal serial bus interface and a network interface.

10. A computer-readable storage medium, characterized in that a computer program for electronic data exchange is stored, wherein the computer program causes a computer to perform the method according to any one of claims 1-6.