CN117149122A

CN117149122A - Method, device, system, equipment and medium for on-screen interaction

Info

Publication number: CN117149122A
Application number: CN202311227518.4A
Authority: CN
Inventors: 马晓兵
Original assignee: Jingdong Technology Information Technology Co Ltd
Current assignee: Jingdong Technology Information Technology Co Ltd
Priority date: 2023-09-21
Filing date: 2023-09-21
Publication date: 2023-12-01

Abstract

The present disclosure provides a method, a device, a system, a device and a medium for on-screen interaction, and relates to the technical field of computers, where the method includes: and monitoring an operation event of a user aiming at a current operation page, generating an operation record, and sending the operation record to a server, so that the server generates a first synchronous operation instruction for controlling a second synchronous screen terminal according to the operation record of the first synchronous screen terminal, wherein the second synchronous screen terminal is a terminal for establishing synchronous screen interactive connection with the first synchronous screen terminal. The terminal monitors the operation event, pushes the operation record to the server, and sends the operation instruction to the terminal participating in the interaction with the screen in real time through the server without pushing video data acquired in a screen recording mode, so that the consumption of network bandwidth can be saved, and the network delay is reduced.

Description

Method, device, system, equipment and medium for on-screen interaction

Technical Field

The disclosure relates to the field of computer technology, and in particular relates to a method, a device, a system, equipment and a medium for on-screen interaction.

Background

With the development of internet technology, the on-screen interaction technology has been widely applied in life and work, but in related technology, when multimedia equipment shares and interacts with a screen, the multimedia equipment establishes a connection with a server, interactive data is pushed mainly in a mode of recording a screen video stream, and the server searches equipment interacting with the multimedia equipment and then pushes shared content.

The video data generated in the screen recording mode is relatively large, the network requirement is relatively high when the data push sharing is carried out, the consumption of network bandwidth is relatively large, and relatively serious network delay exists.

It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

The present disclosure aims to provide a method, an apparatus, a system, a device and a medium for on-screen interaction, which at least reduce the consumption of network bandwidth to a certain extent and shorten the time of network delay.

Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure.

According to a first aspect of the present disclosure, there is provided a method for on-screen interaction, applied to a first on-screen terminal, including:

monitoring an operation event of a user aiming at a current operation page, and generating an operation record;

the operation record is sent to a server, so that the server generates a first synchronous operation instruction for controlling a second synchronous screen terminal according to the operation record of the first synchronous screen terminal; the second same-screen terminal is a terminal which establishes same-screen interactive connection with the first same-screen terminal.

In one embodiment of the present disclosure, before the monitoring the operation event of the user for the current operation page, the method further includes:

sending a screen-sharing interaction request to the server so that the server generates a screen-sharing interaction link according to the screen-sharing interaction request;

receiving the same-screen interactive link sent by the server;

and establishing the same-screen interactive connection with the second same-screen terminal through the same-screen interactive link.

In an embodiment of the present disclosure, the establishing a on-screen interactive connection with the second on-screen terminal through the on-screen interactive link includes:

responding to the operation of opening the same-screen interactive link by a user, and displaying a message notification sent by the server and waiting for the access of a second same-screen terminal;

receiving a same-screen access notification of the second same-screen terminal sent by the server; the on-screen access notification of the second on-screen terminal is sent to the first on-screen terminal by the server after the second on-screen terminal opens the on-screen interactive link sent by the first on-screen terminal or the server;

and according to the on-screen access notification of the second on-screen terminal, determining to establish on-screen interactive connection with the second on-screen terminal.

In one embodiment of the present disclosure, the method further comprises:

the receiving server sends a second synchronous operation instruction; the second synchronous operation instruction is generated by the server according to the operation record of the second same-screen terminal and is used for controlling the operation instruction of the first same-screen terminal;

and executing the second synchronous operation instruction.

In one embodiment of the present disclosure, the operation event includes a focus lost event; the event of losing focus is an event of losing focus of the current operation page;

after the operation record is sent to the server, the method further comprises:

and receiving waiting indication information sent by the server, displaying an interface waiting for re-acquiring a focus until receiving a second synchronous operation instruction for enabling the current operation page to re-acquire the focus, and displaying the current operation page.

According to a second aspect of the present disclosure, there is provided a method for on-screen interaction, applied to a server, including:

receiving an operation record sent by a first co-screen terminal; the operation record is generated for the first screen-sharing terminal to monitor an operation event of a current operation page in the first screen-sharing terminal;

Generating a first synchronous operation instruction for controlling the second synchronous screen terminal according to the operation record of the first synchronous screen terminal; the second same-screen terminal is a terminal which establishes same-screen interactive connection with the first same-screen terminal.

In one embodiment of the present disclosure, before the receiving the operation record sent by the first on-screen terminal, the method further includes:

receiving a screen-sharing interaction request sent by a first screen-sharing terminal;

generating a same-screen interactive link according to the same-screen interactive request;

and sending the same-screen interactive link to the first same-screen terminal or the first same-screen terminal and the second same-screen terminal so that the first same-screen terminal and the second same-screen terminal establish same-screen interactive connection through the same-screen interactive link.

In one embodiment of the present disclosure, the on-screen interactive link includes a unique identifier; the method further comprises the steps of:

receiving unique identifiers sent after the first same-screen terminal and the second same-screen terminal open the same-screen interactive link;

establishing a corresponding relationship of the same-screen interaction between the first same-screen terminal and the second same-screen terminal through the unique identifier;

and sending a synchronous operation instruction between the first same-screen terminal and the second same-screen terminal through the corresponding relationship of the same-screen interaction.

In one embodiment of the present disclosure, after the generating the first synchronization operation instruction for controlling the second on-screen terminal, the method further includes:

and determining the second same-screen terminal which is in same-screen interactive connection with the first same-screen terminal according to the same-screen interactive corresponding relation, and sending the first synchronous operation instruction to the second same-screen terminal.

In one embodiment of the present disclosure, the operational event includes a loss of focus event; the event of losing focus is an event that a current operation page in the first same screen terminal loses focus;

wherein after the generating the first synchronization operation instruction for controlling the second synchronization screen terminal, the method further comprises:

receiving an operation record comprising a focus losing event sent by the first screen sharing terminal, and sending waiting indication information to the first screen sharing terminal so that the first screen sharing terminal displays an interface waiting for re-acquiring a focus;

sending a first synchronous operation instruction of which the operation event is a focus losing event to the second synchronous screen terminal;

receiving an operation record comprising a focus event sent by a second same-screen terminal, and generating a second synchronous operation instruction for controlling a current operation page of the first same-screen terminal to reacquire a focus;

And sending the second synchronous operation instruction to the first synchronous screen terminal, so that a current operation page in the first synchronous screen terminal acquires a focus again, and the current operation page is displayed.

In one embodiment of the present disclosure, the method further comprises:

if the operation record of the first same screen terminal comprises a plurality of continuous identical operation events in a preset time period, merging the plurality of continuous identical operation events to obtain a merged operation event.

In one embodiment of the present disclosure, the plurality of consecutive identical operation events is a plurality of consecutive page-sliding operation events;

generating a first synchronous operation instruction for controlling the second synchronous screen terminal according to the operation record of the first synchronous screen terminal, including:

combining the plurality of continuous page sliding operation events to obtain an operation record corresponding to the combined sliding operation event;

and generating a first synchronous operation instruction corresponding to the combined sliding operation event according to the operation record corresponding to the combined sliding operation event.

In one embodiment of the present disclosure, the method further comprises:

if the operation record of the first same screen terminal comprises a clicking operation event which has no influence on the current operation page in a preset time period, deleting the operation record corresponding to the clicking operation event.

According to a third aspect of the present disclosure, there is provided a on-screen interaction device, applied to a first on-screen terminal, including:

the monitoring unit is used for monitoring an operation event of a user aiming at a current operation page and generating an operation record;

the first sending unit is used for sending the operation record to the server so that the server generates a first synchronous operation instruction for controlling the second synchronous screen terminal according to the operation record of the first synchronous screen terminal; the second same-screen terminal is a terminal which establishes same-screen interactive connection with the first same-screen terminal.

In one embodiment of the present disclosure, the on-screen interaction device further includes:

the interaction unit is used for sending a same-screen interaction request to the server so that the server generates a same-screen interaction link according to the same-screen interaction request;

receiving the same-screen interactive link sent by the server;

In an embodiment of the present disclosure, the interaction unit is further configured to:

the execution unit is used for receiving a second synchronous operation instruction sent by the server; the second synchronous operation instruction is generated by the server according to the operation record of the second same-screen terminal and is used for controlling the operation instruction of the first same-screen terminal;

and executing the second synchronous operation instruction.

the execution unit is further configured to receive the waiting indication information sent by the server, display an interface waiting for re-acquiring the focus until receiving a second synchronous operation instruction for enabling the current operation page to re-acquire the focus, enable the current operation page to re-acquire the focus, and display the current operation page.

According to a fourth aspect of the present disclosure, there is provided a on-screen interaction device, applied to a server, including:

the receiving unit is used for receiving the operation record sent by the first same screen terminal; the operation record is generated for the first screen-sharing terminal to monitor an operation event of a current operation page in the first screen-sharing terminal;

the second sending unit is used for generating a first synchronous operation instruction for controlling the second synchronous screen terminal according to the operation record of the first synchronous screen terminal; the second same-screen terminal is a terminal which establishes same-screen interactive connection with the first same-screen terminal.

the link issuing unit is used for receiving the same-screen interaction request sent by the first same-screen terminal;

The corresponding relation establishing unit is used for receiving unique identifiers sent after the first same-screen terminal and the second same-screen terminal open the same-screen interactive link;

In one embodiment of the disclosure, the second transmitting unit is further configured to:

the second sending unit is further configured to receive an operation record including a focus losing event sent by the first co-screen terminal, and send waiting indication information to the first co-screen terminal, so that the first co-screen terminal displays an interface waiting for re-acquiring a focus;

and the data processing unit is used for merging the plurality of continuous identical operation events to obtain a merged operation event if the operation record of the first same screen terminal comprises the plurality of continuous identical operation events in a preset time period.

In one embodiment of the present disclosure, the plurality of consecutive identical operation events is a plurality of consecutive page-sliding operation events; the data processing unit is further configured to:

In an embodiment of the present disclosure, the data processing unit is further configured to:

According to a fifth aspect of the present disclosure, there is provided a on-screen interactive system, comprising: the system comprises a first co-screen terminal, a second co-screen terminal and a server;

the first co-screen terminal is used for monitoring an operation event of a user aiming at a current operation page and generating an operation record; the operation record is sent to a server, so that the server generates a first synchronous operation instruction for controlling the second synchronous screen terminal according to the operation record of the first synchronous screen terminal; the second same-screen terminal is a terminal which establishes same-screen interactive connection with the first same-screen terminal;

the server is used for receiving the operation record sent by the first same screen terminal; the operation record is generated for the first screen-sharing terminal to monitor an operation event of a current operation page in the first screen-sharing terminal; generating a first synchronous operation instruction for controlling the second synchronous screen terminal according to the operation record of the first synchronous screen terminal;

The second synchronous screen terminal is used for receiving a first synchronous operation instruction sent by the server.

According to a sixth aspect of the present disclosure, there is provided an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the on-screen interaction method described above via execution of the executable instructions.

According to a seventh aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the on-screen interaction method described above.

The embodiment of the disclosure provides a method, a device, a system, equipment and a medium for on-screen interaction, wherein the method comprises the following steps: the first screen sharing terminal generates an operation record by monitoring an operation event of a user aiming at a current operation page, and sends the operation record to the server, so that the server generates a first synchronous operation instruction for controlling the second screen sharing terminal according to the operation record of the first screen sharing terminal, and sends the first synchronous operation instruction to the second screen sharing terminal in real time, and the second screen sharing terminal is a terminal which establishes screen sharing interaction connection with the first screen sharing terminal, so that video data acquired in a screen recording mode does not need to be pushed, consumption of network bandwidth can be saved, and network delay is reduced.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort.

FIG. 1 illustrates a schematic diagram of a computer system in an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a system for on-screen interaction according to an embodiment of the disclosure;

FIG. 3 is a schematic flow chart of establishing a connection for on-screen interaction in an embodiment of the disclosure;

FIG. 4 is a schematic flow chart of a method of on-screen interaction according to an embodiment of the disclosure;

FIG. 5 is a second flow chart of a method for on-screen interaction according to an embodiment of the disclosure;

FIG. 6 is a third diagram illustrating an interaction scheme of a method of on-screen interaction according to an embodiment of the disclosure;

FIG. 7 is an interaction diagram of a method for on-screen interaction in an embodiment of the disclosure;

FIG. 8 is a schematic structural diagram of a one-screen interaction device according to an embodiment of the disclosure;

FIG. 9 is a schematic diagram of another embodiment of the present disclosure;

fig. 10 shows a schematic structural diagram of an electronic device in an embodiment of the disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.

In the related art, when the multimedia device shares and interacts with the screen, the multimedia device establishes a connection with the server, and the server pushes the interaction data mainly in a mode of recording the screen video stream, and then pushes the shared content by searching the device which interacts with the multimedia device. The video data generated in the screen recording mode is relatively large, the network requirement is relatively high when the data push sharing is carried out, the consumption of network bandwidth is relatively large, and relatively serious network delay exists.

Based on the above technical problems, an embodiment of the present disclosure provides a method, an apparatus, a system, a device, and a medium for on-screen interaction, where the method includes: the first screen sharing terminal generates an operation record by monitoring an operation event of a user aiming at a current operation page, and sends the operation record to the server, so that the server generates a first synchronous operation instruction for controlling the second screen sharing terminal according to the operation record of the first screen sharing terminal, and sends the first synchronous operation instruction to the second screen sharing terminal in real time, and the second screen sharing terminal is a terminal which establishes screen sharing interaction connection with the first screen sharing terminal, so that video data acquired in a screen recording mode does not need to be pushed, consumption of network bandwidth can be saved, and network delay is reduced.

Fig. 1 is a schematic diagram of a computer system according to an exemplary embodiment of the present application. The system comprises: a number of terminals 120 and a server 140.

The terminal 120 may be a mobile terminal such as a mobile phone, a game console, a tablet computer, an electronic book reader, a smart glasses, an MP4 (MovingPicture Experts Group Audio Layer IV, dynamic image expert compression standard audio layer 4) player, a smart home device, an AR (Augmented Reality ) device, a VR (Virtual Reality) device, or the terminal 120 may be a personal computer (Personal Computer, PC) such as a laptop portable computer and a desktop computer, etc.

The terminal 120 may have an application program installed therein for providing the on-screen interaction method.

The terminal 120 is connected to the server 140 through a communication network. Optionally, the communication network is a wired network or a wireless network.

Server 140 is a server, or is composed of several servers, or is a virtualization platform, or is a cloud computing service center. Optionally, the server 140 takes on primary computing work and the terminal 120 takes on secondary computing work; alternatively, the server 140 takes on secondary computing work and the terminal 120 takes on primary computing work; alternatively, a distributed computing architecture is employed between the terminal 120 and the server 140 for collaborative computing.

Alternatively, the clients of the applications installed in different terminals 120 are the same, or the clients of the applications installed on both terminals 120 are clients of the same type of application of different control system platforms. The specific form of the client of the application program may also be different based on the difference of the terminal platforms, for example, the application program client may be a mobile phone client, a PC client, or a World Wide Web (Web) client.

Those skilled in the art will appreciate that the number of terminals 120 may be greater or lesser. Such as the above-mentioned terminals may be only one, or the above-mentioned terminals may be several tens or hundreds, or more. The embodiment of the application does not limit the number of terminals and the equipment type.

Optionally, the system may further include a management device (not shown in fig. 1) connected to the server 140 through a communication network. Optionally, the communication network is a wired network or a wireless network.

Alternatively, the wireless network or wired network described above uses standard communication techniques and/or protocols. The network is typically the Internet, but may be any network including, but not limited to, a local area network (Local Area Network, LAN), metropolitan area network (Metropolitan Area Network, MAN), wide area network (Wide Area Network, WAN), mobile, wired or wireless network, private network, or any combination of virtual private networks. In some embodiments, data exchanged over a network is represented using techniques and/or formats including HyperText Mark-up Language (HTML), extensible markup Language (Extensible MarkupLanguage, XML), and the like. All or some of the links may also be encrypted using conventional encryption techniques such as secure sockets layer (Secure Socket Layer, SSL), transport layer security (Transport Layer Security, TLS), virtual private network (Virtual Private Network, VPN), internet protocol security (Internet ProtocolSecurity, IPsec), etc. In other embodiments, custom and/or dedicated data communication techniques may also be used in place of or in addition to the data communication techniques described above.

In one possible embodiment, fig. 2 shows a schematic structural diagram of an on-screen interaction system, where the on-screen interaction system includes: a first on-screen terminal 210, a second on-screen terminal 220, and a server 230; taking the first and second on-screen terminals 210 and 220 as mobile phones, for example.

The first on-screen terminal 210 is configured to monitor an operation event of a user for a current operation page, and generate an operation record; the operation record is sent to the server 230, so that the server 230 generates a first synchronous operation instruction for controlling the second synchronous screen terminal 220 according to the operation record of the first synchronous screen terminal 210; the second on-screen terminal 220 is a terminal that establishes an on-screen interactive connection with the first on-screen terminal 210.

The server 230 is configured to receive an operation record sent by the first on-screen terminal 210; the operation record is generated by the first on-screen terminal 210 for monitoring the operation event of the current operation page in the first on-screen terminal 210; generating a first synchronous operation instruction for controlling the second synchronous screen terminal 220 according to the operation record of the first synchronous screen terminal 210; the second on-screen terminal 220 is a terminal that establishes an on-screen interactive connection with the first on-screen terminal 210.

The second synchronization screen terminal 220 is configured to receive a first synchronization operation instruction sent by the server 230.

The steps of the on-screen interaction method in this exemplary embodiment will be described in more detail below with reference to the accompanying drawings and examples.

The same-screen interaction of any two terminals is taken as an example for explanation, wherein any one terminal can be used as a first same-screen terminal.

In a possible embodiment, before the on-screen interaction, the first on-screen terminal and the second on-screen terminal need to establish an on-screen interaction connection, taking the execution subject as the first on-screen terminal as an example, fig. 3 shows a schematic flow chart for establishing the on-screen interaction connection, which specifically may include the following steps:

s302: and sending the on-screen interaction request to the server so that the server generates an on-screen interaction link according to the on-screen interaction request.

In one possible embodiment, the on-screen interaction request is a request sent by the first on-screen terminal to the server to establish on-screen interaction. The server can generate the on-screen interactive link according to the on-screen interactive request and feed back the on-screen interactive link to the first on-screen terminal. The unique identifier generated by the server is provided in the on-screen interactive link issued by the server, and is used for recording the on-screen interaction corresponding relation between terminals participating in the on-screen interaction by the server, specifically, the corresponding relation can be a specific UID (User Identification), and the server can be used for realizing the on-screen interaction between the first on-screen terminal and the second on-screen terminal through the on-screen interaction corresponding relation.

S304: and receiving the same-screen interactive link sent by the server.

In one possible embodiment, the on-screen interactive link may be a communication protocol of H5, a protocol that performs full duplex communication over a single TCP connection, such as a WebSocket link.

S306: and establishing the same-screen interactive connection with the second same-screen terminal through the same-screen interactive link.

In one possible embodiment, the first on-screen terminal and the second on-screen terminal are both connected in an on-screen interactive manner by an on-screen interactive link.

The on-screen interactive link of the second on-screen terminal may be transmitted by the first on-screen terminal. The server can also send the same-screen interaction request to the server, and the first same-screen terminal is attached with a terminal identifier of the second same-screen terminal, such as a mobile phone number, and the server sends the same-screen interaction link to the second same-screen terminal through the terminal identifier.

The specific way of establishing the interaction connection with the screen can be as follows:

responding to the operation of opening the same-screen interactive link by a user, displaying a message notification which is sent by a server and is waiting for the access of a second same-screen terminal, and receiving the same-screen access notification of the second same-screen terminal which is sent by the server; after the on-screen access notification of the second on-screen terminal is that the second on-screen terminal opens the on-screen interactive link sent by the first on-screen terminal or the server, the server sends the on-screen interactive link to the first on-screen terminal, and the on-screen interactive link is established with the second on-screen terminal according to the on-screen access notification of the second on-screen terminal.

After the on-screen interactive link is opened, the server recognizes the on-screen interactive link, sends a message notification waiting for the access of the terminal at the other side to the terminal which opens the on-screen interactive link until the on-screen interactive link is opened at the other side, completes the access, and sends an on-screen access notification to both the two on-screen terminals, informs both the on-screen terminals to access, and can operate. After the two same-screen terminals open the same-screen interactive links, the two same-screen terminals enter pages corresponding to the same-screen interactive links, wherein the pages can be produced by the server according to the transmitted same-screen interactive requests and are configured in advance.

Through the mode, the first screen-sharing terminal can more accurately determine that the connection is established with the second screen-sharing terminal through the server, so that the operation is started without establishing the connection, the screen-sharing interaction effect is influenced, and the screen-sharing interaction can be performed after the screen-sharing interaction connection is established.

Fig. 4 shows a flowchart of a method for on-screen interaction in an embodiment of the present disclosure, and illustrates a first on-screen terminal as an execution body, including the following steps:

s402: and monitoring an operation event of a user aiming at the current operation page, and generating an operation record.

In one possible embodiment, the current operation page may be an interface currently displayed by the first on-screen terminal, and the operation event may be an operation of the user in the current operation page, for example, a sliding event, a clicking event, a focus acquisition event, a focus loss event, and the like. The specific sliding event may be up-down and left-right sliding for the current operation page, and the clicking event may be classified into touching a return/confirm control, clicking a link, etc.

After the terminal monitors the operation event of the user aiming at the current operation page, a corresponding operation record can be generated, and the data format of the operation record can be the json format operation record. After the user-friendly information is conveniently uploaded to the server, the server performs identification and generates a first synchronous operation instruction.

S404: and sending the operation record to the server so that the server generates a first synchronous operation instruction for controlling a second synchronous screen terminal according to the operation record of the first synchronous screen terminal, wherein the second synchronous screen terminal is a terminal which establishes synchronous screen interactive connection with the first synchronous screen terminal.

In one possible embodiment, after sending the operation record to the server, the server can identify a second on-screen terminal that establishes an on-screen interactive connection with the first on-screen terminal, and generate a first synchronous operation instruction for controlling the second on-screen terminal.

For example, the operation event may be a sliding event, and after the json format operation record is generated, the server generates a first synchronization operation instruction for controlling the second synchronization screen terminal to slide, and sends the first synchronization operation instruction to the second synchronization screen interaction terminal, so that the first synchronization operation instruction is executed.

In one possible embodiment, the interaction between the first on-screen terminal and the second on-screen terminal may be mutual, and the second on-screen terminal may also monitor the operation event of its user, and generate an operation record and upload the operation record to the server, so that the server generates the second synchronous operation instruction. The second synchronous operation instruction is generated by the server according to the operation record of the second same-screen terminal and is used for controlling the operation instruction of the first same-screen terminal.

The first synchronous screen terminal can receive the second synchronous operation instruction sent by the server and execute the second synchronous operation instruction.

In a specific application scenario, a user a does not use corresponding APP software, cannot understand an operation manual, needs instruction of a marketer, and can also instruct through a mode of interaction with a screen in the related art, but performs transmission data instruction through a mode of recording a video stream, so that a large amount of network bandwidth resources are consumed, network delay exists, and the user a is likely to have misoperation and other phenomena according to the video recording, so that the user a cannot be helped in time.

In this scene, take marketing personnel to use first together screen terminal, user A uses the second to hold the screen terminal as an example, then two terminals establish with the screen interactive connection after, marketing personnel can carry out various operation events on first together screen terminal in real time, through with screen interaction, user A need not oneself to operate by hand, just can accomplish the use, and in same screen interactive period, user A also can carry out one time operation by oneself, let marketing personnel instruct, two terminals can control each other, marketing personnel help the user to carry out operation guidance with this kind of mode, and the user can realize accessible communication, through above-mentioned with screen interactive experience, can help the user to know the product fast.

The operation event may also be an out-of-focus event, which is an event that the current operation page loses focus, for example. The out-of-focus event may be an event for the first on-screen terminal that is responsive to the marketer operating on the current operating page of the first on-screen terminal such that the current operating page is out of focus.

For the second on-screen terminal, the event of losing focus of the first on-screen terminal may be that the second on-screen terminal pops up a dialog box or other interfaces in the current operation page to be displayed on the upper layer of the current operation page, so that the current operation page of the first on-screen terminal loses focus. The dialog box or other interface of the second on-screen terminal may be a page or the like requiring the user to fill in information or payment. Such interfaces require the user to operate themselves and are not able to be interfered by marketers.

After the first synchronous screen terminal triggers the event of losing focus, the server generates a first synchronous operation instruction from the operation record corresponding to the event of losing focus and sends the first synchronous operation instruction to the second synchronous screen terminal.

In order to protect user information or enable the user to operate by himself, the server can send waiting indication information to a first screen sharing terminal used by marketing personnel until a current operation page reacquires a focus, so that information safety of the user can be protected, and when the user is required to operate, misoperation of the marketing personnel can be prevented from affecting the user.

The first synchronous screen terminal can receive the waiting indication information sent by the server, display an interface waiting for re-acquiring the focus until receiving a second synchronous operation instruction for enabling the current operation page to re-acquire the focus, enable the current operation page to re-acquire the focus, and display the current operation page.

The second synchronous operation instruction for enabling the current operation page to acquire the focus may be an operation event for closing the dialog box after the user a completes the specific operation in the dialog box popped up by the current operation page of the second co-screen terminal, and may enable the current operation page of the first co-screen terminal to acquire the focus again, which may be referred to as an acquisition focus event of the current operation page.

It should be noted that, the first co-screen terminal and the second co-screen terminal can both receive the synchronous operation instruction of the server, and according to the operation automatically executed by the synchronous operation instruction, the terminal does not monitor and generate an operation record, or after generating the operation record, an identifier which is not required to control the co-screen interactive terminal is added, and the identifier is uploaded to the server for backup, so that the control effect is not further realized. The problems of network resource waste and terminal downtime caused by the fact that one operation event is repeatedly executed between two terminals after one terminal performs the operation event are solved.

And the operation records are generated and uploaded to the server by the users of the two same-screen interaction terminals aiming at the operation events of the users of the same-screen interaction terminals, and are used for controlling the opposite-side same-screen interaction terminals.

In a possible implementation manner, the embodiment of the present disclosure provides a flowchart of another on-screen interaction method, as shown in fig. 5, where the execution body may be a server, and specifically includes the following steps:

s502: and receiving an operation record sent by the first co-screen terminal, wherein the operation record is generated by the first co-screen terminal for monitoring an operation event of a current operation page in the first co-screen terminal by a user.

S504: and generating a first synchronous operation instruction for controlling a second synchronous screen terminal according to the operation record of the first synchronous screen terminal, wherein the second synchronous screen terminal is a terminal which establishes synchronous screen interactive connection with the first synchronous screen terminal.

In one possible embodiment, the server establishes a on-screen interactive connection between the first on-screen terminal and the second on-screen terminal, and may receive operation records from the first on-screen terminal and the second on-screen terminal, for generating the synchronous operation instruction. The same screen interaction can be realized in real time.

In one possible embodiment, the server receives the on-screen interaction request from the terminal and issues the on-screen interaction link before receiving the operation record and generating the synchronous operation instruction. The first and second same-screen terminals can both send the same-screen interaction request to the server, and the first same-screen terminal sends the same-screen interaction request is taken as an example.

The server receives an on-screen interaction request sent by the first on-screen terminal, generates an on-screen interaction link according to the on-screen interaction request, and sends the on-screen interaction link to the first on-screen terminal or the first on-screen terminal and the second on-screen terminal, so that the first on-screen terminal and the second on-screen terminal establish on-screen interaction connection through the on-screen interaction link.

The on-screen interactive link comprises a unique identifier generated by a server, and the server can record the corresponding relation between customer service and a user through the unique identifier.

And the server only sends the same-screen interactive link to the first same-screen terminal, and the first same-screen terminal sends the same-screen interactive link to the second same-screen terminal.

In a possible embodiment, after the first and second on-screen terminals both receive the on-screen interactive link, the on-screen interactive link is opened, and then the server receives the unique identifier sent after the first and second on-screen terminals open the on-screen interactive link, establishes an on-screen interactive corresponding relationship between the first and second on-screen terminals through the unique identifier, and sends a synchronous operation instruction between the first and second on-screen terminals through the on-screen interactive corresponding relationship.

After the first same-screen terminal and the second same-screen terminal open the same-screen interactive link, specific feedback is sent to the server, the feedback comprises unique identification in the same-screen interactive link, the server records that the first same-screen terminal and the second same-screen terminal open the same-screen interactive link through the unique identification, and the corresponding relationship of the same-screen interaction is established.

The server receives the operation record of the first synchronous screen terminal, generates a first synchronous operation instruction, determines a second synchronous screen terminal which is in synchronous screen interaction connection with the first synchronous screen terminal according to the synchronous screen interaction corresponding relation, and sends the first synchronous operation instruction to the second synchronous screen terminal.

In a possible embodiment, if the operation record generated by the focus losing event is the operation record received by the server, waiting indication information is sent to the first synchronous screen terminal, so that the first synchronous screen terminal displays an interface waiting for re-acquiring the focus, the first synchronous operation instruction of the focus losing event is sent to the second synchronous screen terminal, the operation record including the focus losing event sent by the second synchronous screen terminal is received, a second synchronous operation instruction for controlling the current operation page of the first synchronous screen terminal to re-acquire the focus is generated, the second synchronous operation instruction is sent to the first synchronous screen terminal, the current operation page in the first synchronous screen terminal is caused to re-acquire the focus, and the current operation page is displayed.

In a possible embodiment, the server may be further configured to process the uploaded operation record, and if some operation events may be repeated or not used in the process of performing on-screen interaction between the marketer and the user, the server may process the data to improve the utilization rate of network resources, so as to prevent the server from forwarding the same operation instruction or not used operation instruction multiple times.

In a possible embodiment, if the operation record of the first on-screen terminal includes a plurality of continuous identical operation events within a preset time period, the plurality of continuous identical operation events are combined to obtain a combined operation event.

The same operation event may be specifically the same sliding event, and slide downwards for multiple times continuously, and after receiving the operation record, the server may merge into one merged sliding operation event, and generate a first synchronous operation instruction corresponding to the merged sliding operation event.

Exemplary, fig. 6 shows a flow chart of another on-screen interaction method, and fig. 6 shows the following steps:

s602: and receiving an operation record sent by the first co-screen terminal, wherein the operation record is generated by the first co-screen terminal for monitoring an operation event of a current operation page in the first co-screen terminal by a user.

S604: the method comprises the steps of identifying that a plurality of continuous sliding events are included in an operation record in a preset time period.

S606: combining a plurality of continuous sliding events to obtain a combined sliding operation event.

S608: and generating a first synchronous operation instruction corresponding to the combined sliding operation event according to the combined sliding operation event.

S610: and sending a first synchronous operation instruction corresponding to the combined sliding operation event to a second synchronous screen terminal, wherein the second synchronous screen terminal is a terminal which establishes synchronous screen interactive connection with the first synchronous screen terminal.

The continuous identical operation event may be a continuous operation event for a control, or another continuous identical operation event. After the data are processed by the server, the rendering times of the customer service end/user end can be reduced while the network transmission is reduced, the network resources are saved, and the user experience can be improved.

If the operation record of the first co-screen terminal includes a click operation event that has no influence on the current operation page in the preset time period, deleting the operation record corresponding to the click operation event.

The click operation event that has no influence on the current operation page can be understood as: the event of continuously clicking at the blank of the current operation page of the terminal has no influence on the current operation page, so that the frequency of network transmission operation instructions can be reduced, and network resources are saved.

In one possible embodiment, the on-screen interaction method provided by the embodiment of the present disclosure is explained through interaction among the first on-screen terminal, the second on-screen terminal and the server. As shown in fig. 7, the method specifically includes the following steps:

S702: and the first on-screen terminal sends an on-screen interaction request to the server.

S704: and the server feeds back the on-screen interactive link to the first on-screen terminal.

In a possible embodiment, the on-screen interaction link is a websocket link, a current operation interface of the two terminals for on-screen interaction can be opened through the websocket link, the server can also determine that the first on-screen terminal and the second on-screen terminal enter the current operation page through the websocket link, and the on-screen interaction corresponding relation between the first on-screen terminal and the second on-screen terminal is established through a unique identifier in the on-screen interaction link.

S706: and the first on-screen terminal sends the on-screen interactive link to the second on-screen terminal.

S708: and responding to the operation of opening the on-screen interactive link by the first user in the first on-screen terminal, and sending a feedback message for opening the on-screen interactive link to the server, wherein the feedback message has a unique identifier.

S710: and responding to the operation of opening the on-screen interactive link by the second user in the second on-screen terminal, and sending a feedback message for opening the on-screen interactive link to the server, wherein the feedback message has a unique identifier.

S712: the server receives unique identifiers sent by the first same-screen terminal and the second same-screen terminal, and establishes a same-screen interaction corresponding relation between the first same-screen terminal and the second same-screen terminal.

S714: and sending the same-screen access notification of the second same-screen terminal to the first same-screen terminal, and sending the same-screen access notification of the first same-screen terminal to the second same-screen terminal.

In one possible embodiment, the first user and the second user may first open the on-screen interactive link, which is not described herein in detail, and the first user and the second user open the on-screen interactive link together.

The first user is a user using the first on-screen terminal, and the second user is a user using the second on-screen terminal.

S716: and the first and second same-screen terminals determine to establish the same-screen interactive connection.

S718: the first co-screen terminal monitors an operation event of a user aiming at a current operation page and generates an operation record.

S720: and the first co-screen terminal uploads the operation record to the server.

S722: the server receives the operation record, processes the data of the operation record, and generates a first synchronous operation instruction for controlling the second synchronous screen terminal.

S724: and the server sends the first synchronous operation instruction to the second synchronous screen terminal through the corresponding relationship of the synchronous screen interaction.

Through the embodiment, the on-screen interaction method can realize the on-screen interaction between two terminals by monitoring the operation event in real time, transmitting the operation record and sending the synchronous operation instruction through the server, so that the on-screen interaction can be realized in real time without barriers, network resources can be saved, the consumption of network bandwidth is reduced, and the network delay is reduced.

In addition, in the method, the server can process continuous identical operation events and operation events without influence, network resources are further saved in the process of transmitting synchronous operation instructions, the rendering times of network rendering participation in the same-screen interaction terminal are reduced, and user experience is improved.

Based on the same inventive concept as the method embodiment, the embodiment of the application also provides a same-screen interaction device. Fig. 8 shows a schematic structural diagram of a one-screen interaction device according to an embodiment of the present application. The on-screen interaction device is applied to a first on-screen terminal, as shown in fig. 8, and the on-screen interaction device 800 includes:

a monitoring unit 801, configured to monitor an operation event of a user for a current operation page, and generate an operation record; a first sending unit 802, configured to send the operation record to the server, so that the server generates a first synchronous operation instruction for controlling the second synchronous screen terminal according to the operation record of the first synchronous screen terminal; the second same-screen terminal is a terminal which establishes same-screen interactive connection with the first same-screen terminal.

Fig. 9 shows a schematic structural diagram of a one-screen interaction device according to an embodiment of the present application. The on-screen interaction device is applied to a server, as shown in fig. 9, and the on-screen interaction device 900 includes:

A receiving unit 901, configured to receive an operation record sent by a first together screen terminal; the operation record is generated by a first co-screen terminal for monitoring an operation event of a current operation page in the first co-screen terminal by a user; a second sending unit 902, configured to generate a first synchronization operation instruction for controlling the second synchronization terminal according to the operation record of the first synchronization terminal; the second same-screen terminal is a terminal which establishes same-screen interactive connection with the first same-screen terminal.

Those skilled in the art will appreciate that the various aspects of the invention may be implemented as a system, method, or program product. Accordingly, aspects of the invention may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.

An electronic device 1000 according to this embodiment of the present invention is described below with reference to fig. 10. The electronic device 1000 shown in fig. 10 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.

As shown in fig. 10, the electronic device 1000 is embodied in the form of a general purpose computing device. Components of electronic device 1000 may include, but are not limited to: the at least one processing unit 1010, the at least one memory unit 1020, and a bus 1030 that connects the various system components, including the memory unit 1020 and the processing unit 1010.

Wherein the storage unit stores program code that is executable by the processing unit 1010 such that the processing unit 1010 performs steps according to various exemplary embodiments of the present invention described in the above section of the "exemplary method" of the present specification. For example, the processing unit 1010 may perform the on-screen interaction method described above.

The memory unit 1020 may include readable media in the form of volatile memory units such as Random Access Memory (RAM) 10201 and/or cache memory unit 10202, and may further include Read Only Memory (ROM) 10203.

The storage unit 1020 may also include a program/utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

Bus 1030 may be representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 1000 can also communicate with one or more external devices 1040 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 1000, and/or with any device (e.g., router, modem, etc.) that enables the electronic device 1000 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 1050. Also, electronic device 1000 can communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 over the bus 1030. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with the electronic device 1000, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.

In an exemplary embodiment of the present disclosure, a computer-readable storage medium having stored thereon a program product capable of implementing the method described above in the present specification is also provided. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the invention as described in the "exemplary methods" section of this specification, when said program product is run on the terminal device.

A program product for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read-only memory (CD-ROM) and comprise program code and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.

Furthermore, although the steps of the methods in the present disclosure are depicted in a particular order in the drawings, this does not require or imply that the steps must be performed in that particular order or that all illustrated steps be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform, etc.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. The on-screen interaction method is characterized by being applied to a first on-screen terminal, and comprises the following steps:

2. The method of claim 1, wherein the monitoring the user for an operational event for a current operational page is preceded by:

receiving the same-screen interactive link sent by the server;

3. The method according to claim 2, wherein the establishing a on-screen interactive connection with the second on-screen terminal through the on-screen interactive link comprises:

4. The method according to claim 1, wherein the method further comprises:

and executing the second synchronous operation instruction.

5. The method of claim 1, wherein the operational event comprises a loss of focus event; the event of losing focus is an event of losing focus of the current operation page;

after the operation record is sent to the server, the method further comprises:

6. A method for on-screen interaction, applied to a server, the method comprising:

7. The method of claim 6, wherein prior to receiving the operation record sent by the first on-screen terminal, the method further comprises:

8. The method of claim 7, wherein the on-screen interactive link includes a unique identifier; the method further comprises the steps of:

9. The method of claim 8, wherein after the generating the first synchronization operation instruction for controlling the second on-screen terminal, the method further comprises:

10. The method of claim 6, wherein the operational event comprises a loss of focus event; the event of losing focus is an event that a current operation page in the first same screen terminal loses focus;

11. The method of claim 6, wherein the method further comprises:

12. The method of claim 11, wherein the plurality of consecutive identical operational events are a plurality of consecutive page-swipe operational events;

13. The method of claim 6, wherein the method further comprises:

14. The utility model provides a with screen interactive device which characterized in that is applied to first together screen terminal, includes:

15. An on-screen interaction device, which is applied to a server, and comprises:

16. An on-screen interactive system, comprising: the system comprises a first co-screen terminal, a second co-screen terminal and a server;

17. An electronic device, comprising:

a processor; and

a memory for storing executable instructions of the processor;

wherein the processor is configured to perform the on-screen interaction method of any one of claims 1-13 via execution of the executable instructions.

18. A computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the on-screen interaction method of any one of claims 1 to 13.