CN114884914A - Application program same-screen communication method and system - Google Patents

Application program same-screen communication method and system Download PDF

Info

Publication number
CN114884914A
CN114884914A CN202210437002.1A CN202210437002A CN114884914A CN 114884914 A CN114884914 A CN 114884914A CN 202210437002 A CN202210437002 A CN 202210437002A CN 114884914 A CN114884914 A CN 114884914A
Authority
CN
China
Prior art keywords
data
application program
client
screen
synchronous
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210437002.1A
Other languages
Chinese (zh)
Other versions
CN114884914B (en
Inventor
路九阳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qingdao Hope Bird Technology Co ltd
Original Assignee
Qingdao Hope Bird Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qingdao Hope Bird Technology Co ltd filed Critical Qingdao Hope Bird Technology Co ltd
Priority to CN202210437002.1A priority Critical patent/CN114884914B/en
Publication of CN114884914A publication Critical patent/CN114884914A/en
Application granted granted Critical
Publication of CN114884914B publication Critical patent/CN114884914B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/04—Real-time or near real-time messaging, e.g. instant messaging [IM]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/02—Standardisation; Integration
    • H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
    • H04L41/0273—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using web services for network management, e.g. simple object access protocol [SOAP]
    • H04L41/028—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using web services for network management, e.g. simple object access protocol [SOAP] for synchronisation between service call and response
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/04—Real-time or near real-time messaging, e.g. instant messaging [IM]
    • H04L51/046—Interoperability with other network applications or services
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/07—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail characterised by the inclusion of specific contents
    • H04L51/10—Multimedia information

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The application provides a method and a system for application program on-screen communication, wherein the method comprises the following steps: the method comprises the steps that a first application program runs on a first client, a user initiates a screen sharing request, a server receives the screen sharing request, a second application program corresponding to the first application program runs in a synchronous running environment, display data are generated based on the second application program and sent to the first client and the second client, the first client runs a third application program to display screen sharing data, and the second client runs a fourth application program to display screen sharing data. And acquiring user operation data based on the third application program or the fourth application program, sending the user operation data to the server, and actually executing user operation in the second application program of the server. Therefore, the method for initiating the same screen in the process of executing the business operation by the client is provided for the same-screen implementation mode of generating the synchronous display data based on the server, so that the client and customer service personnel can initiate the same-screen request, the business use scene is expanded, and the flexibility is high.

Description

Application program same-screen communication method and system
Technical Field
The invention relates to the field of internet communication, in particular to a method and a system for application program on-screen communication.
Background
People often need to communicate with another user to perform explanation and assist operations, such as handling business and performing operations through the internet. For example, the user a accesses the cloud virtual device, executes an operation in the virtual device, and simultaneously sends the web link corresponding to the picture of the available virtual device to the user B, and the user B accesses the web link to obtain the picture of the virtual device, so that the requirement of synchronous browsing of the user a and the user B can be met. However, the technology is limited in that the user a needs to first access the cloud virtual device, and then generates a link capable of accessing the device screen based on the virtual device to send the link to the user B, and the user B cannot initiate a request for on-screen communication, so that the technology is not suitable for a conventional internet service scenario, that is, cannot be initiated by a user browsing a service webpage. As described above, the customer a is usually a customer service person, and the user B is usually a customer, which provides a way for the customer service to communicate with the contacted customers on the same screen, but for the customers from the network, the customer service is sought to assist in meeting the problem, and the above on-screen technology cannot be realized.
Therefore, it is necessary to provide a new method for implementing application programs on the same screen to solve the above problems, so that both the customer service and the client can initiate requests to invite the other party to enter the on-screen communication.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method and a system for on-screen communication of an application program, and aims to provide a method for initiating on-screen communication from a business operation terminal for an on-screen implementation manner in which synchronous display data is generated from a server, so as to enhance flexibility of on-screen application, enable a client to obtain on-screen assistance of a customer service staff in real time while executing business operation, and improve communication efficiency. The technical scheme provided by the invention is as follows:
in a first aspect, the present application provides an application program on-screen communication method, where the method is applied to a first client, where the first client includes a first application program and a third application program, and the method includes:
the method comprises the steps of running a first application program, receiving a co-screen request initiating event triggered by a user, collecting first running information based on the first application program, generating a co-screen request and sending the co-screen request to a server, or collecting user online information and sending the user online information to the server;
receiving a control instruction sent by a server, starting a third application program and entering a same-screen communication state when the control instruction is determined to be a same-screen entering instruction;
the third application program establishes communication connection with the server, receives first synchronous data sent by the server, and executes synchronous processing based on the first synchronous data, wherein the first synchronous data comprises one or more of display data, audio data, video data, graffiti data and instruction data, and the synchronous processing comprises one or more of display picture generation, audio and video playing, graffiti drawing execution and instruction execution;
and acquiring one or more of user operation data, audio and video data and scrawling data, generating second synchronous data, and sending the second synchronous data to the server, wherein the user operation data is generated based on a third application program acquisition user operation event and is used for being executed in a second application program of the server.
In the above implementation manner, the first application program is an application program for a user to perform a business operation, the same-screen request is received by the server, a second application program having the same content as the first application program is set and operated at the server, the first operation information is used for starting and/or setting the second application program, and the second application program is used for providing display data for the third application program.
The implementation mode generates synchronous display data at the server side and sends the synchronous display data to the client side to realize the same screen, so that a mode of initiating the same screen in the process of executing the business by the client is realized, and the flexibility of the business is enhanced.
In a possible implementation mode, after a first client enters a same-screen communication state, receiving a doodle starting control instruction sent by a server, or receiving a doodle starting event triggered by a user, and creating or displaying a Canvas for doodling on the first client; and/or collecting scrawling data based on the scrawling Canvas to generate second synchronous data for sending; based on the graffiti data in the first synchronization data, drawing is performed in a graffiti Canvas. The implementation implements synchronous graffiti in a third application.
In a possible implementation manner, first synchronization data sent by a server is received to execute synchronization processing, when the first synchronization data is determined to include instruction data for calling a local interface program, the local interface program is called at a first client, and second synchronization data is acquired and generated based on a local interface program execution result and sent to the server. The implementation mode solves the problem of calling the local interface program of the first client in the process of the same screen.
In a possible implementation manner, after the control instruction is received, when it is determined that the control instruction is the instruction for ending the same screen, the third application program is closed, the first application program is started to run, and the running state of the first application program is set based on the second running information of the second application program in the first synchronization data. After the same screen is finished, the first application program is recovered to run based on the result of the same screen process business operation, so that the business operation process is kept continuous.
In a second aspect, the present application provides an application program on-screen communication method, where the method is applied to a second client, the second client includes a fourth application program, and the method includes:
the fourth application program receives a control instruction sent by the server, and executes the instruction for entering the same screen to enter the same screen communication state when the control instruction is determined to be the instruction for entering the same screen;
receiving first synchronous data sent by a server, and executing synchronous processing based on the first synchronous data, wherein the first synchronous data comprises one or more of display data, audio data, video data and graffiti data, and the synchronous processing comprises one or more of generating a display picture, playing audio and video and executing graffiti drawing;
and acquiring one or more of user operation data, audio/video data and scrawling data, generating third synchronous data, and sending the third synchronous data to the server, wherein the user operation data is generated by acquiring a user operation event based on a fourth application program and is used for executing in a second application program of the server.
In the foregoing implementation manner, the second client presents the display content of the second application by running the fourth application, so as to implement the on-screen browsing with the first client, and a method for executing the on-screen operation in the fourth application is further provided.
In a possible implementation manner, user online information of a first application program sent by a server is received, a user is selected based on the user online information, and a co-screen invitation request is created and sent to the server. This implementation provides a method of initiating a co-screen request at a second client.
In a possible implementation manner, after the second client enters the same-screen communication state, receiving a doodle starting control instruction sent by the server, or receiving a doodle starting event triggered by a user, and creating or displaying a Canvas for doodling on the second client; collecting doodle data based on the doodle Canvas to generate third synchronous data to be sent; based on the graffiti data in the first synchronization data, drawing is performed in a graffiti Canvas. The implementation implements synchronized graffiti in a fourth application.
In a third aspect, the present application provides a method for implementing on-screen communication of an application, where the method is applied to a server, and the method includes:
receiving a co-screen request sent by a first client or a second client;
processing the same-screen request and sending a same-screen invitation notification to another client;
creating or starting a synchronous operation environment;
starting and/or setting a second application program corresponding to the first application program to run in the synchronous running environment based on first running information of the first application program contained in the on-screen request;
establishing communication connection between the server and the third application program and between the server and the fourth application program;
acquiring display data based on a second application program to generate first synchronization data, and sending the first synchronization data to a first client and a second client, wherein the display data comprises one of image data, video data, snapshot data and drawing instruction data; receiving second synchronous data sent by a first client or third synchronous data sent by a second client, acquiring user operation data in the second synchronous data, and executing user operation events contained in the second synchronous data in a second application program;
in the foregoing implementation manner, the second application is configured to generate synchronous display data for the first client and the second client, and start and/or set the second application based on the first running information of the first application included in the on-screen request, so that the second application obtains a running state that is the same as or similar to that of the first application.
Through the implementation mode, according to the running state of the first application program, a same-screen reference execution environment is established at the server side, the first client side and the second client side are synchronized by taking the running picture of the second application program as a reference, and meanwhile, the operation event received from the client side is executed in the second application program, so that the effect of the client side in operating the application program is achieved.
In a possible implementation manner, a second application program is run in a synchronous running environment, when the behavior that the second application program calls a local interface program of a first client is determined, instruction data for calling the local interface program is generated and generated as first synchronous data to be sent to the first client, so that the problem of calling the local interface program of the first client in the process of the same screen is solved. Optionally, the second synchronization data sent by the first client is received, the local interface program execution result included in the second synchronization data is analyzed, and the second application program is executed based on the local interface program execution result.
In a possible implementation manner, a second application program is run in a synchronous running environment, when the behavior that the second application program calls a local interface program of a second client is determined, instruction data for calling the local interface program is generated and generated as first synchronous data to be sent to the second client, so that the problem of calling the local interface program of the second client in the process of the same screen is solved. Optionally, the third synchronization data sent by the second client is received, the local interface program execution result included in the third synchronization data is analyzed, and the second application program is executed based on the local interface program execution result.
In a possible implementation manner, a request for ending the same screen sent by a first client or a second client is received, the request for ending the same screen is processed, a command for ending the same screen is sent to the first client or the second client, second running information is collected based on a second application program to generate first synchronization data, and the first synchronization data is sent to the first client, wherein the second running information is used for setting a running state of the first application program at the first client, so that a business operation process of the first client is kept continuous after the same screen is ended.
In a fourth aspect, the application further provides an application program co-screen system, where the system includes a first client, a server, and a second client, and the first client includes a first application program and a third application program; the server comprises a synchronous operation environment, and the synchronous operation environment operates a second application program; the second client includes a fourth application, wherein:
the first client is used for running a first application program and generating and sending a same-screen request; the system is also used for collecting and sending the user online information; the device is also used for receiving and executing a control instruction, and starting a third application program when the control instruction is determined to be a command for entering the same screen;
the first client is also used for running a third application program, receiving the first synchronization data and executing synchronization processing in the third application program based on the first synchronization data; the system is also used for acquiring and sending second synchronous data based on a third application program; the system is also used for creating a doodle Canvas, collecting doodle data and drawing based on the doodle data; the server is also used for calling the local interface program according to the received instruction of the local interface program, acquiring an execution result, generating second synchronous data and sending the second synchronous data to the server; the first synchronization data is used for receiving a control instruction of the first application program, and the control instruction is used for receiving a first synchronization data;
the server is used for receiving and processing the same-screen request and ending the same-screen request; the system is also used for receiving and sending user state information and a same-screen invitation request; the system is also used for sending notification and control instructions; the system is also used for configuring and starting a synchronous operation environment after processing the same-screen request, and setting and operating a second application program; the synchronous operation system is also used for sending a control instruction to the synchronous operation environment;
the server is also used for establishing long connection with the first client and the second client; the first synchronization data is also used for generating and sending the first synchronization data; the display data acquisition module is also used for acquiring display data based on a second application program and generating first synchronization data; the first application program is used for executing the first synchronous data and the second synchronous data; the system is also used for forwarding second synchronous data and third synchronous data, wherein the forwarded second synchronous data and third synchronous data comprise one or more of audio data, video data and doodle data;
the synchronous operation environment is used for operating the second application program based on the first operation information of the first application program in the on-screen request; the system is also used for acquiring display data of a second application program;
the second application program is used for generating display data of the same screen; the data processing device is also used for executing user operation events contained in the second synchronous data or the third synchronous data;
the server is also used for generating instruction data for calling the local interface program and sending the instruction data serving as first synchronization data to the first client when the behavior that the second application program calls the local interface program of the first client is determined; the second synchronous data is used for executing the second application program based on the execution result of the local interface program when the execution result of the local interface program is determined to be contained in the second synchronous data; the first application program is used for generating instruction data for calling the local interface program and sending the instruction data serving as first synchronization data to the first client side when determining that the second application program calls the behavior of the local interface program of the first client side; the second application program is executed based on the execution result of the local interface program when the third synchronous data is determined to contain the execution result of the local interface program; the first operating information acquisition module is also used for acquiring second operating information based on a second application program and sending the second operating information as first synchronization data to the first client when the on-screen request is processed;
the second client is also used for running a fourth application program, receiving the first synchronization data and executing synchronization processing in the fourth application program based on the first synchronization data; the device is also used for receiving and executing the control command, and entering on-screen communication when the control command is determined to be an on-screen command; the server is also used for receiving user online information of the first application program sent by the server, selecting a user based on the user online information, creating a same-screen invitation request and sending the same; the system is also used for acquiring and sending third synchronous data based on a fourth application program; the system is also used for creating a doodle Canvas, collecting doodle data and drawing based on the doodle data; the server is also used for calling the local interface program according to the received instruction of the local interface program, acquiring an execution result, generating third synchronous data and sending the third synchronous data to the server;
the first synchronous data comprise one or more of display data, instruction data and operation data, and the second synchronous data and the third synchronous data comprise one or more of user operation data, audio and video data and scrawling data.
In a fifth aspect, the present application further provides a computer storage medium, where a software program is stored, and when the software program is read and executed by one or more processors, the software program can implement the method provided by any one of the implementation manners of any one of the foregoing aspects.
In a sixth aspect, this application further provides a computer program product, which when read and executed by one or more processors can implement the method provided in any one of the above-mentioned aspects.
Therefore, on the basis of a same-screen implementation mode that a server generates synchronous display data, the method for initiating the same-screen request at a client executing business operation is realized. Therefore, the application program on-screen method provided by the application program can be initiated by any party aiming at the real-time scene of the operation service, and is strong in flexibility and wide in applicable service scene.
Drawings
Fig. 1 is a schematic structural diagram of an application on-screen communication system according to an embodiment of the present application;
fig. 2 is a schematic flowchart of an application program co-screen communication according to an embodiment of the present disclosure;
fig. 3 is a schematic flowchart of initiating an application on-screen communication at a second client according to an embodiment of the present application;
fig. 4 is a schematic flowchart of a graffiti processing in the same-screen communication of an application program according to an embodiment of the present disclosure;
fig. 5 is a schematic flowchart illustrating a process of calling a first client local interface program in the application program on-screen communication according to an embodiment of the present application;
fig. 6 is a schematic view of a process flow for ending the on-screen communication of the application program in the embodiment of the present application.
Detailed Description
The embodiments of the present application are in part illustrative of the present application and are not intended to be limiting of the present application.
Hereinafter, embodiments of the present application will be described, and it is apparent that the described embodiments are only some embodiments of the present application, and not all embodiments. As can be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solution provided in the embodiments of the present application is also applicable to similar technical problems.
In the existing on-screen technology, an application program is run at a server, and then display data generated by running is sent to a client to obtain an on-screen effect. For example, the APP is run in the virtual mobile phone device environment of the server, the display picture of the virtual device is converted into a video stream and sent to the two mobile phone terminals, and the video stream is received in the client application program of each mobile phone terminal, so that the effect of on-screen communication between the two mobile phone terminal users can be achieved. However, the above one-screen implementation techniques greatly limit the service usage scenario, and generally, a customer service person starts a service-side virtual device to operate through a client program, and obtains an operation screen and controls the APP on the virtual device to operate, at the same time, a web page link is generated and sent to a customer, the customer accesses the link to load a web page, and obtains the APP screen on the specified virtual device through the web page, thereby implementing the one-screen browsing between the customer service person and the customer. Analyzing the above implementation manner, the client can only access the link sent by the customer service staff to enter the same screen, namely, the client is a first-contact and second-screen business mode, the applicable scene is very narrow, and for the scene of business operation actively performed by the client, the client cannot initiate the same-screen communication in real time, so that the client is not suitable for the requirement of internet business operation communication.
The application achieves the aim that: aiming at the on-screen implementation mode of generating on-screen reference display data at the server, a method for initiating on-screen communication by a client in the process of executing business operation is provided, so that communication and help support can be obtained according to the current operation content.
First, the noun in this application section is briefly described to facilitate understanding by those skilled in the art.
1) The client, also called client, is a locally running application provided for a user, and usually the client corresponds to the server.
2) The same screen means that the same or similar browsing and executing effect is realized on a certain program between two clients.
3) An application program refers to a computer program for performing one or more specific tasks, and the application is not limited to the specific form thereof, for example: web pages, APPs, etc. In addition, in the present application, terms such as "first application", "second application", "third application", "fourth application", and the like all refer to functional divisions logically included therein. For example, in the case where a client performs an insurance application and the client serves as an aid, the "first application" is a program for the client to perform the insurance application from the internet; the second application program is a first application program operated by a client, is an application program operated in a service end operation environment, has the operation effect approximately same as that of the first application program, is used for generating reference display data on the same screen, and is used as an actual executive program of business operation in the communication process; the third application program is an application program which runs at the client side after entering the same screen, receives the reference display data of the same screen and obtains the same screen effect; the "fourth application" is an application running on the customer service side, which receives the reference display data on the same screen, and can generally send an operation instruction to operate the server side "second application" to execute.
4) The server-side synchronous running environment is used for running a second application program, such as: virtual devices, cloud handsets, browsers, headless browsers, and the like.
5) Display data refers to data collected based on the second application and displayed in the third application and the fourth application, such as images, video streams, snapshot data (e.g.: web page DOM data), drawing instructions, etc.
6) And/or, describing the association relationship of the associated object, indicating that there may be three relationships, e.g., a and/or B, which may indicate: a exists alone, A and B exist simultaneously, B exists alone, the character "/" generally indicates that the associated objects before and after the character are in an "OR" relationship.
The technical solution of the present application will be described in detail by using several specific examples.
Fig. 1 shows a possible application on-screen communication system structure, which includes a first client 10, a server 20, and a second client 30, where the first client 10 includes a first application 101 and a third application 102, the server 20 includes a synchronous running environment 201, the synchronous running environment 201 runs a second application 2011, and the second client 30 includes a fourth application 301.
The method for the application program on-screen communication provided in the embodiment of the present application may be applied to the application program on-screen communication system structure shown in fig. 1, and with reference to fig. 2, a specific process of the method includes:
in the first client 10:
s201, running a first application program; s202, receiving a co-screen request initiating event triggered by a user, acquiring first running information based on a first application program, and generating a co-screen request; s203, sending a co-screen request to the server 20. The first application program is a program used by a user to execute business operation, such as an insurance program provided by an insurance company and accessed by an internet user. Optionally, the online customer service staff information is displayed in the first application program, and S202 selects the customer service staff to initiate the on-screen request. The operation information collected in S202 includes, but is not limited to, one or more of a page address, program data, an operation environment parameter, a UI focus, a browsing parameter, and the like, for example, the first application is a web page, and the collected information is, for example, a current web page URL, inputted or operated page data, browser version information, a current focus control, a cursor position, a current web page sliding position, a display area, and the like, and is used for setting a second application to operate at the server.
In the server 20:
receiving a co-screen request sent by a first client 10; s204, processing the same-screen request; s205, send the on-screen invitation to the second client 30.
In the second client 30:
running a fourth application program, and receiving the same-screen invitation sent by the server 20; s206, processing the same-screen invitation and generating a notice; according to the processing result, S207, a notification is sent.
In the server 20:
after receiving the same-screen invitation notification sent by the second client 30, indicating that both clients agree to start the same screen, and S208, creating or starting a synchronous operation environment; s209, sending an instruction to enter the same screen to the first client 10; s210, sending an instruction to enter the same screen to the second client 30. The synchronous execution environment is used for executing a second application program, such as a virtual device, a browser and the like. The execution time of S209 and S210 is not particularly limited, and may be executed after the second application is started in S213, for example.
In the second client 30:
after receiving the instruction of entering the same screen sent by the server 20, S211 establishes a long connection with the server, and enters the same screen state. Optionally, communication between the fourth application and the synchronous operation environment is established.
In the first client 10:
after receiving the instruction of entering the same screen sent by the server 20, S212 starts the third application program, establishes a long connection with the server, and enters the same screen state. Optionally, communication between the third application and the synchronous operation environment is established.
Optionally, in the above S211 and S212, establishing audio-video communication between the first client 20 and the second client 30 is further included.
In the server 20:
s213, starting and/or setting the second application program to run; s214, collecting display data to generate first synchronization data; s215, sending the first synchronization data to the second client 30; s216, the first synchronization data is sent to the first client 10. The second application program is a program having the same or similar function as the first application program, and is run in an environment designated by the server to generate the on-screen reference display data, and the second application program is set to run in S213, that is, the second application program is set by using the first running information acquired in S202, and the main purpose of the second application program is to obtain a running state the same as or similar to that of the first application program, for example, a client performs an application operation through the first application program to input part of form data, carries a page address, the part of form data and the like in the process of initiating an on-screen request, and sets the second application program as the first running information, so that the second application program loads a corresponding page, and writes the form data input by the user, thereby obtaining an on-screen effect that service operation is not interrupted. Optionally, the second application program is the same as the first application program. Optionally, the second application program and the first application program are programs with similar functions. In S214, the display data includes, for example, an application screenshot, a virtual device display screenshot, a video stream, a sequence image, snapshot data (e.g., web page DOM data), a drawing instruction, and the like. Optionally, communication is established between the second application program and the third application program to directly transmit the synchronous data, and communication is established between the second application program and the fourth application program to directly transmit the synchronous data.
In the first client 10:
s217 receives the first synchronization data, and performs synchronization processing.
In the second client 30:
s218, the first synchronization data is received, and the synchronization process is executed.
In the above S217 and S218, the first synchronization data includes, but is not limited to, one or more of video data, audio data, display data, instruction data, and graffiti data, and the synchronization process includes displaying a picture, playing audio and video, executing an instruction, executing graffiti drawing, and the like according to the nature of the first synchronization data. For example, the display data in the first synchronization data is a video stream, and the video stream is played in S218 to display the running screen of the second application.
For example, a user operation occurs in the second client 30, and the following steps are performed:
in the second client 30:
s219, collecting a user operation event to generate third synchronous data; s220, sending the third synchronization data to the server 20. For example, on the basis of playing the video stream to display the running picture of the second application program in S218 above, S219 monitors the operation event of the obtaining user, obtains the operation behavior of the user on the video playing window, and generates third synchronization data, such as collecting position coordinate information clicked by the user, inputting data, and the like. Optionally, in S219, the collected user operation event is sent to the server 20, the server generates a return message according to an execution result of the user operation event in the second application 30, and sends the return message to the second client, then, the second client receives the return message and executes corresponding processing, for example, a user clicks an input component in a video frame of a fourth application, after the clicked position information is sent to the server, the user click event corresponding to the input component is executed in the second application, the generated execution result is "input in an input control", the return message is generated based on the execution result and sent to the second client, the second client receives the return message and processes the return message, invokes an input method to receive input content, then, monitors the input event executed by the user, and continues to execute the step S219, the input content is displayed in an input component of the second application.
Similarly, the related processing in step S219 above may also be executed in the first client 10, and based on the user operation occurring in the first client 10, the second synchronization data generated by collecting the user operation event is sent to the server 20.
In the server 20:
s221, receiving third synchronization data sent by the second client 30, acquiring user operation data therein, and executing the third synchronization data in the second application program; s222, collecting display data to generate first synchronization data; s223, sending the first synchronization data to the second client 30; s224, the first synchronization data is sent to the first client 10.
In the first client 10:
s225, receives the first synchronization data sent by the server 20, and executes the synchronization process. This step is identical to step S217 in nature and will not be described herein.
In the second client 30:
s226, the first synchronization data sent by the server 20 is received, and the synchronization process is executed. The step is identical to step S218 in nature, and is not described herein again.
In the above S214 and S222, the second application display data is collected to generate the first synchronization data, and optionally, when it is monitored that the display content of the second application changes, the collection of the display data, the generation of the first synchronization data, and the sending process are executed.
The "synchronization data" in this application may be understood as data transmitted for realizing display content synchronization, audio synchronization, video synchronization, and graffiti synchronization between the first client and the second client, and data transmitted for realizing display content synchronization between the first client, the second client, and the server and the second application. The method comprises the steps of defining synchronous data generated at the server side as first synchronous data, defining synchronous data generated at the first client side as second synchronous data, and defining synchronous data generated at the second client side as third synchronous data.
The above embodiment takes the establishment of the on-screen communication between the first client and the second client as an example to describe the implementation principle of the present application, but it should be noted that the above embodiment is also applicable to the establishment of the on-screen communication between a plurality of first clients and a plurality of second clients.
In a specific implementation scenario, the first client is a network program, such as a company webpage, an APP program, an applet, and the like, accessed by an internet client, a service end uses a virtual device as a synchronous operation environment, such as a "cloud mobile phone", and the second client is software capable of remotely accessing and controlling the "cloud mobile phone".
The core of the implementation mode is that according to a first application program operated by a first client, a second application program which is the same as or similar to the first application program is started in a synchronous operation environment of a server and is used as a display reference picture of the same screen for communication, so that the problem of inconsistent pictures of the same screen caused by the difference of client terminal equipment is effectively avoided, and in the communication process based on the second application program, both parties cannot check the calling process of a local interface program of the client and cannot directly access local data of the client, thereby being beneficial to protecting the privacy of a user in the process of the same screen.
Through the implementation mode, the Internet client can initiate the same-screen request to invite the customer service personnel in the process of executing the business operation by using the first application program, and enter the same-screen communication after the customer service personnel answer, so that the problem of initiating the business operation end of the same-screen implementation mode for generating the reference display data at the service end is effectively solved.
In another possible implementation, as shown in fig. 3, a manner in which a customer service person initiates a request for the same screen is shown, and the detailed steps are as follows:
in the first client 10:
s301, running a first application program; s302, collecting user online information; s303, sending the user online information. The above steps are, for example, the internet client accessing the first application program to perform the business operation, and the first application program collects the user online information and sends the user online information to the server 20. The user online information includes, but is not limited to, user identity information, browsing status information, operation data information, and the like.
After receiving the user online information, the server 20 executes S304 to send the user online information, and sends the user online information of the first application program to the second client 30.
In the second client 30:
s305, selecting a user based on the user online information and creating a same-screen invitation request; s306, sending the same-screen invitation request. In the step, the customer service staff selects a certain user to initiate a co-screen invitation request according to the acquired user online information of the first application program, namely, the customer service staff actively assists the customer in operating the service. Optionally, the customer service staff knows the operation state of the user through the user online information, for example, the time of staying at a certain service operation part, and analyzes according to a predetermined judgment rule, and selects the client to initiate the on-screen invitation when determining that the user has an operation problem.
In the server 20:
s307, processing the same-screen invitation request; s308, sending the same-screen invitation. The server 20 forwards the received on-screen invitation request to the first client.
In the first client 10:
s309, processing the same screen invitation and generating a notice; s310, sending a notice. The processing result of S309 comprises accepting and rejecting, and corresponding notice is generated. Optionally, after the on-screen invitation is accepted in S309, the running state of the current first application program is collected as first running information, and the first running information is sent together with the notification of accepting the on-screen invitation, where the first running information is used by the server 20 to set and run the second application program based on the first running information.
In the server 20:
s311, creating or starting a synchronous operation environment; s312, sending an instruction to enter the same screen to the first client 10; s313, send an instruction to enter the same screen to the second client 30. After receiving the notification of accepting the same-screen invitation sent in S310, it indicates that both parties agree to the same-screen communication, and executes S311 to create or start a synchronous running environment to run a second application, and notifies both parties of the client.
S314, S315, S316 and the subsequent steps are substantially the same as the previous embodiment, and are not described herein again. It should be noted that in S314, a third application program is started to enter the on-screen communication, where the third application program is used to display the running screen of the second application program in the server 20. In addition, on the basis that the notification sent in S310 includes the first running information, S316 sets the second application program based on the first running information, so that after the on-screen communication is performed, the content displayed by the first client in the third application program is substantially the same as the content displayed by the first application program before the on-screen communication is performed, the continuity of the service operation is maintained, and similarly, the second client can obtain the current operating state of the user, and the user experience is good.
Through the implementation manner, the following beneficial effects can be obtained: customer service personnel can obtain the online information and the operation state of the user; customer service personnel can actively initiate a co-screen communication request to assist customers; after the same-screen communication is carried out, the operation service content of the client is not interrupted.
In a possible implementation manner, the present application further provides a method for implementing synchronous doodling in the same screen communication of an application program, as shown in fig. 4:
after the first client 10 and the second client 30 enter the on-screen communication state, for example:
in the second client 30:
s401, monitoring a scrawling starting event; s402, sending a doodle starting control instruction.
In the application, the graffiti is used for marking and guiding in the process of same-screen communication, and graffiti switches and operation authorities are distributed to both sides of a communicator according to business requirements. After the scrawling is started, the party who executes the scrawling operation in the third application program and the first application program regards the generated user operation event as the scrawling behavior execution and does not send the scrawling behavior execution to the second application program any more for execution. In a specific implementation scene, according to business requirements, both parties can scrawl, or one party executes scrawling and the other party executes business operation. After the second client 30 user initiates the graffiti, S402 sends a graffiti control instruction to notify the first client 10 to simultaneously initiate a graffiti operation or a graffiti drawing.
And the server 20, S403, sends a doodle starting control instruction, that is, forwards the doodle starting instruction.
In the first client 10:
s404, a Canvas for the graffiti is created or displayed. This step is performed after receiving a control command to initiate a graffiti, and the Canvas for graffiti is typically a separate component placed on top of the application.
In the second client 30:
s405, creating or displaying a Canvas for doodling; s406, collecting the doodle data to generate third synchronous data; s407, the third synchronization data is transmitted. The method comprises the steps of stopping user operation event collection, and starting scrawling data collection in a scrawling working mode, wherein the scrawling data comprises but is not limited to scrawling operation behavior data executed by a user, generated scrawling result data and the like.
In the server 20:
s408, receiving the third synchronous data to generate first synchronous data for forwarding; s409, the first synchronization data is transmitted. And forwarding the scrawling data at the server side. In the present application, the type of the synchronization data is defined based on the nature of the transmission subject, and in the specific implementation, the third synchronization data is generated as the first synchronization data, and actual data processing may not be performed.
In the first client 10:
and S410, drawing is executed in the doodle Canvas based on the doodle data in the first synchronization data. This step ultimately synchronizes the display of the results of the doodling operation occurring in the second client 30 in the first client 10.
In the same principle, a user doodling operation occurs in the first client 10, and the following steps are performed:
in the first client 10:
s411, collecting the doodle data to generate second synchronous data; and S412, sending third synchronous data.
In the server 20:
s413, receiving the second synchronization data, and generating the second synchronization data as the first synchronization data; s414, the first synchronization data is sent.
In the second client 30:
and S415, drawing is executed in a doodle Canvas based on the doodle data in the first synchronization data.
The above steps synchronously display the result of the doodling operation occurring in the first client 10 in the second client 30.
The above implementation manner realizes synchronous doodling by creating Canvas for doodling in the first client 10 and the second client 30, respectively.
Optionally, a Canvas for doodling is not created in the first client 10 and the second client 30, a Canvas for doodling is created in the server, after the doodling control instruction is received and the doodling operation mode is entered, a user operation event is collected in the first client 10 and the second client 30, after the server receives the user operation event, the user operation event is not executed in the second application program, the user operation event is executed in the Canvas for doodling, after a doodling result is generated, the part of display content is sent as the first synchronization data, and the first client 10 and the second client 30 receive and display the doodling result. Optionally, the above implementation creates a Canvas for the graffiti in the second application. Optionally, an independent application program is started in the synchronous running environment to create the Canvas for the doodle, and in the implementation scenario, the display contents of the third application program and the fourth application program are display pictures received in the synchronous running environment, and the display pictures contain the doodle contents. The technical characteristics of the implementation mode are that the doodling operation of the client side is received, and the doodling content is generated at the server side and sent to the client side for display.
In a possible implementation manner, the present application further provides a method for calling a local interface program of a first client in the application program on-screen communication, as shown in fig. 5, the steps are as follows:
after the first client 10 and the second client 30 enter the on-screen communication state, for example:
in the server 20:
s501, operating a second application program, generating instruction data for calling a local interface program when determining that the second application program calls a behavior of a local interface program of a first client, and generating the instruction data into first synchronization data; s502, transmitting the first synchronization data. Such as local authorization, payment, local input methods, etc. After the application enters the same-screen communication, the second application program is an actual service operation program and operates at the server, so that the local interface cannot be called, and the instruction data for calling the local interface program at the client locally is generated through the steps and is sent.
In the first client 10:
s503, receiving first synchronization data, calling a local interface program when the first synchronization data comprises instruction data for calling the local interface program, and acquiring and generating second synchronization data based on the execution result of the local interface program; s504, second synchronous data is sent. The first client 10 includes a local interface program, and after receiving the instruction data for invoking the local interface program, invokes the corresponding local interface program, for example, receives an instruction for invoking local payment, invokes the local payment program, generates a payment result after the payment is completed, and generates the payment result as second synchronous data to be sent.
In the server 20:
s505, receiving the second synchronous data, analyzing the local interface program execution result contained in the second synchronous data, and executing a second application program based on the local interface program execution result; s506, collecting display data to generate first synchronization data; s507, sending first synchronization data to the first client; s508, sending the first synchronization data to the second client. For example, a payment result returned by the first client is received, such as payment completion, and the second application program is continuously executed according to the result. As described above, S506, S507, and S508 are unnecessary steps in the case where the display content is not changed in S505.
Through the implementation mode, the local application program interface is called in the same-screen communication process, and the beneficial effects that the programs of the server and the client can be executed in the same-screen process, so that the whole business operation can be finished in the same-screen state, and the privacy of a user is effectively protected.
According to the same implementation principle, the implementation mode is also suitable for calling the local interface program of the second client by the server, namely, the second application program is operated at the server, and when the behavior that the second application program calls the local interface program of the second client is determined, the instruction data for calling the local interface program is generated and is generated into first synchronization data to be sent to the second client; receiving first synchronous data at a second client, calling a local interface program when the first synchronous data is determined to comprise instruction data for calling the local interface program, acquiring and generating third synchronous data based on the execution result of the local interface program, and sending the third synchronous data to a server; and the server receives the third synchronous data, analyzes the execution result of the local interface program contained in the third synchronous data, and executes the second application program based on the execution result of the local interface program. For example, the above implementation manner may be applied to a scenario in which the second client displays the display content of the second application program in a video form, when the user clicks an input frame component in the video frame at the second client, a user operation instruction is sent to the server, the second application program executes the user operation event, places the corresponding input frame component into a focus, generates and sends an instruction for invoking local input of the second client, invokes the input program and inputs text content after the second client receives the instruction, sends the input text content to the server, writes the text content in the input frame component after the server receives the text content, and then synchronizes the frame to the client to enable the client to obtain the input display effect.
In a possible implementation manner, the present application further provides a processing method when the application program is finished with the on-screen communication, as shown in fig. 6, for example, an end-on-screen request is initiated at the first client 10, and the following steps are performed:
in the first client 10:
s601, running a third application program; s602, generating a request for finishing the same screen; s603, sending a request for finishing the same screen.
In the server 20:
s604, processing the request for finishing the same screen, and acquiring second running information to generate first synchronization data based on a second application program; s605, sending an instruction to end the on-screen operation to the second client 30; s606, sending an instruction to end the same screen and first synchronization data to the first client 10. In step S604, the second operation information is collected and used to set the operation state of the first application program at the first client.
In the second client 30:
and S607, ending the same screen.
In the first client 10:
s608, closing the third application program, starting the first application program to run, and setting the running state of the first application program based on the second running information in the first synchronization data.
Through the implementation mode, after the on-screen communication is finished, the first application program is set based on the running information of the second application program, so that the business operation of the first application program is kept in a continuous state after the on-screen communication is finished. For example, the customer service staff and the client perform on-screen communication to assist the client in executing the insurance service operation, both parties actually perform the service through the second application program in the process, for example, part of form data is entered in the communication process, after the on-screen communication assistance is finished, S604 collects the part of data as second running information, and S608 sets the part of data in the first application program, so that the client continues to perform the service operation after the on-screen communication is finished, and the repeated work is avoided. Therefore, the realization method has the beneficial effects that after the same-screen communication is finished, the service operation is kept continuous, and the user experience is improved.
Based on the same inventive concept, an application program on-screen system is further provided in the embodiments of the present application, and as shown in fig. 1, the system includes a first client 10, a server 20, and a second client 30, where the first client 10 includes a first application program 101 and a third application program 102; the server 20 includes a synchronous execution environment 201, and the synchronous execution environment 201 executes a second application 2011; the second client 30 comprises a fourth application 301, wherein:
the first client 10 is configured to run the first application 101, and is further configured to generate and send a co-screen request; the system is also used for collecting and sending the user online information; the third application program 102 is further configured to receive and execute a control instruction, and start the third application program when the control instruction is determined to be an instruction to enter the same screen;
the first client 10 is further configured to run the third application 102, receive first synchronization data, and perform synchronization processing in the third application 102 based on the first synchronization data; also for collecting and sending second synchronization data based on the third application 102; the system is also used for creating a doodle Canvas, collecting doodle data and drawing based on the doodle data; the third application program 102 is also used for closing the third application program, starting the first application program 101 to run and setting the running state of the first application program 101 based on the second running information in the first synchronization data when the received control instruction is determined to be the instruction for ending the same screen;
the server 20 is configured to receive and process the on-screen request and end the on-screen request; the system is also used for receiving and sending user state information and a same-screen invitation request; the system is also used for sending notification and control instructions; the application program is further configured to configure and start the synchronous running environment 201 after processing the on-screen request, and set and run the second application program 2011; and is further configured to send a control instruction to the synchronous operation environment 201;
the server 20 is further configured to establish a long connection with the first client 10 and the second client 30; also for generating and transmitting the first synchronization data; also for acquiring display data and generating the first synchronization data based on the second application 2011; the application program 2011 is further configured to receive the second synchronization data and the third synchronization data, and send the second synchronization data and the third synchronization data to the second application program 2011 for execution when the second synchronization data and the third synchronization data are determined to be user operation data; the server is also used for forwarding the second synchronous data and the third synchronous data, wherein the forwarded second synchronous data and third synchronous data comprise one or more of audio data, video data and doodle data;
the synchronous running environment 201 is configured to run the second application 2011 based on the first running information of the first application 101 in the on-screen request; also for obtaining display data of the second application 2011;
the second application 2011 is configured to generate display data on the same screen; the data processing device is also used for executing user operation events contained in the second synchronous data or the third synchronous data;
the server 20 is further configured to, when the on-screen request is processed, acquire second running information based on the second application 2011 and send the second running information to the first client 10 as first synchronization data;
the second client 30 is further configured to run the fourth application 301, and is further configured to receive first synchronization data and perform synchronization processing in the fourth application 301 based on the first synchronization data; the device is also used for receiving and executing a control instruction, and entering on-screen communication when the control instruction is determined to be an on-screen entering instruction; the server 20 is further configured to receive user online information of the first application 101 sent by the server, select a user based on the user online information, create a co-screen invitation request, and send the co-screen invitation request; also for collecting and sending third synchronization data based on said fourth application 301; the system is also used for creating a doodle Canvas, collecting doodle data and drawing based on the doodle data;
the first synchronous data comprise one or more of display data, instruction data and operation data, and the second synchronous data and the third synchronous data comprise one or more of user operation data, audio and video data and scrawling data.
Optionally, the first client 10 further comprises a local interface program 103. The first client 10 is further configured to call the local interface program 103 according to an instruction of the local interface program, acquire an execution result, generate second synchronization data, and send the second synchronization data to the server 20. The server 20 is further configured to generate instruction data for invoking a local interface program and send the instruction data as first synchronization data to the first client 10 when it is determined that the second application program 2011 invokes a behavior of the local interface program of the first client 10; and is further configured to execute the second application 2011 based on a local interface program execution result when it is determined that the local interface program execution result is included in the second synchronization data.
Optionally, the second client 30 further comprises a local interface program. The second client 30 is further configured to call the local interface program according to an instruction of the local interface program, acquire an execution result, generate third synchronization data, and send the third synchronization data to the server 20. The server 20 is further configured to generate instruction data for invoking a local interface program and send the instruction data as first synchronization data to the second client 30 when it is determined that the second application program 2011 invokes a behavior of the local interface program of the second client 30; and is further configured to execute the second application 2011 based on a local interface program execution result when it is determined that the local interface program execution result is included in the third synchronization data.
The embodiment of the application also provides another on-screen communication method, which comprises the following steps:
in a first client:
the first client comprises a first application program, runs the first application program, receives a co-screen request initiating event triggered by a user, collects first running information based on the first application program, generates a co-screen request and sends the co-screen request to a server, or collects user online information and sends the co-screen request to the server.
In the server side:
receiving a co-screen request sent by a first client or a second client; processing the on-screen request and sending an on-screen invitation notification to another client; creating or starting a synchronous operation environment; and starting and/or setting the running of a second application program corresponding to the first application program in the synchronous running environment based on the first running information of the first application program contained in the on-screen request.
In the second client:
the second client comprises a fourth application program, the fourth application program receives a control instruction sent by the server, and when the control instruction is determined to be an on-screen instruction, the on-screen instruction is executed to enter an on-screen communication state.
In the above, it is not limited that any one of the first client and the second client initiates the on-screen request, and receives the on-screen request from the other client or the on-screen instruction sent by the server to enter the on-screen communication state. After the first client and the second client enter the same-screen communication state, one of the first application program and the second application program can be selected as a program for actually executing the service, and the other application program is taken as a snapshot of the program for actually executing the service. The snapshot is only used for synchronously displaying the operation result, and actual service is not executed. The snapshot and the actual service program are synchronized by transmitting the operation result data and the user operation data. In addition, according to the operation authority distribution condition, the operation can be carried out in both the clients. For example, taking the first application program as a program for actually executing a service, taking the second application program as a snapshot program, and if a user operation occurs at the first client, performing the following steps:
in a first client:
the method comprises the steps of collecting one or more of user operation data, program operation result data, audio and video data and doodle data, generating synchronous data and sending the synchronous data to a server side, wherein the user operation data are collected based on user operation, and the program operation result data are collected based on operation results, operation information, display data and the like of a first application program.
In the server side:
and receiving the synchronous data sent by the first client, and executing corresponding synchronous processing based on the property of the synchronous data, wherein the synchronous processing comprises generating a snapshot display picture, executing graffiti drawing, transmitting audio and video and the like, wherein the generating of the snapshot display picture comprises restoring user operation behaviors in the second application program based on the contained user operation data, and generating an operation result display picture in the second application program based on the contained program operation result data. After the snapshot is generated in the second application program, the method further comprises the steps of collecting display data based on the second application program to generate synchronous data, and sending the synchronous data to the second client, wherein the display data comprises one or more of image data, video data, snapshot data and drawing instruction data.
In the second client:
and receiving synchronous data sent by the server, and executing synchronous processing based on the synchronous data, wherein the synchronous processing comprises one or more of generating a display picture, playing audio and video and executing graffiti drawing. For example, the second application program sent by the playing server runs the picture video stream.
In the above on-screen communication execution process, for example, a user operation occurs in the second client, and the execution steps are as follows:
in the second client:
and acquiring one or more of user operation data, audio and video data and scrawling data to generate synchronous data, and sending the synchronous data to the server, wherein the user operation data is generated by acquiring a user operation event based on the fourth application program.
In the server side:
and receiving the synchronous data sent by the second client and sending the synchronous data to the first client. Optionally, the user operation data included in the synchronization request is processed to generate new synchronization data, and the new synchronization data is sent to the first client.
In a first client:
and receiving the synchronous data sent by the server, acquiring the user operation data in the synchronous data, and executing the user operation event contained in the synchronous data in the first application program. And then, continuously executing the steps of collecting the user operation data, the program operation result data and the like and sending the user operation data, the program operation result data and the like to the server.
Through the above three steps, when a user operation occurs in the second client, the operation is actually performed at the first client, and the second client can obtain a synchronous execution effect.
In the above implementation, the first application is taken as a program actually running a service, and the second application is taken as a snapshot for exemplary explanation. The implementation mode has the advantages that the first application program directly calls the local interface program, so that the private data of the user is effectively protected, and the service operation process is kept continuous before and after the same-screen starting and ending. Optionally, audio-video communication between the first client and the second client is established for transmitting audio-video data. Optionally, the second application program is used as a program for actually running the service, the first application program is implemented as a snapshot, when a user operation occurs in the first application program, user operation data is collected and sent to the server, the user operation data is actually executed in the second application program, then, the second application program sends running result data to the first application program and enables the first application program to present an execution result, and display data is sent to the fourth application program to enable the fourth application program to display a synchronization picture, so that the displays of the two client terminals are kept in synchronization. In addition, related graffiti, audio-video communication, and the like are also described in the above embodiments, and are not described again.
The present application further provides a computer-readable storage medium, in which a software program is stored, and when the software program is read and executed by one or more processors, the steps of the application program on-screen method according to any of the above embodiments can be implemented.
The embodiment of the present application further provides a computer program product, which when being read and executed by one or more processors, can implement the steps of the application co-screen method according to any one of the above embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware associated with instructions of a computer program, which can be stored in a computer-readable storage medium, and when the computer program is executed, the processes of the embodiments of the methods described above can be included. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
It should be noted that, in the above embodiment of the method of the present application, some steps may be replaced by similar steps, and some steps may be combined, decomposed, and performed in an adjusted order, so that the above embodiment does not constitute a limitation on the application program on-screen method presented in the present application.
To sum up, the application program on-screen method and system provided by the embodiment of the present application are implemented in an on-screen manner for providing on-screen display data when an application program is run on a server, and the beneficial effects that can be produced can be summarized as follows:
the client can initiate the on-screen request in the process of executing the application program operation at the client, and other customer service personnel can also select the client to initiate the on-screen request, so that the flexibility of the service is enhanced, and the method is suitable for the service scene of the Internet;
when entering the same screen, the service operation and running state executed by the current application program are not interrupted;
the same-screen communication process comprises audio and video communication and synchronous doodling, and the face-to-face communication effect is achieved;
in the process of on-screen communication, an application program is operated at a server side, a local interface program of a client side can be called, but the calling process is not on-screen, so that the business operation process is complete, and the privacy data of a user are effectively protected;
after the same screen is finished, the service operation and the running state executed by the application program are kept continuous.
It will be apparent to those skilled in the art that various changes and modifications may be made in the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if such modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to encompass such modifications and variations.

Claims (10)

1. An application program on-screen communication method is applied to a first client, wherein the first client comprises a first application program and a third application program, and the method comprises the following steps:
the first application program is operated, a same-screen request initiating event triggered by a user is received, first operation information is collected based on the first application program, a same-screen request is generated and sent to a server side, or online information of the user is collected and sent to the server side;
receiving a control instruction sent by the server, starting the third application program and entering a same-screen communication state when the control instruction is determined to be a same-screen entering instruction;
receiving first synchronous data sent by the server, and executing synchronous processing based on the first synchronous data, wherein the first synchronous data comprises one or more of display data, audio data, video data, doodle data and instruction data, and the synchronous processing comprises one or more of generating a display picture, playing audio and video, executing doodle drawing and executing instructions;
acquiring one or more of user operation data, audio/video data and graffiti data, generating second synchronous data, and sending the second synchronous data to the server, wherein the user operation data is generated based on the third application program to acquire a user operation event and is used for being executed in a second application program of the server;
the first application program is an application program for a user to execute business operation, the on-screen request is received and processed by the server, the server is provided with a second application program which has the same or similar content with the first application program, the first running information is used for starting and/or setting the second application program, and the second application program is used for providing display data for the third application program.
2. The method of claim 1, wherein after the first client enters the on-screen communication state, further comprising:
receiving a doodle starting control instruction sent by a server side, or receiving a doodle starting event triggered by a user, and creating or displaying a Canvas for doodle on the first client side;
collecting scrawling data based on the scrawling Canvas to generate second synchronous data to be sent; and executing drawing in the scribble Canvas based on the scribble data in the first synchronization data.
3. The method of claim 1, wherein receiving first synchronization data sent by the server to perform the synchronization process further comprises:
and when the first synchronization data is determined to comprise instruction data for calling a local interface program, calling the local interface program at the first client, acquiring and generating second synchronization data based on the execution result of the local interface program, and sending the second synchronization data to the server.
4. The method of claim 1, after receiving the control instruction, further comprising:
and when the control instruction is determined to be the instruction for ending the same screen, closing the third application program, starting the operation of the first application program, and setting the operation state of the first application program based on the second operation information of the second application program in the first synchronization data.
5. An application on-screen communication method is applied to a second client, and the second client comprises a fourth application program, and the method comprises the following steps:
the fourth application program receives a control instruction sent by the server, and executes the instruction for entering the same screen to enter the same screen communication state when the control instruction is determined to be the instruction for entering the same screen;
receiving first synchronization data sent by the server, and executing synchronization processing based on the first synchronization data, wherein the first synchronization data comprises one or more of display data, audio data, video data and graffiti data, and the synchronization processing comprises one or more of generating a display picture, playing audio and video and executing graffiti drawing;
acquiring one or more of user operation data, audio/video data and graffiti data, generating third synchronous data, and sending the third synchronous data to the server, wherein the user operation data is generated by acquiring a user operation event based on the fourth application program and is used for being executed in a second application program of the server;
also comprises the following steps of (1) preparing,
receiving user online information of a first application program sent by a server, selecting a user based on the user online information, creating a same-screen invitation request, and sending the same-screen invitation request to the server.
6. The method of claim 5, wherein after the second client enters the on-screen communication state, further comprising:
receiving a doodle starting control instruction sent by a server side, or receiving a doodle starting event triggered by a user, and creating or displaying a Canvas for doodle on the second client side;
collecting scrawling data based on the scrawling Canvas to generate third synchronous data for sending; and executing drawing in the scribble Canvas based on the scribble data in the first synchronization data.
7. The method for realizing the same-screen communication of the application program is applied to a server side, and comprises the following steps:
receiving a co-screen request sent by a first client or a second client;
processing the on-screen request and sending an on-screen invitation notification to another client;
creating or starting a synchronous operation environment;
starting and/or setting a second application program corresponding to the first application program to run in the synchronous running environment based on first running information of the first application program contained in the on-screen request;
acquiring display data based on the second application program to generate first synchronization data, and sending the first synchronization data to the first client and the second client, wherein the display data comprises one of image data, video data, snapshot data and drawing instruction data; receiving second synchronous data sent by the first client or third synchronous data sent by the second client, acquiring user operation data in the second synchronous data, and executing user operation events contained in the second synchronous data in the second application program;
the second application program is used for generating synchronous display data for the first client side and the second client side, starting and/or setting the second application program based on first running information of the first application program contained in the on-screen request, and enabling the second application program to obtain the same or similar running state with the first application program.
8. The method of claim 7, wherein running the second application in the synchronous execution environment further comprises:
when determining that the second application program calls the behavior of the first client local interface program, generating instruction data for calling the local interface program, generating the instruction data as first synchronization data and sending the first synchronization data to the first client;
the method also comprises the steps of receiving the second synchronous data sent by the first client, analyzing the execution result of the local interface program contained in the second synchronous data, and executing the second application program based on the execution result of the local interface program.
9. The method of claim 7, further comprising:
receiving a request for ending the same screen sent by the first client or the second client, processing the request for ending the same screen, sending an instruction for ending the same screen to the first client or the second client, acquiring second running information based on the second application program, generating first synchronous data, and sending the first synchronous data to the first client, wherein the second running information is used for setting the running state of the first application program at the first client.
10. An application program co-screen system comprises a first client, a server and a second client, wherein the first client comprises a first application program and a third application program; the server comprises a synchronous operation environment, and the synchronous operation environment operates a second application program; the second client includes a fourth application, wherein:
the first client is used for running the first application program and generating and sending a same-screen request; the system is also used for collecting and sending the user online information; the third application program is also used for receiving and executing a control instruction, and starting the third application program when the control instruction is determined to be a command for entering the same screen;
the first client is further used for running the third application program, receiving first synchronization data and executing synchronization processing in the third application program based on the first synchronization data; the third application program is also used for acquiring and sending second synchronous data based on the third application program; the system is also used for creating a doodle Canvas, collecting doodle data and drawing based on the doodle data; the server is also used for calling the local interface program according to the instruction of the local interface program, acquiring an execution result, generating second synchronous data and sending the second synchronous data to the server; the third application program is closed, the first application program is started to run, and the running state of the first application program is set based on second running information in the first synchronization data when the received control instruction is determined to be the instruction for finishing the same screen;
the server is used for receiving and processing the same-screen request and ending the same-screen request; the system is also used for receiving and sending user state information and a same-screen invitation request; the system is also used for sending notification and control instructions; the system is also used for configuring and starting a synchronous operation environment after processing the same-screen request, and setting and operating the second application program; the synchronous operation environment is also used for sending a control instruction to the synchronous operation environment;
the server is also used for establishing long connection with the first client and the second client; also for generating and transmitting the first synchronization data; further configured to collect display data based on the second application and generate the first synchronization data; the second synchronous data and the third synchronous data are also used for receiving and sending the second synchronous data and the third synchronous data to the second application program for execution when the second synchronous data and the third synchronous data are determined to be user operation data; the server is also used for forwarding the second synchronous data and the third synchronous data, wherein the forwarded second synchronous data and third synchronous data comprise one or more of audio data, video data and doodle data;
the synchronous operation environment is used for operating the second application program based on the first operation information of the first application program in the on-screen request; the second application program display data is also acquired;
the second application program is used for generating display data of the same screen; the data processing device is also used for executing user operation events contained in the second synchronous data or the third synchronous data;
the server is further used for generating instruction data for calling a local interface program and sending the instruction data serving as first synchronization data to the first client when determining that the second application program calls the behavior of the local interface program of the first client; the second application program is further used for executing the second application program based on the execution result of the local interface program when the second synchronous data is determined to contain the execution result of the local interface program; the second client is also used for generating instruction data for calling the local interface program and sending the instruction data serving as first synchronization data to the second client when the behavior that the second application program calls the local interface program of the second client is determined; the second application program is further used for executing the second application program based on the execution result of the local interface program when the third synchronous data is determined to contain the execution result of the local interface program; the first client is also used for acquiring second running information based on the second application program and sending the second running information as first synchronization data to the first client when the on-screen request is processed;
the second client is further used for running the fourth application program, receiving first synchronization data and executing synchronization processing in the fourth application program based on the first synchronization data; the device is also used for receiving and executing a control instruction, and entering on-screen communication when the control instruction is determined to be an on-screen entering instruction; the server is also used for receiving user online information of the first application program sent by the server, selecting a user based on the user online information, creating a same-screen invitation request and sending the same; the system is also used for acquiring and sending third synchronous data based on the fourth application program; the system is also used for creating a doodle Canvas, collecting doodle data and drawing based on the doodle data; the server is also used for calling the local interface program according to the instruction of the local interface program, acquiring an execution result, generating third synchronous data and sending the third synchronous data to the server;
the first synchronous data comprise one or more of display data, instruction data and operation data, and the second synchronous data and the third synchronous data comprise one or more of user operation data, audio and video data and scrawling data.
CN202210437002.1A 2022-04-27 2022-04-27 Application program on-screen communication method and system Active CN114884914B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210437002.1A CN114884914B (en) 2022-04-27 2022-04-27 Application program on-screen communication method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210437002.1A CN114884914B (en) 2022-04-27 2022-04-27 Application program on-screen communication method and system

Publications (2)

Publication Number Publication Date
CN114884914A true CN114884914A (en) 2022-08-09
CN114884914B CN114884914B (en) 2024-04-02

Family

ID=82670678

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210437002.1A Active CN114884914B (en) 2022-04-27 2022-04-27 Application program on-screen communication method and system

Country Status (1)

Country Link
CN (1) CN114884914B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115033335A (en) * 2022-08-11 2022-09-09 青岛中正合力软件有限公司 Display processing method applied to same screen of webpage

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104794008A (en) * 2015-04-28 2015-07-22 北京数字天域科技有限责任公司 Application program synchronizing method, device and system
CN111107145A (en) * 2019-12-17 2020-05-05 青岛希望鸟科技有限公司 Online synchronous communication system based on HTML5 webpage
CN111581564A (en) * 2020-05-10 2020-08-25 青岛希望鸟科技有限公司 Webpage synchronous communication method realized by applying Canvas
US20200342003A1 (en) * 2016-09-27 2020-10-29 Opro Co., Ltd. Synchronization program and connection program for cloud service
CN113868573A (en) * 2021-09-07 2021-12-31 青岛希望鸟科技有限公司 Method and system for quickly establishing one-screen interaction based on webpage
CN114020396A (en) * 2021-11-18 2022-02-08 湖南麒麟信安科技股份有限公司 Display method of application program and data generation method of application program

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104794008A (en) * 2015-04-28 2015-07-22 北京数字天域科技有限责任公司 Application program synchronizing method, device and system
US20200342003A1 (en) * 2016-09-27 2020-10-29 Opro Co., Ltd. Synchronization program and connection program for cloud service
CN111107145A (en) * 2019-12-17 2020-05-05 青岛希望鸟科技有限公司 Online synchronous communication system based on HTML5 webpage
CN111581564A (en) * 2020-05-10 2020-08-25 青岛希望鸟科技有限公司 Webpage synchronous communication method realized by applying Canvas
CN113868573A (en) * 2021-09-07 2021-12-31 青岛希望鸟科技有限公司 Method and system for quickly establishing one-screen interaction based on webpage
CN114020396A (en) * 2021-11-18 2022-02-08 湖南麒麟信安科技股份有限公司 Display method of application program and data generation method of application program

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115033335A (en) * 2022-08-11 2022-09-09 青岛中正合力软件有限公司 Display processing method applied to same screen of webpage
CN115033335B (en) * 2022-08-11 2023-01-10 青岛中正合力软件有限公司 Display processing method applied to same screen of webpage

Also Published As

Publication number Publication date
CN114884914B (en) 2024-04-02

Similar Documents

Publication Publication Date Title
US11785056B2 (en) Web browser interface for spatial communication environments
US11088971B2 (en) Virtual area communications
US9009603B2 (en) Web browser interface for spatial communication environments
US7599989B2 (en) System and method for gathering and reporting screen resolutions of attendees of a collaboration session
US9973829B2 (en) Method for video communications and terminal, server and system for video communications
US20060092268A1 (en) Teleconference method with a shared working space
CN111639503B (en) Conference data processing method and device, storage medium and equipment
CN101484886A (en) Methods and apparatuses for dynamically sharing a portion of a display for application based screen sampling
CN111596985A (en) Interface display method, device, terminal and medium in multimedia conference scene
US20060195520A1 (en) System and method for providing one class of users of an application a view of what another class of users of the application is visually experiencing
CN112306607A (en) Screenshot method and device, electronic equipment and readable storage medium
JP2018518769A (en) Method and apparatus for transmitting business objects
CN112437004A (en) Information processing method, information processing device, electronic equipment and storage medium
CN111880695A (en) Screen sharing method, device, equipment and storage medium
US20140006915A1 (en) Webpage browsing synchronization in a real time collaboration session field
CN113778583A (en) Method, device, equipment and medium for publishing local application of cloud desktop
CN113222549A (en) Information interaction method, device and system and electronic equipment
CN114884914B (en) Application program on-screen communication method and system
WO2022111193A1 (en) Meeting control method, apparatus, device, and medium
CN112751681B (en) Image processing method, device, equipment and computer readable storage medium
KR100989122B1 (en) Service method for online virtual cooperative work using remote control
CN108900794B (en) Method and apparatus for teleconferencing
Anumba et al. Telepresence in concurrent lifecycle design and construction
KR20230062350A (en) Method and apparatus for displaying state information, device and storage medium
CN113992882A (en) Packet processing method and device for multi-person conversation, electronic device and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant