CN114884914B - Application program on-screen communication method and system - Google Patents

Application program on-screen communication method and system Download PDF

Info

Publication number
CN114884914B
CN114884914B CN202210437002.1A CN202210437002A CN114884914B CN 114884914 B CN114884914 B CN 114884914B CN 202210437002 A CN202210437002 A CN 202210437002A CN 114884914 B CN114884914 B CN 114884914B
Authority
CN
China
Prior art keywords
data
application program
client
screen
synchronous
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210437002.1A
Other languages
Chinese (zh)
Other versions
CN114884914A (en
Inventor
路九阳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qingdao Hope Bird Technology Co ltd
Original Assignee
Qingdao Hope Bird Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qingdao Hope Bird Technology Co ltd filed Critical Qingdao Hope Bird Technology Co ltd
Priority to CN202210437002.1A priority Critical patent/CN114884914B/en
Publication of CN114884914A publication Critical patent/CN114884914A/en
Application granted granted Critical
Publication of CN114884914B publication Critical patent/CN114884914B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/04—Real-time or near real-time messaging, e.g. instant messaging [IM]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/02—Standardisation; Integration
    • H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
    • H04L41/0273—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using web services for network management, e.g. simple object access protocol [SOAP]
    • H04L41/028—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using web services for network management, e.g. simple object access protocol [SOAP] for synchronisation between service call and response
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/04—Real-time or near real-time messaging, e.g. instant messaging [IM]
    • H04L51/046—Interoperability with other network applications or services
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/07—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail characterised by the inclusion of specific contents
    • H04L51/10—Multimedia information

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The application provides a method and a system for on-screen communication of application programs, wherein the method comprises the following steps: the method comprises the steps that a first application program is operated on a first client, a user initiates a same-screen request, a server receives the same-screen request, a second application program corresponding to the first application program is operated in a synchronous operation environment, display data are generated based on the second application program and sent to the first client and the second client, the first client operates a third application program to display same-screen data, and the second client operates a fourth application program to display same-screen data. And acquiring user operation data based on the third application program or the fourth application program, sending the user operation data to the server, and actually executing user operation in the second application program of the server. Therefore, the method for initiating the same screen in the process of executing the business operation by the client is provided for the same-screen implementation mode for generating the synchronous display data based on the server, so that the client and the customer service personnel can initiate the same-screen request, the service use scene is widened, and the flexibility is high.

Description

Application program on-screen communication method and system
Technical Field
The invention relates to the field of internet communication, in particular to a method and a system for on-screen communication of application programs.
Background
People transact business through the internet, perform operations and the like, and often need to communicate with another party user to perform explanation and assist operations. For example, the user a accesses the cloud virtual device, performs an operation in the virtual device, and simultaneously sends a web link corresponding to a screen of the acquirable virtual device to the user B, and the user B accesses the web link to acquire the virtual device screen, thereby meeting the requirement of synchronous browsing of the user a and the user B. However, this technique is limited in that user a needs to first access the cloud virtual device and then generate a link based on the virtual device that can access the device screen to send to user B, which cannot initiate an on-screen communication request, and thus is not suitable for a conventional internet service scenario, i.e., cannot be initiated by a user browsing a service web page. As above, customer a is typically a customer service person, while customer B is typically a customer, which provides a way for customer service to communicate with contacted customers on the same screen, but for customers from the network, seeking assistance in customer service when experiencing problems, the same screen technique is not practical.
Therefore, a new method for implementing the application program on the same screen is needed to solve the above problems, so that both customer service and clients can initiate a request to invite the other party to enter the on-screen communication.
Disclosure of Invention
Aiming at the problems, the invention provides a method and a system for on-screen communication of an application program, which aim to provide a method for initiating on-screen communication from a service operation terminal for generating an on-screen implementation mode of synchronous display data from a service end so as to enhance the flexibility of on-screen application, enable a customer to acquire on-screen assistance of customer service personnel in real time while executing service operation, and improve communication efficiency. The technical scheme provided by the invention is as follows:
in a first aspect, the present application provides an on-screen communication method of an application program, where the method is applied to a first client, and the first client includes a first application program and a third application program, and the method includes:
running a first application program, receiving an initiation on-screen request event triggered by a user, and acquiring first running information based on the first application program, generating an on-screen request and sending the same to a server, or acquiring online information of the user and sending the same to the server;
receiving a control instruction sent by a server, and starting a third application program to enter a same-screen communication state when the control instruction is determined to be an on-screen entering instruction;
the third application program establishes communication connection with the server, receives first synchronous data sent by the server, and executes synchronous processing based on the first synchronous data, wherein the first synchronous data comprises one or more of display data, audio data, video data, graffiti data and instruction data, and the synchronous processing comprises one or more of generating a display picture, playing audio and video, executing graffiti drawing and executing instructions;
And collecting one or more of user operation data, audio and video data and doodling data, generating second synchronous data, and sending the second synchronous data to the server, wherein the user operation data is generated based on a third application program collecting user operation event and is used for being executed in a second application program of the server.
In the implementation manner, the first application program is an application program for executing business operation for a user, the on-screen request is received by the server, the second application program with the same content as the first application program is set and operated at the server, the first operation information is used for starting and/or setting the second application program, and the second application program is used for providing display data for the third application program.
The realization method generates synchronous display data at the server and sends the client to realize the same screen, thereby realizing a mode of initiating the same screen in the process of executing the service by the client and enhancing the flexibility of the service.
In one possible implementation manner, after the first client enters the on-screen communication state, receiving a start-up graffiti control instruction sent by the server or receiving a start-up graffiti event triggered by a user, and creating or displaying a Canvas for graffiti on the first client; and/or based on the graffiti Canvas, collecting the graffiti data to generate second synchronous data and sending the second synchronous data; drawing is performed in the graffiti Canvas based on the graffiti data in the first synchronization data. The implementation enables synchronous graffiti in a third application.
In one possible implementation manner, the first synchronization data sent by the server side is received to execute synchronization processing, when it is determined that the first synchronization data includes instruction data for calling the local interface program, the local interface program is called at the first client side, and based on an execution result of the local interface program, second synchronization data is acquired and generated, and sent to the server side. The implementation method solves the problem of calling the local interface program of the first client in the on-screen process.
In one possible implementation manner, after receiving the control instruction, when the control instruction is determined to be an end on-screen instruction, closing the third application program, starting the first application program to run, and setting the running state of the first application program based on the second running information of the second application program in the first synchronous data. After the same screen is finished, the implementation mode resumes the operation of the first application program based on the result of the business operation of the same screen process, so that the business operation process is kept continuous.
In a second aspect, the present application provides an on-screen communication method of an application program, where the method is applied to a second client, and the second client includes a fourth application program, and the method includes:
the fourth application program receives a control instruction sent by the server, and executes the same-screen entering instruction to enter a same-screen communication state when the control instruction is determined to be the same-screen entering instruction;
Receiving first synchronous data sent by a server side, and executing synchronous processing based on the first synchronous data, wherein the first synchronous data comprises one or more of display data, audio data, video data and graffiti data, and the synchronous processing comprises one or more of generating a display picture, playing audio and video and executing graffiti drawing;
and collecting one or more of user operation data, audio and video data and doodling data, generating third synchronous data, and sending the third synchronous data to the server, wherein the user operation data is generated based on a fourth application program collecting user operation event and is used for being executed in a second application program of the server.
In the above implementation manner, the second client presents the display content of the second application program by running the fourth application program, so as to realize the on-screen browsing with the first client, and the method for executing the on-screen operation in the fourth application program is further provided.
In one possible implementation, user online information of a first application program sent by a server is received, a user is selected based on the user online information, and a screen invitation request is created and sent to the server. The implementation provides a method for initiating a request on screen at a second client.
In one possible implementation manner, after the second client enters the on-screen communication state, receiving a start-up graffiti control instruction sent by the server or receiving a start-up graffiti event triggered by a user, and creating or displaying a Canvas for graffiti on the second client; based on the graffiti Canvas, collecting graffiti data to generate third synchronous data and sending the third synchronous data; drawing is performed in the graffiti Canvas based on the graffiti data in the first synchronization data. The implementation enables synchronous graffiti in a fourth application.
In a third aspect, the present application provides a method for implementing on-screen communication of an application program, where the method is applied to a server, and the method includes:
receiving a request of the same screen sent by a first client or a second client;
processing the same-screen request and sending a same-screen invitation notification to another client;
creating or starting a synchronous operation environment;
starting and/or setting a second application program operation corresponding to the first application program in the synchronous operation environment based on first operation information of the first application program contained in the on-screen request;
establishing communication connection between the server and the third application program, and between the server and the fourth application program;
generating first synchronous data based on the display data acquired by the second application program, and sending the first synchronous data to the first client and the second client, wherein the display data comprises one of image data, video data, snapshot data and drawing instruction data; receiving second synchronous data sent by a first client or third synchronous data sent by the second client, acquiring user operation data in the second synchronous data, and executing user operation events contained in the second application program;
In the above implementation manner, the second application program is configured to generate synchronous display data for the first client and the second client, and start and/or set the second application program based on the first running information of the first application program included in the on-screen request, so that the second application program obtains a running state identical to or similar to that of the first application program.
According to the implementation mode, according to the running state of the first application program, a standard execution environment of the same screen is established at the server side, the first client side and the second client side are synchronous by taking the running picture of the second application program as a standard, and meanwhile, the operation event from the client side is received to be executed in the second application program, so that the effect of operating the application program by the client side is achieved.
In one possible implementation manner, the second application program is operated in the synchronous operation environment, and when the second application program is determined to call the behavior of the local interface program of the first client, instruction data for calling the local interface program is generated and is generated into first synchronous data to be sent to the first client, so that the problem of calling the local interface program of the first client in the on-screen process is solved. Optionally, the second synchronous data sent by the first client is received, the execution result of the local interface program contained in the second synchronous data is analyzed, and the second application program is executed based on the execution result of the local interface program.
In one possible implementation manner, the second application program is operated in the synchronous operation environment, and when the action of calling the local interface program of the second client for the second application program is determined, instruction data for calling the local interface program is generated and is generated into first synchronous data to be sent to the second client, so that the problem of calling the local interface program of the second client in the on-screen process is solved. Optionally, the third synchronous data sent by the second client is received, the execution result of the local interface program contained in the third synchronous data is analyzed, and the second application program is executed based on the execution result of the local interface program.
In one possible implementation manner, a request of ending the same screen sent by the first client or the second client is received, the request of ending the same screen is processed, an instruction of ending the same screen is sent to the first client or the second client, and based on the second application program, second operation information is collected to generate first synchronous data and send the first synchronous data to the first client, wherein the second operation information is used for setting an operation state of the first application program at the first client, so that after ending the same screen, a business operation process of the first client is kept continuous.
In a fourth aspect, the application program on-screen system further provides the system, the system comprises a first client, a server and a second client, and the first client comprises a first application program and a third application program; the server side comprises a synchronous operation environment, and the synchronous operation environment operates the second application program; the second client includes a fourth application, wherein:
The first client is used for running a first application program and generating and sending a same screen request; the system is also used for collecting and sending user online information; the control instruction is also used for receiving and executing the control instruction, and starting a third application program when the control instruction is determined to be an on-screen entering instruction;
the first client is also used for running a third application program, receiving the first synchronization data and executing synchronization processing in the third application program based on the first synchronization data; the system is also used for acquiring and transmitting second synchronous data based on a third application program; the method comprises the steps of creating a graffiti Canvas, collecting graffiti data, and executing drawing based on the graffiti data; the system is also used for calling the local interface program according to the received instruction of the local interface program, collecting the execution result to generate second synchronous data and sending the second synchronous data to the server; the control method is also used for closing a third application program when the received control instruction is determined to be an end on-screen instruction, starting the first application program to operate, and setting the operating state of the first application program based on the second operating information in the first synchronous data;
the server is used for receiving and processing the same-screen request and ending the same-screen request; the method is also used for receiving and sending user state information and a same screen invitation request; the method is also used for sending a notification and control instruction; the system is also used for configuring and starting a synchronous operation environment after processing the same-screen request and setting and operating a second application program; the control instruction is also used for sending a control instruction to the synchronous operation environment;
The server is also used for establishing long connection with the first client and the second client; and is further configured to generate and transmit first synchronization data; the display data is also used for acquiring display data and generating first synchronous data based on the second application program; the system is also used for receiving second synchronous data and third synchronous data, and transmitting the second synchronous data and the third synchronous data to a second application program for execution when the second synchronous data and the third synchronous data are determined to be user operation data; the system is also used for forwarding second synchronous data and third synchronous data, wherein the forwarded second synchronous data and third synchronous data comprise one or more of audio data, video data and graffiti data;
the synchronous operation environment is used for operating the second application program based on the first operation information of the first application program in the same screen request; the method is also used for acquiring second application program display data;
the second application program is used for generating display data of the same screen; and is further configured to perform a user operation event included in the second synchronization data or the third synchronization data;
the server side is further used for generating instruction data for calling the local interface program and sending the instruction data to the first client side as first synchronous data when determining that the second application program calls the local interface program of the first client side; and executing the second application program based on the local interface program execution result when the second synchronous data is determined to contain the local interface program execution result; the method is also used for generating instruction data for calling the local interface program and sending the instruction data to the second client as first synchronous data when determining that the second application program calls the second client local interface program; and executing a second application program based on the local interface program execution result when the third synchronous data is determined to contain the local interface program execution result; the method is also used for acquiring second running information based on a second application program and sending the second running information to the first client as first synchronous data when the request of the same screen is processed;
The second client is also used for running a fourth application program, receiving the first synchronization data and executing synchronization processing in the fourth application program based on the first synchronization data; the control instruction is also used for receiving and executing the control instruction, and when the control instruction is determined to be an on-screen entering instruction, the on-screen communication is entered; the server is also used for receiving the user online information of the first application program sent by the server, selecting a user based on the user online information, creating a same-screen invitation request and sending the same-screen invitation request; the system is also used for acquiring and transmitting third synchronous data based on a fourth application program; the method comprises the steps of creating a graffiti Canvas, collecting graffiti data, and executing drawing based on the graffiti data; the system is also used for calling the local interface program according to the received instruction of the local interface program, collecting the execution result to generate third synchronous data and sending the third synchronous data to the server;
the first synchronous data comprises one or more of display data, instruction data and operation data, and the second synchronous data and the third synchronous data comprise one or more of user operation data, audio and video data and graffiti data.
In a fifth aspect, embodiments of the present application further provide a computer storage medium having stored therein a software program that, when read and executed by one or more processors, implements a method provided by any one of the embodiments of the above aspects.
In a sixth aspect, there is also provided in embodiments of the present application a computer program product which, when read and executed by one or more processors, performs the method provided by any one of the embodiments of any one of the above aspects.
Therefore, the method for initiating the on-screen request at the client for executing the business operation is realized on the basis of the on-screen implementation mode of generating the synchronous display data at the server, in practical application, the first client is usually used by a client, the second client is usually used by customer service personnel, and through the above aspects, the method for initiating the on-screen request to the customer service personnel and entering the on-screen communication when the client executes the business operation is realized, so that the efficient business operation auxiliary support is obtained, and on the other hand, the customer service personnel can initiate the on-screen request according to the on-line client, and execute the on-screen to assist the customer operation. Therefore, the application program on-screen method provided by the application program on-screen method can be initiated by any party aiming at the real-time scene of the operation service, and has strong flexibility and wide applicable service scene.
Drawings
Fig. 1 is a schematic structural diagram of an application on-screen communication system according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of on-screen communication of an application according to an embodiment of the present application;
fig. 3 is a schematic flow chart of initiating on-screen communication of an application program at a second client according to an embodiment of the present application;
FIG. 4 is a schematic flow chart of graffiti processing in on-screen communication of an application according to an embodiment of the present application;
fig. 5 is a schematic flow chart of calling a first client local interface program in on-screen communication of an application program according to an embodiment of the present application;
fig. 6 is a schematic diagram of an end on-screen processing flow of on-screen communication of an application according to an embodiment of the present application.
Detailed Description
The detailed description of the present application is merely for purposes of explanation and is not intended to limit the present application.
In the following, embodiments of the present application are described, it being apparent that the described embodiments are only some, but not all, of the embodiments of the present application. As one of ordinary skill in the art can appreciate, with the development of technology and the appearance of new scenes, the technical solutions provided in the embodiments of the present application are applicable to similar technical problems.
In the existing on-screen technology, an application program is run on a server side, and display data generated by running is sent to a client side to obtain an on-screen effect. For example, running an APP application in a virtual mobile phone device environment of a server, converting a display screen of the virtual device into a video stream, sending the video stream to two mobile phone terminals, and receiving the video stream in a client application program of each mobile phone terminal, thereby achieving the effect that two mobile phone terminal users communicate with each other on the same screen. However, the above on-screen implementation technology greatly limits the service usage scenario, typically, a customer service person starts the operation of the server-side virtual device through a client program, obtains an operation picture and controls the APP operation on the virtual device, and simultaneously generates a web page link, and sends the web page link to a customer, and the customer accesses the link to load a web page, and obtains the APP picture on the designated virtual device through the web page, thereby realizing on-screen browsing of the customer service person and the customer. By analyzing the implementation manner, the client can only access the link sent by the customer service personnel to enter the same screen, namely, the service mode of first contact and then the same screen is adopted, the application scene is very narrow, and the scene of actively carrying out service operation by the client cannot initiate the communication with the screen in real time, so that the method is not suitable for the internet service operation communication requirement.
The aim of the application is to: aiming at the on-screen implementation mode of generating on-screen reference display data at a service end, the method for initiating on-screen communication by a client in the process of executing business operation is provided, so that communication can be carried out according to the current operation content and help support can be obtained.
First, some nouns of the present application are briefly described for understanding by those skilled in the art.
1) A client, or referred to as a client, is a locally running application provided for a user, and typically the client refers to a client corresponding to a server, where the client of the present application may include multiple applications.
2) The same screen means that the same or similar browsing and executing effects are realized for a certain program between two clients.
3) An application refers to a computer program that performs one or more specific tasks, and the application is not limited to a particular form thereof, such as: web pages, APP, etc. In addition, in this application, the terms "first application", "second application", "third application", "fourth application", and the like all refer to a logical division according to the functions they contain. For example, in the case where a customer performs an application operation and customer service assists, a "first application" is a program in which the customer performs the application operation from the internet; the second application program is a first application program operated according to a client, and the operation effect of the second application program is approximately the same as that of the first application program in a service end operation environment, and the second application program is used for generating reference display data of the same screen and is used as an actual execution program of business operation in the communication process; the third application program is an application program running on one side of a client after entering the same screen, and receives reference display data of the same screen to obtain the same screen effect; the fourth application is an application running on the customer service side, receives the reference display data of the same screen, and can generally send an operation instruction to operate the second application of the server side for execution.
4) The server side synchronous operation environment is used for operating a second application program, such as: virtual devices, cloud handsets, browsers, headless browsers, etc.
5) Display data, which refers to data collected based on the second application program and displayed in the third application program and the fourth application program, such as images, video streams, snapshot data (such as: web DOM data), drawing instructions, and the like.
6) And/or, the association relationship describing the association object, the representation may have three relationships, e.g., a and/or B, may represent: the character "/" generally indicates that the associated object is an "or" relationship, where a alone exists, a and B together, and B alone.
The following describes the technical solution of the present application in detail using several specific examples.
Fig. 1 shows a possible on-screen communication system structure of an application program, which includes a first client 10, a server 20, and a second client 30, wherein the first client 10 includes a first application program 101 and a third application program 102, the server 20 includes a synchronous running environment 201, the synchronous running environment 201 runs a second application program 2011, and the second client 30 includes a fourth application program 301.
The method for on-screen communication of an application program provided in the embodiment of the present application may be applied to the above-mentioned on-screen communication system structure of an application program shown in fig. 1, and referring to fig. 2, a specific flow of the method includes:
In the first client 10:
s201, a first application program is operated; s202, receiving an initiation on-screen request event triggered by a user, and acquiring first running information based on a first application program to generate an on-screen request; s203, sending a request of the same screen to the server 20. The first application program is a program for a user to perform business operations, such as an application program accessed by an internet user and provided by an insurance company. Optionally, on-line customer service personnel information is displayed in the first application program, and S202 selects customer service personnel to initiate a same screen request. The operation information collected in S202 includes, but is not limited to, one or more of a page address, program data, an operation environment parameter, a UI focus, a browsing parameter, etc., for example, the first application is a web page, and the collected information is, for example, a current web URL, inputted or operated page data, browser version information, a current focus control, a cursor position, a current web sliding position, a display area, etc., and is used to set a second application operation on the server side.
In the server 20:
receiving a request of the same screen sent by a first client 10; s204, processing the same screen request; s205, a one-screen invitation is sent to the second client 30.
In the second client 30:
running a fourth application program, and receiving the on-screen invitation sent by the server 20; s206, processing the same-screen invitation and generating a notification; according to the processing result, S207, a notification is sent.
In the server 20:
after receiving the on-screen invitation notification sent by the second client 30, it indicates that both clients agree to start the on-screen, S208, create or start a synchronous running environment; s209, sending an instruction of entering the same screen to the first client 10; s210, sending an on-screen entering instruction to the second client 30. The synchronous running environment is used for running a second application program, such as a virtual device, a browser and the like. The execution time of S209 and S210 is not particularly limited, and may be executed after the second application is started in S213, for example.
In the second client 30:
after receiving the instruction of entering the same screen sent by the server 20, S211, a long connection with the server is established, and the same screen state is entered. Optionally, communication between the fourth application and the synchronous running environment is established.
In the first client 10:
after receiving the on-screen entering instruction sent by the server 20, S212, the third application program is started, and long connection with the server is established, so that the on-screen entering state is achieved. Optionally, communication between the third application and the synchronous running environment is established.
Optionally, in S211 and S212 above, further including establishing audio-video communication between the first client 20 and the second client 30.
In the server 20:
s213, starting and/or setting a second application program to run; s214, collecting display data and generating first synchronous data; s215 transmitting the first synchronization data to the second client 30; s216, the first synchronization data is sent to the first client 10. The second application program is a program with the same or similar function as the first application program, and operates in an environment appointed by a server side so as to generate on-screen reference display data, the second application program is set to operate in S213, namely, the first operation information acquired in S202 is used for setting the second application program, and the main purpose of the second application program is to obtain the same or similar operation state as the first application program, for example, a client executes an application operation through the first application program, inputs part of form data, carries a page address, the part of form data and the like in the process of initiating an on-screen request, and sets the second application program as the first operation information so as to enable the second application program to load corresponding pages and write form data input by a user, thereby obtaining the on-screen effect of uninterrupted service operation. Optionally, the second application is the same program as the first application. Optionally, the second application program and the first application program are programs with similar functions. In S214, the display data is, for example, an application screenshot, a virtual device display screenshot, a video stream, a sequence image, snapshot data (such as web page DOM data), a drawing instruction, or the like. Optionally, the communication is established between the second application program and the third application program to directly transmit the synchronous data, and the communication is established between the second application program and the fourth application program to directly transmit the synchronous data.
In the first client 10:
s217, receiving the first synchronization data and executing the synchronization process.
In the second client 30:
s218, receiving the first synchronization data and executing the synchronization processing.
In the above, in S217 and S218, the first synchronization data includes, but is not limited to, one or more of video data, audio data, display data, instruction data, and graffiti data, and according to the nature of the first synchronization data, the synchronization process includes displaying a picture, playing an audio/video, executing an instruction, executing graffiti drawing, and the like. For example, the display data in the first synchronization data is a video stream, and the video stream is played in S218 to display a running picture of the second application.
For example, a user operation occurs in the second client 30, and the following steps are performed:
in the second client 30:
s219, collecting user operation events to generate third synchronous data; s220, sending the third synchronous data to the server 20. For example, on the basis of playing the video stream to display the running screen of the second application in S218 above, S219 listens to acquire the user operation event, acquires the operation behavior of the user on the video playing window, and generates the third synchronization data, such as acquiring the position coordinate information clicked by the user, the input data, and the like. Optionally, S219 sends the collected user operation event to the server 20, the server generates, according to the execution result of the user operation event in the second application program 30, return information, sends the return information to the second client, then the second client receives the return information and performs corresponding processing, for example, the user clicks a certain input part in a video screen of the fourth application program, after sending the clicked position information to the server, executes a user click event corresponding to the input part in the second application program, the generated execution result is "input in an input control", generates, based on the execution result, the return information and sends the return information to the second client, the second client receives and processes the return information, invokes an input method to receive input content, then monitors the input event executed by the user, continues to perform S219, and displays the input content in the input part of the second application program.
Similarly, the above related processing in step S219 may also be performed in the first client 10, and based on the user operation occurring in the first client 10, the second synchronization data generated by collecting the user operation event is sent to the server 20.
In the server 20:
s221, receiving third synchronous data sent by the second client 30, acquiring user operation data therein, and executing the third synchronous data in the second application program; s222, collecting display data and generating first synchronous data; s223, sending the first synchronous data to the second client 30; s224, the first synchronization data is sent to the first client 10.
In the first client 10:
s225, the first synchronization data sent by the server 20 is received, and the synchronization process is executed. This step is the same as the step S217, and will not be described here again.
In the second client 30:
s226, the first synchronization data sent by the server 20 is received, and the synchronization process is performed. This step is the same as the step S218, and will not be described here again.
The step S214 and the step S222 collect the display data of the second application program to generate the first synchronization data, and optionally, when the second application program display content is monitored to change, the steps of collecting the display data, generating the first synchronization data and sending are executed.
The term "synchronization data" in the present application is understood to mean data transmitted for achieving display content synchronization, audio synchronization, video synchronization, and graffiti synchronization between the first client and the second client, and data transmitted for achieving display content synchronization between the first client, the second client, and the second application on the server. The method comprises the steps of defining 'synchronous data' generated at the server side as first synchronous data, defining 'synchronous data' generated at the first client side as second synchronous data, and defining 'synchronous data' generated at the second client side as third synchronous data.
The above embodiment is taken as an example of establishing the on-screen communication between the first client and the second client, and the implementation principle of the application is described, but it should be noted that the above embodiment is also applicable to establishing the on-screen communication between a plurality of first clients and a plurality of second clients.
In a specific implementation scenario, the first client is a network program accessed by an internet client, such as a company webpage, an APP program, an applet, etc., and the server uses a virtual device as a synchronous operation environment, such as a "cloud mobile phone", and the second client is software capable of remotely accessing and controlling the "cloud mobile phone".
The core of the implementation mode is that according to the first application program operated by the first client, a second application program which is the same as or similar to the first application program is started in the synchronous operation environment of the server, and is used as a display reference picture of the same screen for communication, so that the problem of inconsistent pictures of the same screen caused by the difference of client terminal equipment is effectively avoided, the calling process of a local interface program of the client cannot be checked in the communication process based on the second application program, the local data of the client cannot be directly accessed, and the privacy of a user is protected in the same-screen process.
Through the implementation mode, the Internet client can initiate the same-screen request to invite the customer service personnel in the process of executing the service operation by using the first application program, and enter the same-screen communication after the customer service personnel answer, so that the problem of initiating the service operation end in the same-screen implementation mode of generating the reference display data at the service end is effectively solved.
In another possible implementation, as shown in fig. 3, a manner in which a customer service person initiates a request for a screen is shown, and the detailed steps are as follows:
in the first client 10:
s301, a first application program is operated; s302, collecting user online information; s303, sending the online information of the user. The above steps are, for example, that the internet client accesses the first application program to perform the service operation, and the first application program collects the online information of the user and sends the online information to the server 20. The user presence information includes, but is not limited to, user identity information, browsing status information, operational data information, and the like.
After receiving the user presence information, the server 20 executes S304 to send the user presence information, and sends the user presence information of the first application to the second client 30.
In the second client 30:
s305, selecting a user based on the online information of the user and creating an on-screen invitation request; s306, sending an on-screen invitation request. In the step, customer service personnel select a certain user to initiate a same screen invitation request according to the acquired user online information of the first application program, namely, actively assisting a customer to operate a service. Optionally, the customer service personnel knows the operation state of the user through the user online information, such as the stay time of a certain service operation part, analyzes according to a preset judgment rule, and selects the client to initiate the same-screen invitation when determining that the user encounters an operation problem.
In the server 20:
s307, processing the same screen invitation request; s308, sending the same screen invitation. The server 20 forwards the received on-screen invite request to the first client.
In the first client 10:
s309, processing the same screen invitation and generating a notification; s310, sending a notification. The S309 processing result includes accepting and rejecting both, and generating a corresponding notification. Optionally, after accepting the on-screen invitation in S309, the running state of the current first application is collected as first running information, and the first running information is sent together with the notification of accepting the on-screen invitation, where the first running information is used by the server 20 to set and run the second application based on the first running information.
In the server 20:
s311, creating or starting a synchronous running environment; s312, sending an instruction of entering the same screen to the first client 10; s313, sending an on-screen entering instruction to the second client 30. After receiving the notification of receiving the on-screen invitation sent in S310, both parties agree to the on-screen communication, and S311 is executed to create or start a synchronous running environment to run the second application program, and notify both party clients.
The steps S314, S315, S316 and the following steps are substantially the same as those in the previous embodiment, and are not described herein. It should be noted that in S314, a third application is started to enter on-screen communication, where the third application is used to display a running picture of the second application in the server 20. In addition, on the basis that the notification sent in S310 includes the first running information, S316 sets the second application program based on the first running information, so that after the on-screen communication is entered, the content displayed by the first client in the third application program is substantially the same as the content displayed by the first application program before the on-screen communication is entered, continuity of service operation is maintained, and similarly, the second client can obtain the current operation state of the user, and user experience is good.
Through the implementation mode, the beneficial effects obtained are as follows: customer service personnel can acquire the online information and the operation state of the user; customer service personnel can actively initiate a same-screen communication request to assist customers; after the communication with the screen is entered, the operation business content of the client is not interrupted.
In one possible implementation manner, the application further provides a method for implementing synchronous graffiti in on-screen communication of an application program, as shown in fig. 4:
after the first client 10 and the second client 30 enter the on-screen communication state, for example:
in the second client 30:
s401, monitoring a start graffiti event; s402, sending a start-up graffiti control instruction.
In the application, the graffiti is used for executing marking and guiding in the same-screen communication process, and a graffiti switch and an operation authority are distributed to both sides of the communication party according to service requirements. After the graffiti is started, the user operation event which occurs on the side executing the graffiti operation in the third application program and the first application program is regarded as the graffiti behavior execution and is not sent to the second application program for execution. In a specific implementation scenario, according to business requirements, the communication effect that both parties can doodle, or one party performs doodle and the other party performs business operation can be achieved. After the user of second client 30 initiates the graffiti, S402 sends a graffiti control instruction to inform first client 10 to initiate the graffiti operation or the graffiti drawing at the same time.
The server 20, S403, sends a command for initiating the graffiti, i.e. forwards the command for initiating the graffiti.
In the first client 10:
s404, creating or displaying Canvas for graffiti. This step is performed after receiving a command to initiate graffiti control, where the Canvas for graffiti is typically a separate component that is placed on top of the application.
In the second client 30:
s405, creating or displaying Canvas for graffiti; s406, collecting graffiti data to generate third synchronous data; s407, transmitting the third synchronous data. The steps stop user operation event collection, and start graffiti data collection in a graffiti working mode, wherein the graffiti data comprises, but is not limited to, graffiti operation behavior data executed by a user, generated graffiti result data and the like.
In the server 20:
s408, receiving third synchronous data, and generating first synchronous data forwarding; s409, the first synchronization data is sent. And forwarding the doodle data at the server. In this application, the type of the synchronization data is a definition based on the nature of the transmission body, and when the third synchronization data is specifically implemented, the third synchronization data is generated as the first synchronization data, and no actual data processing may be performed.
In the first client 10:
s410, drawing is performed in the graffiti Canvas based on the graffiti data in the first synchronization data. This step ultimately synchronously displays the result of the graffiti operation occurring in second client 30 in first client 10.
The same principle occurs in the first client 10 that the user graffiti operation is performed as follows:
in the first client 10:
s411, collecting the graffiti data to generate second synchronous data; s412, sending the third synchronous data.
In the server 20:
s413, receiving the second synchronous data, and generating the second synchronous data as first synchronous data for forwarding; s414, sending the first synchronization data.
In the second client 30:
s415, drawing is performed in the graffiti Canvas based on the graffiti data in the first synchronization data.
The above steps synchronously display the result of the graffiti operation occurring in the first client 10 in the second client 30.
The above implementation achieves synchronized graffiti by creating Canvas for the graffiti in the first client 10, second client 30, respectively.
Optionally, a Canvas for graffiti is not created in the first client 10 and the second client 30, but a Canvas for graffiti is created in the server, after a graffiti control instruction is received to enter a graffiti working mode, a user operation event is collected in the first client 10 and the second client 30, after the user operation event is received by the server, the user operation event is not executed in the second application program, and the user operation event is executed in the Canvas for graffiti, after a graffiti result is generated, the part of display content is sent as the first synchronization data, and the first client 10 and the second client 30 receive and display the graffiti result. Optionally, the above implementation creates a Canvas for graffiti in the second application. Optionally, an independent application program is started in the synchronous running environment to create the Canvas for the graffiti, and in the implementation scene, the third application program and the fourth application program display contents are display screens received in the synchronous running environment and contain the graffiti contents. The technical characteristics of the implementation mode are that the graffiti operation of the client is received, and graffiti content is generated at the server and sent to the client for display.
In a possible implementation manner, the application further provides a method for calling the first client local interface program in the on-screen communication of the application program, as shown in fig. 5, which includes the following steps:
after the first client 10 and the second client 30 enter the on-screen communication state, for example:
in the server 20:
s501, running a second application program, and generating instruction data for calling a local interface program as first synchronous data when determining that the second application program calls the behavior of the local interface program of the first client; s502, sending first synchronous data. Such as local authorization, payment, local input methods, etc. After the communication with the same screen is entered, the second application program is an actual service running program and runs on the server, so that the local interface cannot be called, and the instruction data for calling the local interface program locally on the client is generated through the steps and is sent.
In the first client 10:
s503, receiving first synchronous data, calling a local interface program when determining that the first synchronous data comprises instruction data for calling the local interface program, and acquiring and generating second synchronous data based on an execution result of the local interface program; s504, sending second synchronous data. The first client 10 includes a local interface program, and after receiving the instruction data for calling the local interface program, invokes a corresponding local interface program, for example, receives an instruction for calling local payment, invokes a local payment service program, generates a payment result after the payment is completed, and generates the payment result as second synchronous data to be sent.
In the server 20:
s505, receiving the second synchronous data, analyzing the execution result of the local interface program contained in the second synchronous data, and executing the second application program based on the execution result of the local interface program; s506, collecting display data and generating first synchronous data; s507, sending first synchronous data to a first client; s508, the first synchronization data is sent to the second client. For example, a payment result returned by the first client is received, and if the payment is completed, the second application program is continuously executed according to the result. As described above, S506, S507, and S508 are unnecessary steps for the case where S505 does not cause a change in the display content.
Through the implementation mode, the calling of the local application program interface in the same-screen communication process is realized, and the beneficial effects can be generated that the programs of the server and the client can be executed in the same-screen process, so that the integral business operation can be completed in the same-screen state, and the privacy of a user is effectively protected.
According to the same implementation principle, the implementation mode is also suitable for the call of the server to the local interface program of the second client, namely, the second application program is operated at the server, when the action of the second application program for calling the local interface program of the second client is determined, instruction data for calling the local interface program is generated, and the instruction data are generated into first synchronous data and sent to the second client; receiving first synchronous data at a second client, calling a local interface program when determining that the first synchronous data comprises instruction data for calling the local interface program, acquiring and generating third synchronous data based on an execution result of the local interface program, and sending the third synchronous data to a server; the server receives the third synchronous data, analyzes the execution result of the local interface program contained in the third synchronous data, and executes the second application program based on the execution result of the local interface program. For example, the above implementation manner may be applied to a scenario in which the second client displays the display content of the second application program in a video form, when the user clicks the input frame component in the video screen at the second client, the second application program sends a user operation instruction to the server, the second application program executes the user operation event, places the corresponding input frame component into focus, generates an instruction for calling the local input of the second client, sends the instruction to the second client, after receiving the instruction, calls the input program and inputs the text content, sends the inputted text content to the server, after receiving the instruction, writes the text content in the input frame component, and synchronizes the screen to the client so that the client obtains the input display effect.
In a possible implementation manner, the application further provides a processing method at the end of the on-screen communication of the application program, as shown in fig. 6, for example, the steps of initiating an on-screen ending request at the first client 10 are as follows:
in the first client 10:
s601, running a third application program; s602, generating a request for ending the same screen; s603, sending an end same screen request.
In the server 20:
s604, processing the end on-screen request, and acquiring second running information to generate first synchronous data based on a second application program; s605, sending an end on-screen instruction to the second client 30; s606, sending an end synchronization instruction and first synchronization data to the first client 10. In step S604, the second running information is collected to set a running state of the first application program at the first client.
In the second client 30:
s607, ending the same screen.
In the first client 10:
s608, closing the third application program, starting the first application program to run, and setting the running state of the first application program based on the second running information in the first synchronous data.
By the implementation mode, after the on-screen communication is finished, the first application program is set based on the running information of the second application program, so that the business operation of the first application program is kept in a continuous state after the on-screen communication is finished. For example, the customer service personnel and the customer communicate with each other on the same screen to assist the customer in executing the operation of the application service, and the two parties actually execute the operation through the second application program, for example, part of form data is recorded in the communication process, after the assistance of the on-screen communication is finished, the part of data is collected as second operation information in S604, and the part of data is set in the first application program in S608, so that the customer continues to execute the operation of the operation after the on-screen communication is finished, and repeated work is avoided. Therefore, the realization method has the beneficial effects that after the communication with the screen is finished, the business operation is kept continuous, and the user experience is improved.
Based on the same inventive concept, the embodiment of the present application further provides an application on-screen system, referring to fig. 1, the system includes a first client 10, a server 20, and a second client 30, where the first client 10 includes a first application 101 and a third application 102; the server 20 includes a synchronous running environment 201, where the synchronous running environment 201 runs a second application 2011; the second client 30 comprises a fourth application 301, wherein:
the first client 10 is configured to run the first application 101 and further generate and send a request for same screen; the system is also used for collecting and sending user online information; the third application 102 is further configured to receive and execute a control instruction, and start the third application when the control instruction is determined to be an on-screen entering instruction;
the first client 10 is further configured to run the third application 102, receive first synchronization data, and perform synchronization processing in the third application 102 based on the first synchronization data; and is further configured to collect and send second synchronization data based on the third application 102; the method comprises the steps of creating a graffiti Canvas, collecting graffiti data, and executing drawing based on the graffiti data; the method is further used for closing the third application program 102, starting the first application program 101 to run when the received control instruction is determined to be an end on-screen instruction, and setting the running state of the first application program 101 based on second running information in the first synchronous data;
The server 20 is configured to receive and process the request for the same screen and end the request for the same screen; the method is also used for receiving and sending user state information and a same screen invitation request; the method is also used for sending a notification and control instruction; the system is further configured to configure a start-up synchronous running environment 201 after processing the on-screen request, and set up to run the second application 2011; and is further configured to send a control instruction to the synchronous running environment 201;
the server 20 is further configured to establish a long connection with the first client 10 and the second client 30; and is further configured to generate and transmit the first synchronization data; and is further configured to collect display data and generate the first synchronization data based on the second application 2011; the second synchronization data and the third synchronization data are also received, and when the second synchronization data and the third synchronization data are determined to be user operation data, the second synchronization data are sent to the second application 2011 to be executed; the device is also used for forwarding the second synchronous data and the third synchronous data, wherein the forwarded second synchronous data and third synchronous data comprise one or more of audio data, video data and graffiti data;
the synchronous running environment 201 is configured to run the second application 2011 based on the first running information of the first application 101 in the on-screen request; and is further configured to obtain display data of the second application 2011;
The second application 2011 is configured to generate on-screen display data; and is further configured to execute a user operation event included in the second synchronization data or the third synchronization data;
the server 20 is further configured to collect, based on the second application 2011, second operation information and send the second operation information as first synchronization data to the first client 10 when the processing ends the request;
the second client 30 is further configured to run the fourth application 301, and further configured to receive first synchronization data and perform synchronization processing in the fourth application 301 based thereon; the control instruction is also used for receiving and executing the control instruction, and entering the same screen communication when the control instruction is determined to be the same screen entering instruction; the server side 20 is further configured to receive user online information of the first application program 101 sent by the server side, select a user based on the user online information, create a request for inviting on the same screen, and send the request; and is further configured to collect and send third synchronization data based on the fourth application 301; the method comprises the steps of creating a graffiti Canvas, collecting graffiti data, and executing drawing based on the graffiti data;
the first synchronous data comprise one or more of display data, instruction data and running data, and the second synchronous data and the third synchronous data comprise one or more of user operation data, audio and video data and graffiti data.
Optionally, the first client 10 further comprises a local interface program 103. The first client 10 is further configured to invoke the local interface program 103 according to an instruction of the local interface program, collect an execution result, generate second synchronization data, and send the second synchronization data to the server 20. The server 20 is further configured to, when determining that the second application 2011 invokes the behavior of the local interface program of the first client 10, generate instruction data for invoking the local interface program and send the instruction data as first synchronization data to the first client 10; and is further configured to execute the second application 2011 based on the local interface program execution result when it is determined that the second synchronization data includes the local interface program execution result.
Optionally, the second client 30 further comprises a local interface program. The second client 30 is further configured to invoke the local interface program according to an instruction of the local interface program, collect an execution result, generate third synchronization data, and send the third synchronization data to the server 20. The server 20 is further configured to, when determining that the second application 2011 invokes the behavior of the local interface program of the second client 30, generate instruction data for invoking the local interface program and send the instruction data as first synchronization data to the second client 30; and is further configured to execute the second application 2011 based on the local interface program execution result when it is determined that the third synchronization data includes the local interface program execution result.
The embodiment of the application also provides another on-screen communication method, which comprises the following steps:
in a first client:
the first client comprises a first application program, the first application program is operated, an initiation on-screen request event triggered by a user is received, and on the basis of the first operation information acquired by the first application program, an on-screen request is generated and sent to a server, or on-line information of the user is acquired and sent to the server.
In the server side:
receiving a request of the same screen sent by a first client or a second client; processing the on-screen request and sending an on-screen invitation notification to another client; creating or starting a synchronous operation environment; and starting and/or setting the operation of a second application program corresponding to the first application program in the synchronous operation environment based on the first operation information of the first application program contained in the on-screen request.
In the second client:
the second client comprises a fourth application program, the fourth application program receives a control instruction sent by the server, and when the control instruction is determined to be an on-screen entering instruction, the on-screen entering instruction is executed to enter an on-screen communication state.
The method is not limited to the method that any one of the first client and the second client initiates the same-screen request, and receives the opposite side request or the same-screen instruction sent by the server to enter the same-screen communication state. After the first client and the second client enter the on-screen communication state, one of the first application program and the second application program can be selected as a program for actually executing the service, and the other application program is taken as a snapshot of the program for actually executing the service. The snapshot is only used to synchronously display the running results, and does not execute the actual business. The snapshot and the actual service program are synchronized by transmitting operation result data and user operation data. In addition, according to the operation authority allocation situation, the operation is performed in both clients. For example, the first application program is used as a program for actually executing the service, the second application program is used as a snapshot program, if a user operation occurs at the first client, the following steps are executed:
In a first client:
and collecting one or more of user operation data, program operation result data, audio and video data and graffiti data, generating synchronous data, and sending the synchronous data to the server, wherein the user operation data is collected based on user operation, and the program operation result data is collected based on operation results, operation information, display data and the like of the first application program.
In the server side:
receiving the synchronous data sent by the first client, and executing corresponding synchronous processing based on the property of the synchronous data, wherein the generation of the snapshot display picture comprises the restoration of user operation behaviors in the second application program based on the contained user operation data and the generation of the operation result display picture in the second application program based on the contained program operation result data. After the snapshot is generated in the second application program, the method further comprises the step of generating synchronous data based on the display data acquired by the second application program, and sending the synchronous data to the second client, wherein the display data comprises one or more of image data, video data, snapshot data and drawing instruction data.
In the second client:
and receiving synchronous data sent by the server side, and executing synchronous processing based on the synchronous data, wherein the synchronous processing comprises one or more of generating a display picture, playing audio and video and executing graffiti drawing. For example, the second application running picture video stream sent by the playing server side.
In the above on-screen communication execution process, for example, user operation occurs in the second client, the execution steps are as follows:
in the second client:
and collecting one or more of user operation data, audio and video data and doodling data, generating synchronous data, and sending the synchronous data to the server, wherein the user operation data is generated based on the fourth application program collecting user operation event.
In the server side:
and receiving the synchronous data sent by the second client and sending the synchronous data to the first client. Optionally, the user operation data contained in the data is processed, new synchronous data is generated, and the new synchronous data is sent to the first client.
In a first client:
and receiving the synchronous data sent by the server, acquiring user operation data in the synchronous data, and executing user operation events contained in the first application program. Then, the above steps of collecting user operation data, program operation result data, and the like, and transmitting to the server side are continued.
Through the above three steps, when a user operation occurs in the second client, the operation is actually executed in the first client, and the second client can obtain a synchronous execution effect.
In the above implementation, the first application program is taken as a program for actually running the service, and the second application program is taken as a snapshot for exemplary illustration. The implementation method has the advantages that the local interface program is directly called by the first application program, so that private data of a user is effectively protected, and the business operation process is kept continuous before and after the same screen is started and ended. Optionally, an audio-video communication between the first client and the second client is established for transmitting audio-video data. Optionally, the second application program is used as a program for actually running the service, the first application program is implemented as a snapshot, when user operation occurs in the first application program, user operation data is collected and sent to the server, the user operation data is actually executed in the second application program, then the second application program sends operation result data to the first application program and makes the operation result be presented, and display data is sent to the fourth application program so as to make the display of the fourth application program synchronous, so that the display of the two clients keeps synchronous, and specific implementation principles are described in the embodiment and are not repeated herein. In addition, related graffiti, audio and video communication, and the like are also described in the above embodiments, and are not repeated.
The present application also provides a computer readable storage medium having stored therein a software program which, when read and executed by one or more processors, performs the steps of any of the embodiments of the application on-screen method described above.
Embodiments of the present application also provide a computer program product that, when read and executed by one or more processors, performs the steps of any of the embodiments of the application on-screen method described above.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by hardware associated with computer program instructions, which may be stored on a computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. The storage medium may be a nonvolatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a random access Memory (Random Access Memory, RAM).
It should be noted that, in the above method embodiments of the present application, some steps may be replaced by similar steps, and some steps may be combined, decomposed, and implemented in an adjusted order, so the above embodiments do not limit an application on-screen method set forth in the present application.
In summary, the method and the system for providing the on-screen display data for the application program in the embodiment of the application program on-screen mode are implemented in the same-screen mode of running the application program on the server, and the beneficial effects can be summarized as follows:
in the process that the client executes the application program operation, the client can initiate the same screen request, and other customer service personnel can select the client to initiate the same screen request, so that the flexibility of the service is enhanced, and the method is suitable for the Internet service scene;
when the screen is accessed, the business operation and the running state executed by the current application program are not interrupted;
the communication process of the same screen comprises audio and video communication and synchronous graffiti, and has the face-to-face communication effect;
in the same screen communication process, an application program is run on a server side, a local interface program of a client side can be called, but the same screen is not carried out on the calling process, so that the business operation process is complete, and the privacy data of a user is effectively protected;
after the end of the same screen, the business operation and running state executed by the application program are kept continuous.
It will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if such modifications and variations of the embodiments of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to encompass such modifications and variations.

Claims (10)

1. An application program on-screen communication method is characterized in that the method is applied to a first client, the first client comprises a first application program and a third application program, the method is used for realizing on-screen communication of the application program between the first client and a second client, and the method comprises the following steps:
running the first application program, receiving an initiation on-screen request event triggered by a user, collecting first running information based on the first application program, generating an on-screen request and sending the same to a server, or collecting user on-line information and sending the same to the server; the method comprises the steps that a first application program is an application program for executing business operation for a user, a server side is used for receiving a same-screen request sent by a first client side, processing the same-screen request, creating or starting a synchronous operation environment, starting and/or setting second application program operation corresponding to the first application program in the synchronous operation environment based on first operation information of the first application program contained in the same-screen request, acquiring display data based on the second application program to generate first synchronous data, and sending the first synchronous data to the first client side and the second client side, wherein the display data comprises one of image data, video data, snapshot data and drawing instruction data;
The first client receives a control instruction sent by the server, and starts the third application program to enter a same-screen communication state when the control instruction is determined to enter the same-screen instruction; the first client receives first synchronous data sent by the server and executes synchronous processing based on the first synchronous data, wherein the synchronous processing comprises one or more of generating a display picture, playing an audio and video, executing a graffiti drawing and executing an instruction; the first client acquires one or more of user operation data, audio and video data and graffiti data, generates second synchronous data and sends the second synchronous data to the server, wherein the user operation data is generated based on the third application program acquiring a user operation event; the server side is further configured to receive second synchronization data sent by the first client side, obtain user operation data therein, and execute a user operation event included in the second application program;
the second client comprises a fourth application program, the fourth application program is used for receiving a control instruction sent by the server, executing the same-screen entering instruction to enter a same-screen communication state when the control instruction is determined to be the same-screen entering instruction, receiving first synchronous data sent by the server, and executing synchronous processing based on the first synchronous data, wherein the synchronous processing comprises one or more of generating a display picture, playing audio and video and executing graffiti drawing; the fourth application program is further configured to collect one or more of user operation data, audio/video data, and graffiti data, generate third synchronization data, and send the third synchronization data to the server, where the server is further configured to receive the third synchronization data sent by the second client, obtain the user operation data therein, and execute a user operation event included in the second application program.
2. The method of claim 1, wherein after the first client enters the on-screen communication state, further comprising:
receiving a start-up graffiti control instruction sent by a server or receiving a start-up graffiti event triggered by a user, and creating or displaying a Canvas for graffiti on the first client;
based on the graffiti Canvas, collecting graffiti data to generate second synchronous data and sending the second synchronous data; and executing drawing in the graffiti Canvas based on the graffiti data in the first synchronization data.
3. The method of claim 1, wherein receiving the first synchronization data sent by the server performs the synchronization process, further comprising:
when the first synchronous data comprise instruction data for calling a local interface program, calling the local interface program at the first client, acquiring and generating second synchronous data based on an execution result of the local interface program, and sending the second synchronous data to the server.
4. The method of claim 1, further comprising, after receiving the control instruction:
and when the control instruction is determined to be an end on-screen instruction, closing the third application program, starting the first application program to run, and setting the running state of the first application program based on the second running information of the second application program in the first synchronous data.
5. An application on-screen communication method, wherein the method is applied to a second client, the second client comprises a fourth application, and the method is used for realizing the on-screen communication of the application between a first client and the second client, and the method comprises the following steps:
the fourth application program receives a control instruction sent by a server, and executes the same-screen entering instruction to enter a same-screen communication state when the control instruction is determined to be the same-screen entering instruction; receiving first synchronous data sent by the server, and executing synchronous processing based on the first synchronous data, wherein the synchronous processing comprises one or more of generating a display picture, playing audio and video and executing graffiti drawing;
the fourth application program collects one or more of user operation data, audio and video data and graffiti data, generates third synchronous data and sends the third synchronous data to the server, wherein the user operation data is generated based on the fourth application program collecting user operation events and is used for being executed in a second application program of the server; the method comprises the steps that a first application program sends a first screen invitation request to a server side, and a second application program sends a second screen invitation request to the server side;
The first client comprises a first application program and a third application program, wherein the first application program is an application program for executing business operation for a user; the first client is used for running the first application program, receiving an initiation on-screen request event triggered by a user, collecting first running information based on the first application program, generating an on-screen request and sending the same to the server, or collecting user on-line information and sending the same to the server; the server is used for receiving a request of the same screen sent by a first client, processing the request of the same screen, creating or starting a synchronous operation environment, starting and/or setting a second application program corresponding to the first application program to operate in the synchronous operation environment based on first operation information of the first application program contained in the request of the same screen, collecting display data based on the second application program to generate first synchronous data, and sending the first synchronous data to the first client and the second client, wherein the display data comprises one of image data, video data, snapshot data and drawing instruction data; the first client is also used for receiving a control instruction sent by the server, and starting the third application program to enter a same-screen communication state with the second client when the control instruction is determined to be a same-screen entering instruction; the first client is further configured to receive first synchronization data sent by the server, and perform synchronization processing based on the first synchronization data, where the synchronization processing includes one or more of generating a display screen, playing an audio/video, performing graffiti drawing, and performing an instruction; the first client is further used for collecting one or more of user operation data, audio and video data and doodling data, generating second synchronous data and sending the second synchronous data to the server, wherein the user operation data is generated based on the user operation event collected by the third application program; the server side is further configured to receive second synchronization data sent by the first client side, obtain user operation data therein, and execute a user operation event included in the second application program.
6. The method of claim 5, wherein after the second client enters the on-screen communication state, further comprising:
receiving a start-up graffiti control instruction sent by a server or receiving a start-up graffiti event triggered by a user, and creating or displaying a Canvas for graffiti on the second client;
based on the graffiti Canvas, collecting graffiti data to generate third synchronous data and sending the third synchronous data; and executing drawing in the graffiti Canvas based on the graffiti data in the first synchronization data.
7. An application program on-screen communication implementation method is characterized in that the method is applied to a server, and the method is used for implementing the application program on-screen communication between a first client and a second client, and comprises the following steps:
receiving a request of the same screen sent by the first client or the second client;
processing the on-screen request and sending an on-screen invitation notification to another client;
creating or starting a synchronous operation environment;
starting and/or setting a second application program operation corresponding to the first application program in the synchronous operation environment based on first operation information of the first application program contained in the on-screen request;
Generating first synchronous data based on the display data acquired by the second application program, and sending the first synchronous data to the first client and the second client, wherein the display data comprises one of image data, video data, snapshot data and drawing instruction data; receiving second synchronous data sent by the first client or third synchronous data sent by the second client, acquiring user operation data in the second synchronous data, and executing user operation events contained in the second application program;
the first client comprises a first application program and a third application program, wherein the first application program is an application program for executing business operation for a user; the first client is used for running the first application program, receiving an initiation on-screen request event triggered by a user, collecting first running information based on the first application program, generating an on-screen request and sending the same to the server, or collecting user on-line information and sending the same to the server; the third application program is started when the first client receives a control instruction for entering the same screen sent by the server, and the first client enters a same screen communication state after the third application program is started; the third application program is used for receiving the first synchronous data and executing synchronous processing based on the first synchronous data, and is also used for collecting and sending the second synchronous data, creating a graffiti Canvas, collecting the graffiti data and executing drawing based on the graffiti data; the second client comprises a fourth application program, the fourth application program is used for receiving a control instruction sent by a server, executing the same-screen entering instruction to enter a same-screen communication state when the control instruction is determined to be the same-screen entering instruction, and receiving first synchronous data sent by the server, and executing synchronous processing based on the first synchronous data, wherein the synchronous processing comprises one or more of generating a display picture, playing audio and video and executing graffiti drawing; the fourth application program is further configured to collect one or more of user operation data, audio/video data, and graffiti data, generate third synchronization data, and send the third synchronization data to the server.
8. The method of claim 7, wherein running the second application in the synchronous running environment further comprises:
when determining that the second application program calls the behavior of the first client local interface program, generating instruction data for calling the local interface program, generating the instruction data into first synchronous data and sending the first synchronous data to the first client;
and the second synchronous data sent by the first client is received, the execution result of the local interface program contained in the second synchronous data is analyzed, and the second application program is executed based on the execution result of the local interface program.
9. The method as recited in claim 7, further comprising:
receiving an end on-screen request sent by the first client or the second client, processing the end on-screen request, sending an end on-screen instruction to the first client or the second client, and acquiring second operation information based on the second application program to generate first synchronous data and send the first synchronous data to the first client, wherein the second operation information is used for setting the operation state of the first application program at the first client.
10. An application program on-screen system comprises a first client, a server and a second client, wherein the first client comprises a first application program and a third application program; the server side comprises a synchronous operation environment, and the synchronous operation environment operates a second application program; the second client includes a fourth application, wherein:
the first client is used for running the first application program and generating and sending a same screen request; the system is also used for collecting and sending user online information; the control instruction is also used for receiving and executing the control instruction, and starting the third application program when the control instruction is determined to be an on-screen entering instruction;
the first client is further configured to run the third application program, receive first synchronization data, and perform synchronization processing in the third application program based on the first synchronization data; and is further configured to collect and send second synchronization data based on the third application; the method comprises the steps of creating a graffiti Canvas, collecting graffiti data, and executing drawing based on the graffiti data; the server is also used for calling the local interface program according to the instruction of the local interface program, collecting the execution result to generate second synchronous data and sending the second synchronous data to the server; the control method is also used for closing the third application program when the received control instruction is determined to be an end on-screen instruction, starting the first application program to run, and setting the running state of the first application program based on second running information in the first synchronous data;
The server is used for receiving and processing the same-screen request and ending the same-screen request; the method is also used for receiving and sending user state information and a same screen invitation request; the method is also used for sending a notification and control instruction; the system is also used for configuring and starting a synchronous operation environment after processing the same-screen request, and setting and operating the second application program; the control instruction is also used for sending a control instruction to the synchronous operation environment;
the server is also used for establishing long connection with the first client and the second client; and is further configured to generate and transmit the first synchronization data; and is further configured to collect display data and generate the first synchronization data based on the second application; the second synchronous data and the third synchronous data are received and are sent to the second application program for execution when the second synchronous data and the third synchronous data are determined to be user operation data; the device is also used for forwarding the second synchronous data and the third synchronous data, wherein the forwarded second synchronous data and third synchronous data comprise one or more of audio data, video data and graffiti data;
the synchronous operation environment is used for operating the second application program based on the first operation information of the first application program in the same screen request; the method is also used for acquiring the second application program display data;
The second application program is used for generating display data of the same screen; and is further configured to execute a user operation event included in the second synchronization data or the third synchronization data;
the server is further configured to, when determining that the second application program invokes the behavior of the first client local interface program, generate instruction data for invoking the local interface program and send the instruction data as first synchronization data to the first client; and the second application program is executed based on the execution result of the local interface program when the second synchronous data is determined to contain the execution result of the local interface program; the method is also used for generating instruction data for calling the local interface program and sending the instruction data to the second client as first synchronous data when determining the action of calling the local interface program of the second client by the second application program; and the second application program is executed based on the execution result of the local interface program when the third synchronous data is determined to contain the execution result of the local interface program; the method is also used for acquiring second running information based on the second application program and sending the second running information to the first client as first synchronous data when the processing ends the same-screen request;
The second client is further configured to run the fourth application program, and further configured to receive the first synchronization data and perform synchronization processing in the fourth application program based on the first synchronization data; the control instruction is also used for receiving and executing the control instruction, and entering the same screen communication when the control instruction is determined to be the same screen entering instruction; the method is also used for receiving user online information of the first application program sent by the server, selecting a user based on the user online information, creating a same-screen invitation request and sending the same-screen invitation request; and is further configured to collect and send third synchronization data based on the fourth application; the method comprises the steps of creating a graffiti Canvas, collecting graffiti data, and executing drawing based on the graffiti data; the method is also used for calling the local interface program according to the instruction of the local interface program, collecting the execution result to generate third synchronous data and sending the third synchronous data to the server;
the first synchronous data comprise one or more of display data, instruction data and running data, and the second synchronous data and the third synchronous data comprise one or more of user operation data, audio and video data and graffiti data.
CN202210437002.1A 2022-04-27 2022-04-27 Application program on-screen communication method and system Active CN114884914B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210437002.1A CN114884914B (en) 2022-04-27 2022-04-27 Application program on-screen communication method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210437002.1A CN114884914B (en) 2022-04-27 2022-04-27 Application program on-screen communication method and system

Publications (2)

Publication Number Publication Date
CN114884914A CN114884914A (en) 2022-08-09
CN114884914B true CN114884914B (en) 2024-04-02

Family

ID=82670678

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210437002.1A Active CN114884914B (en) 2022-04-27 2022-04-27 Application program on-screen communication method and system

Country Status (1)

Country Link
CN (1) CN114884914B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115033335B (en) * 2022-08-11 2023-01-10 青岛中正合力软件有限公司 Display processing method applied to same screen of webpage

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104794008A (en) * 2015-04-28 2015-07-22 北京数字天域科技有限责任公司 Application program synchronizing method, device and system
CN111107145A (en) * 2019-12-17 2020-05-05 青岛希望鸟科技有限公司 Online synchronous communication system based on HTML5 webpage
CN111581564A (en) * 2020-05-10 2020-08-25 青岛希望鸟科技有限公司 Webpage synchronous communication method realized by applying Canvas
CN113868573A (en) * 2021-09-07 2021-12-31 青岛希望鸟科技有限公司 Method and system for quickly establishing one-screen interaction based on webpage
CN114020396A (en) * 2021-11-18 2022-02-08 湖南麒麟信安科技股份有限公司 Display method of application program and data generation method of application program

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3522022A4 (en) * 2016-09-27 2020-05-27 Opro Co., Ltd. Cloud service synchronization program and connection program

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104794008A (en) * 2015-04-28 2015-07-22 北京数字天域科技有限责任公司 Application program synchronizing method, device and system
CN111107145A (en) * 2019-12-17 2020-05-05 青岛希望鸟科技有限公司 Online synchronous communication system based on HTML5 webpage
CN111581564A (en) * 2020-05-10 2020-08-25 青岛希望鸟科技有限公司 Webpage synchronous communication method realized by applying Canvas
CN113868573A (en) * 2021-09-07 2021-12-31 青岛希望鸟科技有限公司 Method and system for quickly establishing one-screen interaction based on webpage
CN114020396A (en) * 2021-11-18 2022-02-08 湖南麒麟信安科技股份有限公司 Display method of application program and data generation method of application program

Also Published As

Publication number Publication date
CN114884914A (en) 2022-08-09

Similar Documents

Publication Publication Date Title
JP7263442B2 (en) System and method for real-time remote control of mobile applications
US11785056B2 (en) Web browser interface for spatial communication environments
US11088971B2 (en) Virtual area communications
CN111596985B (en) Interface display method, device, terminal and medium in multimedia conference scene
US9009603B2 (en) Web browser interface for spatial communication environments
EP1451672B1 (en) Rich communication over internet
US10187439B2 (en) Dynamic recording of online conference
US20060092268A1 (en) Teleconference method with a shared working space
CN101484886A (en) Methods and apparatuses for dynamically sharing a portion of a display for application based screen sampling
WO2017189634A1 (en) Online conference broadcast using broadcast component
WO2022111193A1 (en) Meeting control method, apparatus, device, and medium
CN112437004A (en) Information processing method, information processing device, electronic equipment and storage medium
CN114884914B (en) Application program on-screen communication method and system
CN114422460A (en) Method and system for establishing same-screen communication sharing in instant messaging application
CN112751681B (en) Image processing method, device, equipment and computer readable storage medium
CN108900794B (en) Method and apparatus for teleconferencing
JP2003323386A (en) Collaboration server, collaboration system, program product with collaboration stored therein, and collaboration method
CN116820792B (en) Interactive processing method applied to web page on-screen communication
US20230344878A1 (en) Visual Communications
US20220382825A1 (en) Method and system for web page co-browsing
WO2017180542A1 (en) Joining executable component to online conference
Peña-Mora et al. An integrated telepresence environment for collaboration in construction
Chang et al. SignalShare: A Secondary Two-Way Mediated Interaction Tool
CN118092762A (en) Mode switching method, mode switching device, mode switching equipment and computer readable storage medium
CN117354451A (en) Video picture editing method and device based on video call and computer equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant