WO2023103586A1

WO2023103586A1 - Remote collaboration method and system based on sharing of auxiliary stream of video conference, and storage device

Info

Publication number: WO2023103586A1
Application number: PCT/CN2022/125416
Authority: WO
Inventors: 赵杰; 何贤英; 刘玉焘; 崔芳芳; 石金铭; 陈保站; 王文超; 张延家
Original assignee: 郑州大学第一附属医院
Priority date: 2021-12-09
Filing date: 2022-10-14
Publication date: 2023-06-15
Also published as: CN114201256A

Abstract

A remote collaboration method and system based on the sharing of an auxiliary stream of a video conference, and a storage device, which are applicable to the field of video conferences. The method comprises: a video conference establishing a connection with two terminals; sending, in the form of a video auxiliary stream, screen playing content of a video conference terminal to a local terminal and a remote terminal; the remote terminal applying for remote collaboration, and after authorization, establishing a remote collaboration channel between the remote terminal and the local terminal; after an operation is performed, by means of a mouse and/or a keyboard, on a screen picture received by the remote terminal, the remote terminal encapsulating trigger event content corresponding to an operation instruction into a data packet, and then sending the data packet to the video conference terminal; and after the data packet is received, the video conference terminal parsing the specific trigger event content, then converting same into an operation instruction for a virtual mouse and/or a virtual keyboard, and executing the operation instruction on the local terminal. Therefore, two terminals in a video conference can both realize remote conference collaboration without the additional deployment of an application program.

Description

Method, system and storage device for remote collaboration based on video conferencing sharing auxiliary stream

technical field

The present invention relates to the technical field of video conferencing, in particular to a method, system and storage device for remote collaboration based on video conferencing sharing auxiliary streams.

Background technique

Video conferencing terminals are widely used in daily work communication, academic seminars, training and teaching, etc. In addition to real-time transmission of images and audio in video conferencing, there is also a need for double-terminal or even multi-terminal remote collaboration. Remote collaboration is the local The computer screen is shared to the remote end to realize remote assistance operation, so as to facilitate efficient communication between the two parties involved.

The current remote collaboration technology generally adopts the client-server dual-terminal application mode. The client first establishes a connection with the server, and the server opens the desktop sharing permission. The client can remotely perform mouse and keyboard operations, just like locally It is as convenient to operate remotely.

The current mainstream remote collaboration software tools are relatively mature, including desktop sharing in QQ software, LookMyPC remote desktop connection software, PCAnywhere sunflower remote control software, sunflower remote control software, etc. The defects and shortcomings of these software tools are reflected in the fact that they must be deployed independently applications and interconnected.

The video conferencing terminal has the real-time communication function that both ends can transmit voice and image in real time. However, if one end of the video conference wants to conduct remote collaboration with the other end, it cannot be controlled by the mouse and keyboard, and can only be guided by voice and video. operate.

Contents of the invention

Aiming at the deficiencies in the related technologies, the technical problem to be solved by the present invention is to provide a method for remote collaboration based on video conference sharing auxiliary stream that can realize remote conference collaboration without additional deployment of application programs on both terminals of the video conference. Method, system and storage device.

In order to solve the problems of the technologies described above, the technical solution adopted in the present invention is:

A method for remote collaboration based on video conferencing sharing content, including:

S10, the video conference establishes a connection with two terminals, wherein one terminal is a local terminal and the other terminal is a remote terminal;

S20, the content played on the screen of the video conferencing terminal is sent to the local terminal and the remote terminal in the form of video auxiliary stream;

S30, the remote terminal applies for remote collaboration, and after being authorized by the video conferencing terminal, establishes a remote collaboration channel between the remote terminal and the local terminal;

S40, after the remote terminal operates on the received screen image through the electrically connected mouse and/or keyboard, according to the operation instruction set, the content of the trigger event corresponding to the operation instruction is encapsulated into a data packet and then sent to the video conference terminal;

S50. After receiving the data packet, the video conferencing terminal parses out the specific trigger event content, converts it into an operation command of the virtual mouse and/or virtual keyboard, and executes it on the local terminal to complete the control corresponding to the remote terminal;

Wherein, the operation instruction set includes operation instructions based on mouse and keyboard action attributes.

Preferably, in the step S50, the operation instructions converted into virtual mouse and/or virtual keyboard and executed include:

S501, calling the SendInput function of the SDK interface to simulate the trigger event of the remote terminal;

S502, insert the SendInput function into the INPUT structure, connect it in series to the input stream of the keyboard or mouse, and send it to the thread;

S503. Listen to the trigger event in the thread, and generate an operation event of the virtual mouse and/or the virtual keyboard.

Preferably, in step S502, during the process of inserting the SendInput function into the INPUT structure, user interface privilege isolation constraints are set to allow the application to only inject event input into the application at or below the integrity level.

Preferably, in step S502, the SendInput function is inserted into the INPUT structure and connected in series to the input stream of the keyboard or mouse, specifically including:

S5021, determine whether it is a mouse event or a keyboard event, if it is a mouse event, go to step S5022, otherwise go to step S5023;

S5022, passing the coordinate position into the MOUSEINPUT structure;

S5023, passing the key value of the keyboard into the KEYBDINPUT structure.

Preferably, the operation instruction is composed of data in json format, including: an event type, and a key value corresponding to the event type.

Preferably, the event types include: mouse button events, mouse movement events, mouse touch events and keyboard button events;

The key values corresponding to mouse button events include: left button, middle button and right button;

The key values corresponding to the mouse movement event include: screen width, screen height, mouse coordinate x and mouse coordinate y;

The key value corresponding to the mouse touch event includes: finger movement and finger coordinate information group;

The key value corresponding to the keyboard key event includes: keyboard key value.

Among them: the mouse coordinate x is the x coordinate of the mouse relative to the screen width; the mouse coordinate y is the y coordinate of the mouse relative to the screen height;

Finger actions include: the first finger is pressed, the last finger is released, the non-first finger is pressed, the non-last finger is released, and the finger moves;

The coordinate information group of the finger includes: the abscissa of the finger and the ordinate of the finger.

Correspondingly, the remote collaboration system based on video conferencing sharing auxiliary stream includes a local terminal, a remote terminal and a video conferencing terminal, and is characterized in that:

The video conferencing terminal is used to send the content displayed on the screen to the local terminal and the remote terminal in the form of video auxiliary stream; including:

An operation instruction set, which is used to establish operation instructions based on mouse and keyboard action attributes;

An authorization module, configured to authorize the remote terminal in response to the remote collaboration request of the application module, and establish a remote collaboration channel for the remote terminal and the local terminal after authorization;

The data analysis module is used to receive and analyze the data packet of the data generation module, and generate the simulation operation instruction after analyzing the content of the specific trigger event;

The local operation generating module is used to convert the trigger event content parsed by the data parsing module into operation instructions of the virtual mouse and/or virtual keyboard and display them on the local display interface of the local terminal;

The remote terminal includes:

The application module is used to initiate a remote collaboration request to the video conferencing terminal;

The remote operation module, on the received screen, operates through the mouse and/or keyboard;

The data generation module is used to generate corresponding trigger event content according to the operation instruction set from the operation of the remote operation module and encapsulate it into a set of commands and send it to the video conference terminal;

The screen image is a display interface of the remote terminal, and is used to display screen playback content.

The local terminal includes:

The local display interface is used to display the playback content of the video conferencing terminal.

Correspondingly, the storage device stores a plurality of instructions, wherein the instructions are adapted to be loaded by the processor and execute the method for remote collaboration based on video conference sharing auxiliary stream as described above.

The beneficial technical effects of the present invention are:

The method and system for remote collaboration based on video conferencing sharing auxiliary stream provided by the present invention establishes a line connection between two terminals, and transmits the video conference content of the local terminal to the remote terminal in the form of video auxiliary stream; the remote terminal When it is necessary to mark the content of the meeting, the user of the remote terminal applies for remote collaboration. After being authorized by the video conference terminal, the remote collaboration channel of the video conference is established; the remote terminal encapsulates the content of the trigger event corresponding to the operation command through the operation of the local mouse and keyboard After receiving the data packet, the video conference terminal parses out the specific trigger event content, converts it into virtual mouse and/or virtual keyboard operation instructions and executes them on the local terminal, through the virtual mouse and /or the virtual keyboard completes the local control and operation, including marking and sharing the screen, thereby realizing the function of remote collaboration, which is extremely practical.

Description of drawings

FIG. 1 is a schematic flowchart of a method for remote collaboration based on video conference sharing auxiliary stream provided by Embodiment 1 of the present invention;

FIG. 2 is a schematic structural diagram of a remote collaboration system based on video conference sharing auxiliary stream provided by Embodiment 1 of the present invention;

FIG. 3 is a schematic flowchart of step S50 in the method for remote collaboration based on video conference sharing auxiliary stream provided by Embodiment 2 of the present invention;

FIG. 4 is a schematic flowchart of step S502 in Embodiment 2 of the present invention;

Fig. 5 is a schematic diagram of equipment deployment during the use of the present invention;

Fig. 6 is a data flow chart in the process of using the present invention;

In the picture:

10 is a local terminal, 20 is a remote terminal, and 30 is a video conferencing terminal;

101 is a local display interface;

201 is an application module, 202 is a remote operation module, 203 is a data generation module, and 204 is a screen image;

301 is an operation instruction set, 302 is an authorization module, 303 is a data analysis module, and 304 is a local operation generation module.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, rather than all embodiments; based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work, all belong to the protection scope of the present invention .

Secondly, the present invention is described in detail in combination with schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of explanation, the cross-sectional view showing the device structure will not be partially enlarged according to the general scale, and the schematic diagram is only an example, and it should not be limited here. The protection scope of the present invention. In addition, the three-dimensional space dimensions of length, width and depth should be included in actual production.

An embodiment of the method, system and storage device for remote collaboration based on video conference sharing auxiliary stream of the present invention will be described in detail below with reference to the accompanying drawings.

Embodiment one

As shown in Figure 1, the remote collaboration method based on video conference sharing content includes:

In the field of programming, json (JavaScriptObjectNotation) is a lightweight data exchange format based on a subset of ECMAScript; json uses a text format that is completely independent of language, but also uses C, C++, C#, Java, JavaScript, Perl, Python, etc. are similar to the habits of the C language family. These characteristics make json an ideal data exchange language, which is easy for people to read and write, and is also easy for machines to analyze and generate, which greatly improves the network transmission rate; the data format of json format data is relatively simple, easy to read and write, and the format is compressed to occupy bandwidth It is relatively small and supports multiple languages at the same time, which is convenient for server-side parsing. For most web applications, there is no need for complex XML to transfer data at all, and the json data exchange format provides better simplicity and flexibility.

Therefore, when the present invention is implemented, the operation instruction is composed of data in json format, including: event type, and a key value corresponding to the event type.

Further, the event types include: mouse button event, mouse movement event, mouse touch event and keyboard button event.

Furthermore, the key values corresponding to mouse button events include: left button, middle button and right button; the specific data in json format can be realized by the following code:

The json data packet of the mouse button event:

{

"method": 1, //event type

"body": {

"button":1//Mouse event type, 1 left button, 2 middle button, 3 right button

}

Furthermore, the key value corresponding to the mouse movement event includes: screen width, screen height, mouse coordinate x and mouse coordinate y; wherein: mouse coordinate x is the x coordinate of the mouse relative to the screen width; mouse coordinate y is the mouse coordinate relative to The y coordinate of the screen height; its specific data in json format can be realized by the following code:

Mouse move event json data packet:

Furthermore, the key value corresponding to the mouse touch event includes: finger motion and finger coordinate information group; wherein, the finger motion includes: the first finger is pressed, the last finger is released, the non-first The finger is pressed, the non-last finger is released, and the finger is moved; the coordinate information group of the finger includes: the abscissa of the finger and the ordinate of the finger; the specific data in json format can be realized by the following code:

Furthermore, the key value corresponding to the keyboard key event includes: a key value of the keyboard.

The invention also provides a system for remote collaboration based on video conference sharing auxiliary stream.

As shown in FIG. 2 , the remote collaboration system based on video conference sharing auxiliary stream includes a local terminal 10, a remote terminal 20 and a video conference terminal 30;

The video conferencing terminal 30 is used to send the content displayed on the screen to the local terminal 10 and the remote terminal 20 in the form of a video auxiliary stream; including:

An operation instruction set 301, configured to establish operation instructions based on mouse and keyboard action attributes;

The authorization module 302 is configured to authorize the remote terminal 20 in response to the remote collaboration request of the application module 201, and establish a remote collaboration channel for the remote terminal 20 and the local terminal 10 after authorization;

The data parsing module 303 is used for receiving the data packet of the data generating module 203 and parsing, after parsing out the specific trigger event content, generating the simulation operation instruction;

The local operation generation module 303 is used to convert the trigger event content analyzed by the data analysis module 303 into an operation instruction of a virtual mouse and/or a virtual keyboard and display it on the local display interface 101 of the local terminal 10;

Described remote terminal 20 comprises:

An application module 201, configured to initiate a remote collaboration request to the video conference terminal 30;

The remote operation module 202, on the received screen image 204, operates through the mouse and/or keyboard;

The data generation module 203 is used to generate corresponding trigger event content according to the operation instruction set by the operation of the remote operation module 202 and encapsulate it into a set of commands and send it to the video conference terminal 30;

The screen image 204 is a display interface of the remote terminal, and is used for displaying content to be played on the screen.

The local terminal 10 includes:

The local display interface 101 is used to display the playback content of the video conferencing terminal 30 .

The method and system for remote collaboration based on video conferencing sharing auxiliary stream provided in this embodiment, establishes a line connection between two terminals, and transmits the video conference content of the local terminal to the remote terminal in the form of video auxiliary stream; When the terminal needs to mark the content of the meeting, the user of the remote terminal applies for remote collaboration. After being authorized by the video conference terminal, a remote collaboration channel for the video conference is established; After being encapsulated into a data packet, it is sent to the video conference terminal; after receiving the data packet, the video conference terminal parses out the specific trigger event content, converts it into virtual mouse and/or virtual keyboard operation instructions and executes them on the local terminal, through the virtual mouse And/or the virtual keyboard to complete the local control and operation, including marking and sharing the screen, so as to realize the function of remote collaboration, which is extremely practical.

Embodiment two

On the basis of Embodiment 1, in the method of remote collaboration based on video conference sharing auxiliary stream, in the step S50, the operation instructions of the virtual mouse and/or virtual keyboard are converted and executed, specifically including:

Specifically, in step S502, during the process of inserting the SendInput function into the INPUT structure, user interface privilege isolation constraints are set to allow the application to only inject event input into the application at or below the integrity level.

In the process of inserting the SendInput function into the INPUT structure in the present invention, it is constrained by UIPI (User Interface Privilege Isolation, user interface privilege isolation), that is, the application program is allowed to only inject event input into the application program equal to or less than the integrity level, Low-level applications should be launched with administrator privileges to enable full control.

In addition, this function inserts the INPUT structure into the keyboard or mouse input stream and sends it to the thread. The system or hook listener intercepts and processes the event from the thread, and finally generates mouse and keyboard events.

Further, in step S502, the SendInput function is inserted into the INPUT structure and connected in series to the input stream of the keyboard or mouse, specifically including:

S5021, judging whether it is a mouse event or a keyboard event, if it is a mouse event, go to step S5022, otherwise go to step S5023;

S5022, passing the coordinate position into the MOUSEINPUT structure;

S5023, passing the key value of the keyboard into the KEYBDINPUT structure.

In this embodiment, the specific structure of INPUT is as follows:

Structure description:

type: event type, 0 mouse event, 1 keyboard event;

If it is a mouse event, the coordinate position is passed into the MOUSEINPUT structure;

If it is a keyboard event, information such as the key value of the keyboard is passed into the KEYBDINPUT structure.

The present invention also provides a storage device, wherein a plurality of instructions are stored, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned remote collaboration method based on video conference sharing auxiliary stream.

The storage device may be a computer-readable storage medium, and may include: ROM, RAM, magnetic disk or optical disk, and the like.

Hereinafter, the application process of the present invention will be described with a specific embodiment.

Fig. 5 is a schematic diagram of device deployment during the use of the present invention. As shown in Fig. 5, the remote collaboration system based on video conferencing sharing auxiliary stream of the present invention is composed of dual-end devices (local terminal and remote terminal) and software, and the local terminal can be Including the video conferencing terminal and the local computer, the remote terminal can include: remote computer and video playback software: among them, the systems of the local terminal and the remote terminal are connected through the network (that is, the line connection between the two terminals is established), but the video The conference terminal and the local computer network are not necessarily connected to the network, but can only share computer screen information through VGA or HDMI connection.

Fig. 6 is the data flow chart in the use process of the present invention; Carry out example illustration with the process of mouse moving event, as shown in Fig. 6, receiver (remote terminal) moves mouse on the picture by operating mouse, forms mouse moving event, And generate the json data packet of the mobile event, and send it to the initiator (local terminal) through the socket channel;

The originator (local terminal) receives the json data and parses it to find out the specific event. Through the SendInput interface provided by windows, fills the specific content of the event into the INPUT data structure, passes it into SendInput, and sends the event to the system thread. Get the event from the thread and process it, and finally move the mouse to the specified position.

In the present invention, the local display interface of the local terminal can be connected with the video conferencing terminal through VGA or HDMI, and at the same time, the screen playback content of the video conferencing terminal is sent to the local terminal and the remote terminal in the form of auxiliary video stream; so that the present invention The local terminal and the remote terminal may not be in the same network;

When in use, the remote terminal first applies for collaboration, and after the video conferencing terminal accepts and authorizes the request, a remote collaboration channel between the remote terminal and the local terminal is established; the virtual keyboard and virtual mouse operations based on the video auxiliary stream of the local terminal are enabled to realize local and local cooperation. Annotation for screen sharing between remote endpoints.

In this embodiment, the roles (local terminal or remote terminal) between the two terminals can be interchanged, and the party applying for remote collaboration is regarded as a remote terminal, so that the other party can be displayed synchronously after being marked by any party. Compared with the traditional technology, Through the built-in operation instruction set of the video conferencing terminal, the two terminals do not need to deploy independent applications, and the operation is convenient.

In addition, in the process of video conferencing, data sharing is limited to video streams, and there is no risk of data leakage; at the same time, remote collaboration is based on advance reservation authorization, and related operations are performed within a controllable range, reducing security risks and being extremely practical. powerful.

In the description of the present invention, terms such as "installation", "connection", "connection" and "fixation" should be interpreted in a broad sense unless otherwise clearly specified and limited. Disconnected connection, or integrated; may be mechanically connected, may also be electrically connected; may be directly connected, may also be indirectly connected through an intermediary, may be an internal communication between two components or an interactive relationship between two components, unless otherwise There are clear limits. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention according to specific situations.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

It can be understood that related features in the above method and system can refer to each other.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system and modules can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the embodiments provided in this application, it should be understood that the disclosed system and method can be implemented in other ways. The system embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims

The method for remote collaboration based on video conferencing sharing auxiliary stream, characterized in that: comprising:

S10, the video conference establishes a connection with two terminals, wherein one terminal is a local terminal and the other terminal is a remote terminal;

S20, the content played on the screen of the video conferencing terminal is sent to the local terminal and the remote terminal in the form of video auxiliary stream;

S30, the remote terminal applies for remote collaboration, and after being authorized by the video conferencing terminal, establishes a remote collaboration channel between the remote terminal and the local terminal;

S40, after the remote terminal operates on the received screen image through the electrically connected mouse and/or keyboard, according to the operation instruction set, the content of the trigger event corresponding to the operation instruction is encapsulated into a data packet and then sent to the video conference terminal;

S50. After receiving the data packet, the video conferencing terminal parses out the specific trigger event content, converts it into an operation command of the virtual mouse and/or virtual keyboard, and executes it on the local terminal to complete the control corresponding to the remote terminal;

Wherein, the operation instruction set includes operation instructions based on mouse and keyboard action attributes.
The method for remote collaboration based on video conference sharing auxiliary stream according to claim 1, characterized in that: in the step S50, converting and executing the operation instructions of the virtual mouse and/or virtual keyboard, specifically including:

S501, calling the SendInput function of the SDK interface to simulate the trigger event of the remote terminal;

S502, insert the SendInput function into the INPUT structure, connect it in series to the input stream of the keyboard or mouse, and send it to the thread;

S503. Listen to the trigger event in the thread, and generate an operation event of the virtual mouse and/or the virtual keyboard.
The method for remote collaboration based on video conference sharing auxiliary stream according to claim 2, characterized in that: in step S502, during the process of inserting the SendInput function into the INPUT structure, a user interface privilege isolation constraint is set to allow the application program to only Event input is injected into the application at or below the integrity level.
The method for remote collaboration based on video conference sharing auxiliary stream according to claim 2, characterized in that: in step S502, the SendInput function is inserted into the INPUT structure and connected in series to the input stream of the keyboard or mouse, specifically including:

S5021, determine whether it is a mouse event or a keyboard event, if it is a mouse event, go to step S5022, otherwise go to step S5023;

S5022, passing the coordinate position into the MOUSEINPUT structure;

S5023, passing the key value of the keyboard into the KEYBDINPUT structure.
The method for remote collaboration based on video conference sharing auxiliary stream according to claim 1, characterized in that: the operation instruction is composed of data in json format, including: event type, and a key value corresponding to the event type.
The method for remote collaboration based on video conference sharing auxiliary stream according to claim 5, characterized in that: said event types include: mouse button events, mouse movement events, mouse touch events and keyboard button events;

The key values corresponding to mouse button events include: left button, middle button and right button;

The key values corresponding to the mouse movement event include: screen width, screen height, mouse coordinate x and mouse coordinate y;

The key value corresponding to the mouse touch event includes: finger movement and finger coordinate information group;

The key value corresponding to the keyboard key event includes: keyboard key value.

Among them: the mouse coordinate x is the x coordinate of the mouse relative to the screen width; the mouse coordinate y is the y coordinate of the mouse relative to the screen height;

Finger actions include: the first finger is pressed, the last finger is released, the non-first finger is pressed, the non-last finger is released, and the finger moves;

The coordinate information group of the finger includes: the abscissa of the finger and the ordinate of the finger.
A system for remote collaboration based on video conferencing sharing auxiliary stream, comprising a local terminal (10), a remote terminal (20) and a video conferencing terminal (30), characterized in that:

The video conferencing terminal (30) is used to send the screen content on it to the local terminal (10) and the remote terminal (20) in the form of video auxiliary stream; including:

An operation instruction set (301), used to establish operation instructions based on mouse and keyboard action attributes;

An authorization module (302), configured to authorize the remote terminal (20) in response to the remote collaboration request of the application module (201), and establish a remote collaboration channel for the remote terminal (20) and the local terminal (10) after authorization;

Data parsing module (303), for receiving the data packet of data generating module (203) and parsing, after parsing out specific trigger event content, generate simulation operation instruction;

The local operation generating module (303) is used to convert the trigger event content parsed by the data parsing module (303) into operation instructions of the virtual mouse and/or virtual keyboard and display on the local display interface (101) of the local terminal (10) ;

Described remote terminal (20) comprises:

An application module (201), configured to initiate a remote collaboration request to a video conferencing terminal (30);

The remote operation module (202), on the received screen image (204), operates through the mouse and/or keyboard;

Data generation module (203), is used to generate corresponding trigger event content according to the operation instruction set with the operation of remote operation module (202) and sends to video conferencing terminal (30) after being encapsulated into a group of commands;

The screen image (204) is a display interface of the remote terminal, and is used for displaying content played on the screen.

The local terminal (10) includes:

The local display interface (101) is used to display the playback content of the video conferencing terminal (30).
A storage device, wherein a plurality of instructions are stored, wherein the instructions are adapted to be loaded by a processor and execute the remote collaboration method based on video conference sharing auxiliary stream according to any one of claims 1 to 6.