WO2022062623A1 - 远程控制方法、装置、设备、存储介质及系统 - Google Patents

远程控制方法、装置、设备、存储介质及系统 Download PDF

Info

Publication number
WO2022062623A1
WO2022062623A1 PCT/CN2021/107920 CN2021107920W WO2022062623A1 WO 2022062623 A1 WO2022062623 A1 WO 2022062623A1 CN 2021107920 W CN2021107920 W CN 2021107920W WO 2022062623 A1 WO2022062623 A1 WO 2022062623A1
Authority
WO
WIPO (PCT)
Prior art keywords
terminal device
data channel
secure data
remote control
control
Prior art date
Application number
PCT/CN2021/107920
Other languages
English (en)
French (fr)
Inventor
杨昊
刘飞
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Publication of WO2022062623A1 publication Critical patent/WO2022062623A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H04L65/1069—Session establishment or de-establishment

Definitions

  • the embodiments of the present application relate to the technical field of terminals, and in particular, to a remote control method, apparatus, device, storage medium, and system.
  • Embodiments of the present application provide a remote control method, apparatus, device, storage medium, and system.
  • the technical solution is as follows:
  • an embodiment of the present application provides a remote control method, which is applied to a first terminal device, and the method includes:
  • a secure data channel is established with the second terminal device through a signaling server, and the secure data channel is established based on web real-time communication (Web Real-Time Communication, WebRTC). ;
  • control data In response to the control operation in the screen after restoration, send control data to the second terminal device through the secure data channel, and the second terminal device is configured to respond to the control data in the screen according to the control data. the control operation.
  • an embodiment of the present application provides a remote control method, which is applied to a second terminal device, and the method includes:
  • a secure data channel is established with the first terminal device through a signaling server, and the secure data channel is established based on WebRTC;
  • an embodiment of the present application provides a remote control apparatus, which is applied to a first terminal device, and the apparatus includes:
  • a request sending module is used to send a remote control request to the second terminal device in an online state through the device online server;
  • a first channel establishment module configured to establish a secure data channel with the second terminal device through a signaling server in response to the second terminal device accepting the remote control request, and the secure data channel is established based on WebRTC;
  • a video data receiving module configured to receive video data sent by the second terminal device through the secure data channel, where the video data is obtained by encoding the captured screen image by the second terminal device;
  • a display module configured to decode the video data and restore and display the screen at the second terminal device
  • control data sending module configured to send control data to the second terminal device through the secure data channel in response to a control operation in the screen after restoration, and the second terminal device is configured to send control data to the second terminal device according to the control data responding to the control operation within the screen.
  • an embodiment of the present application provides a remote control apparatus, which is applied to a second terminal device, and the apparatus includes:
  • a request receiving module configured to receive a remote control request sent by the first terminal device through the device online server
  • a second channel establishment module configured to accept an operation in response to the request for the remote control request, establish a secure data channel with the first terminal device through a signaling server, and the secure data channel is established based on WebRTC;
  • a video data sending module configured to send video data to the first terminal device through the secure data channel, where the video data is obtained by encoding the collected screen images, and the first terminal device is used to send the video data to the first terminal device. Decode the video data, and restore and display the screen image at the second terminal device;
  • a control data receiving module configured to receive control data sent by the first terminal device through the secure data channel, where the control data is generated when the first terminal device receives a control operation in the restored screen ;
  • a control response module configured to respond to the control operation in the screen according to the control data.
  • an embodiment of the present application provides a terminal device, the terminal device includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the above-mentioned
  • the remote control method on the side of the first terminal device described in the aspect, or the remote control method on the side of the second terminal device described in the above aspect is implemented.
  • an embodiment of the present application provides a computer-readable storage medium, where the storage medium stores at least one instruction, and the at least one instruction is configured to be executed by a processor to implement the first terminal according to the above aspect
  • the remote control method on the device side, or the remote control method on the second terminal device side described in the above aspect is implemented.
  • an embodiment of the present application provides a remote control system, where the remote control system includes: a first terminal device, a second terminal device, a device online server, and a signaling server;
  • the first terminal device is respectively connected to the device online server and the signaling server through a wired or wireless network;
  • the second terminal device is respectively connected to the device online server and the signaling server through a wired or wireless network;
  • the first terminal device is configured to implement the remote control method on the side of the first terminal device according to the above aspect
  • the second terminal device is used to implement the remote control method on the side of the second terminal device as described in the above aspects.
  • an embodiment of the present application provides a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the remote control method provided by the above aspects.
  • FIG. 1 shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application
  • FIG. 2 shows a flowchart of a remote control method provided by an exemplary embodiment of the present application
  • FIG. 3 is an implementation schematic diagram of an implementation process of a remote control method provided by an exemplary embodiment of the present application
  • FIG. 4 shows a flowchart of a remote control method provided by another exemplary embodiment of the present application.
  • FIG. 5 is a schematic diagram of the implementation of a touch position mapping process according to an exemplary embodiment of the present application.
  • FIG. 6 shows a flowchart of a remote control method provided by another exemplary embodiment of the present application.
  • FIG. 7 is a schematic diagram of the implementation of a signaling service acquisition process shown in an exemplary embodiment of the present application.
  • FIG. 8 is a schematic diagram of the implementation of a process of establishing a secure data channel according to an exemplary embodiment of the present application.
  • FIG. 9 shows a flowchart of a remote control method provided by another exemplary embodiment of the present application.
  • FIG. 10 shows a structural block diagram of a remote control device provided by an embodiment of the present application.
  • FIG. 11 shows a structural block diagram of a remote control device provided by another embodiment of the present application.
  • FIG. 12 shows a structural block diagram of a terminal device provided by an exemplary embodiment of the present application.
  • plural refers to two or more.
  • “And/or”, which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone.
  • the character “/” generally indicates that the associated objects are an "or" relationship.
  • FIG. 1 shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application.
  • the implementation environment includes a first terminal device 110 , a second terminal device 120 , a device online server 130 and a signaling server 140 .
  • the first terminal device 110 and the second terminal device 120 are electronic devices with remote control functions, and the electronic devices may include smart phones, tablet computers, smart TVs, personal computers or wearable devices, etc.
  • the type is not limited.
  • the remote control function of the terminal device can be implemented through an installed remote control application program, or through a web page.
  • one of the first terminal device 110 and the second terminal device 120 is a control terminal, and the other terminal device is a controlled terminal.
  • the controlling terminal is a terminal device that initiates remote control
  • the controlled terminal is a terminal device that accepts remote control.
  • the controlling terminal and the controlled terminal can be switched.
  • both parties performing remote control may be terminal devices of the same platform (or type), or may be terminal devices of different platforms, that is, cross-platform remote control may be implemented.
  • a user can use a smartphone to remotely control other smartphones; or, a user can use a smartphone to remotely control a smart TV or a wearable device.
  • the embodiments of the present application do not limit the platforms of both remote control parties.
  • the device online server 130 is a server for providing device discovery and message notification services, which may be one server, a server cluster composed of several servers, or a cloud computing center.
  • the terminal devices 110 and the second terminal device 120 need to perform remote control, both need to be connected to the device online server 130, wherein the terminal devices can establish a connection with the device online server 130 through a wired or wireless network.
  • the signaling server 140 is a server used to provide information exchange services for both remote control parties, thereby assisting the remote control parties to establish a secure data channel, which may be a server, a server cluster composed of several servers, or a cloud computing center.
  • both the first terminal device 110 and the second terminal device 120 need to connect to the signaling server 140, and send a message to the opposite end through the signaling server 140, so that the first terminal device 110 and the second terminal device 120 can communicate with each other. Establish a secure data channel between them.
  • the terminal device may establish a connection with the signaling server 140 through a wired or wireless network.
  • the secure data channel is established based on WebRTC, and is used to transmit video data sent by the controlled terminal and control data sent by the control terminal, and can realize low-latency data transmission and improve the real-time performance of remote control.
  • the first terminal device 110 and the second terminal device 120 are located in different places. As shown in FIG. 1 , the first terminal device 110 is located in an office, and the second terminal device 120 is located in a home. When the first terminal device 110 needs to be used to remotely control the second terminal device 120, the first terminal device 110 Start the remote control application to connect to the device presence server 130 .
  • the first terminal device 110 After connecting to the device online server 130 , the first terminal device 110 sends a remote control request to the second terminal device 120 through the device online server 130 .
  • the first terminal device 110 and the second terminal device 120 further establish a secure data channel 150 through the signaling server 140 .
  • the second terminal device 120 encodes the screen image (and the audio on the side of the second terminal device 120 ), and sends the encoded video data to the first terminal device 110 through the secure data channel 150 , which is decoded and displayed by the first terminal device 110 , so that the user can view the real-time display content of the second terminal device 120 through the first terminal device 110 .
  • the user can perform a control operation in the content displayed by the first terminal device 110, and the first terminal device 110 transmits the control data corresponding to the control operation (through the secure data channel 150) to the second terminal device 120, so that the second The terminal device 120 responds to the control operation according to the control data, changes the currently displayed content, and updates the content to the first terminal device 110 through the secure data channel 150 .
  • the remote control methods provided in the embodiments of the present application can also be applied to remote maintenance scenarios, that is, after-sales engineers can configure or debug the sold terminal devices through the terminal devices.
  • the scene is not limited.
  • FIG. 2 shows a flowchart of a remote control method provided by an exemplary embodiment of the present application. This embodiment is described by taking the method applying the implementation environment shown in FIG. 1 as an example, and the method may include the following steps .
  • Step 201 the first terminal device sends a remote control request to the second terminal device in an online state through the device online server.
  • the first terminal device is a control terminal
  • the second terminal device is a controlled terminal.
  • the first terminal device when the first terminal device needs to control the second terminal device, the first terminal device first connects to the device online server, and then sends a remote control request to the device online server, where the remote control request contains Contains the device identifier of the second terminal device.
  • the remote control request contains Contains the device identifier of the second terminal device.
  • different terminals correspond to different device identifiers, and the device identifiers are allocated by the device online server.
  • the device online server receives After the remote control request is received, according to the device identification of the second terminal device contained therein, it is queried whether the second terminal device is in an online state. If it is in an online state, send a remote control request to the second terminal device; if it is in an offline state, send a device offline message to the first terminal device, so that the first terminal device user prompts the second terminal device user to put the second terminal device The device is connected to the device online server.
  • the first terminal device 31 when the first terminal device 31 needs to remotely control the second terminal device 32 , it connects to the device online server by starting the remote control application program.
  • the first terminal device 31 When receiving the selection operation of the device identification corresponding to the second terminal device 32 in the device selection interface 311 , the first terminal device 31 sends a remote control request to the second terminal device 32 through the device online server.
  • Step 202 the second terminal device receives the remote control request sent by the first terminal device through the device online server.
  • the second terminal device when receiving a remote control request sent by the device online server, displays a remote control prompt window, and receives a remote control response (accept or reject response) through the remote control prompt window.
  • the second terminal device 32 in the online state displays a remote control prompt window 321 .
  • Step 203 in response to the request accepting operation for the remote control request, the second terminal device establishes a secure data channel with the first terminal device through the signaling server, and the secure data channel is established based on WebRTC.
  • the second terminal device when receiving the request to accept the operation for the remote control request, connects to the signaling server through the Web Socket Secure (WSS) protocol, so as to communicate with the first terminal through the signaling server.
  • WSS Web Socket Secure
  • the device exchanges information, and then establishes a secure data channel with the first terminal device.
  • the first terminal device and the second terminal device establish a secure data channel based on WebRTC.
  • the second terminal device 32 when a click operation on an accept control in the remote control prompt window 321 is received, the second terminal device 32 connects to the signaling server and establishes a secure data channel with the first terminal device 31 .
  • Step 204 in response to the second terminal device accepting the remote control request, the first terminal device establishes a secure data channel with the second terminal device through the signaling server, and the secure data channel is established based on WebRTC.
  • the second terminal device after the second terminal device accepts the remote control request, it sends a request acceptance response to the device online server, and the device online server notifies the first terminal device after receiving the request acceptance response, so that the first terminal device Connect to the signaling server, and then establish a secure data channel with the second terminal device through the signaling service provided by the signaling server.
  • Step 205 the second terminal device sends video data to the first terminal device through the secure data channel, and the video data is obtained by encoding the collected screen images.
  • the second terminal device After the secure data channel is established, in order to be able to display the screen on the side of the second terminal device at the first terminal device so that subsequent users can perform remote control through the first terminal device, the second terminal device captures the screen image and encodes the screen image The video data is obtained, so that the video data is sent to the first terminal device (through the secure data channel) in the form of a video stream.
  • the second terminal device performs audio collection (including system audio or external audio collected by a microphone, and performs audio mixing processing) while collecting the screen image, and encodes the collected audio to obtain audio data, thereby converting the audio.
  • the data and video data are sent to the first terminal device synchronously.
  • the second terminal device 32 encodes the currently collected screen image 322 to obtain video data, and transmits the video data to the first terminal device 31 through a secure data channel.
  • Step 206 the first terminal device receives the video data sent by the second terminal device through the secure data channel.
  • the first terminal device receives the audio data sent by the second terminal device through the secure data channel.
  • Step 207 The first terminal device decodes the video data, and restores and displays the screen image at the second terminal device.
  • the first terminal device decodes the received video data to obtain continuous video frames, so as to display the video frames to achieve the effect of restoring and displaying the screen of the second terminal device on the side of the first terminal device.
  • the first terminal device displays the decoded and restored screen image in full screen, or the first terminal device displays and displays the decoded and restored screen image in a window, and the size of the window is adjustable.
  • the first terminal device decodes and plays the audio data, and restores the audio collected by the second terminal device.
  • the first terminal device 31 decodes the video data and displays the restored screen image 312 .
  • Step 208 in response to the control operation in the restored screen, the first terminal device sends control data to the second terminal device through the secure data channel.
  • the first terminal device and the second terminal device display the same picture. Since the first terminal device displays the decoded video picture, the actual operation authority for the content displayed on the screen is still in the second terminal device.
  • the first terminal device needs to send the control data corresponding to the control operation to the second terminal device, and the second terminal device controls the control operation according to the control data. Action responds.
  • control operation includes at least one of touch operation, peripheral input operation (such as moving and clicking by controlling a mouse, and text input through a keyboard), voice control operation, or gesture operation.
  • peripheral input operation such as moving and clicking by controlling a mouse, and text input through a keyboard
  • voice control operation or gesture operation.
  • the first terminal device 31 when the user performs a touch operation on the font size adjustment control in the restored screen 312 (adjusting the font size from standard to extra large), the first terminal device 31 generates a touch control operation. The corresponding control data is operated, and the control data is sent to the second terminal device 32 .
  • Step 209 the second terminal device receives the control data sent by the first terminal device through the secure data channel.
  • Step 210 the second terminal device responds to the control operation in the screen according to the control data.
  • the second terminal device parses the received control data, restores the control operation received by the first terminal device on the second terminal device side according to the parsing result, and then responds to the control operation (screen image after the response). subject to change). It should be noted that, during the process of responding to the control operation, the second terminal device still sends video data to the first terminal device through the secure data channel to ensure the consistency of the displayed images at the first terminal device and the second terminal device.
  • the second terminal device 32 adjusts the font size adjustment control in the screen image 322 according to the received control data to restore the touch operation of the user on the first terminal device 31 .
  • the children can use the mobile phone to remotely control the mobile phone used by the old man, so as to increase the font size of the mobile phone used by the old man, and there is no need to operate the mobile phone used by the old man on site.
  • the first terminal device initiates a remote control request to the second terminal device through the device online server, and establishes a WebRTC-based secure data channel with the second terminal device by means of the signaling server, thereby using
  • the secure data channel receives the video data sent by the second terminal device, restores the screen at the second terminal device, and sends control data to the second terminal device through the secure data channel according to the received control operation, so as to realize the Remote control of two terminal devices;
  • the embodiments of the present application implement screen sharing and remote control between terminal devices based on WebRTC, which is applicable to terminal devices on different platforms, and realizes low-latency transmission of data based on WebRTC, which helps to improve remote control performance. real-time.
  • control operation includes a touch operation
  • the first terminal device sends control data to the second terminal device through the secure data channel, including:
  • the first terminal device In response to the control operation in the restored screen, the first terminal device generates control data according to the operation information of the control operation, and the operation information at least includes the operation type and the operation position;
  • the first terminal device sends control data to the second terminal device through the secure data channel, and the second terminal device is configured to respond to the control operation in the screen according to the control data.
  • the first terminal device generates control data according to the operation information of the control operation, including:
  • the control data is generated according to the operation information of the control operation and the first screen size, and the second terminal device is used to generate the control data according to the first screen size.
  • a screen size and a second screen size are used to map the operation position.
  • the first terminal device in response to the second terminal device accepting the remote control request, establishes a secure data channel with the second terminal device through the signaling server, including:
  • the first terminal device and the second terminal device In response to the second terminal device accepting the remote control request, acquiring the room identifier generated by the device online server, the first terminal device and the second terminal device have the same room identifier;
  • the first terminal device obtains the signaling service provided by the signaling server based on the room identifier, and the signaling service refers to that the signaling server provides a message forwarding service for terminal devices with the same room identifier;
  • the first terminal device establishes a secure data channel with the second terminal device through the signaling service.
  • the first terminal device establishes a secure data channel with the second terminal device through a signaling service, including:
  • the first terminal device sends a first message to the second terminal device through the signaling service.
  • the first message includes the first network port information, the first user name, the first password, and the first certificate verification fingerprint of the first certificate.
  • the first The user name and the first password are randomly generated by the first terminal device;
  • the first terminal device receives the second message sent by the second terminal device through the signaling service.
  • the second message includes the second network port information, the second user name, the second password, and the second certificate verification fingerprint of the second certificate.
  • the second user name and the second password are randomly generated by the second terminal device;
  • the first terminal device establishes a secure data channel according to the first message and the second message.
  • the security data channel includes a first security data channel and a second security data channel, the first security data channel is used for transmitting control data, and the second security data channel is used for transmitting video data;
  • the first terminal device establishes a secure data channel according to the first message and the second message, including:
  • a first secure data channel is established between the first network port and the second network port based on the first user name, the first password and the second certificate verification fingerprint.
  • the first terminal device decodes the video data, and restores and displays the screen at the second terminal device, including:
  • the first terminal device decrypts the video data by using a preset encryption key, and the preset encryption key is set by the second terminal device when the second secure data channel is established;
  • the first terminal device decodes the decrypted video data, and restores and displays the screen at the second terminal device.
  • the first terminal device sends a remote control request to the second terminal device in an online state through the device online server, including:
  • the first terminal device displays the device identification of at least one candidate terminal device, and the candidate terminal device is a historical terminal device that has been remotely controlled with the first terminal device and is in an online state; in response to the second terminal device in the at least one candidate terminal device The selection operation of the device sends a remote control request to the second terminal device through the device online server;
  • the first terminal device obtains the input target device identification, and the target device identification is the device identification of the second terminal device; sends a remote control request including the target device identification to the device online server, and the device online server is used to query the second terminal according to the target device identification.
  • the second terminal device responds to the control operation on the screen according to the control data, including:
  • the second terminal device generates an operation event according to the operation type and operation position in the control data
  • the second terminal device reports the operation event to the operating system, and the operating system is used to respond to the operation event.
  • control data further includes a first screen size, where the first screen size is the size of the screen after restoration;
  • the second terminal device generates a touch event according to the operation type and operation position in the control data, including:
  • the second terminal device performs position mapping on the operation position according to the first screen size and the second screen size of the screen image at the second terminal device;
  • the second terminal device generates an operation event according to the operation type and the mapped operation position.
  • the second terminal device in response to the request accepting operation for the remote control request, establishes a secure data channel with the second terminal device through the signaling server, including:
  • the second terminal device obtains the room identifier generated by the device online server, and the first terminal device and the second terminal device have the same room identifier;
  • the second terminal device obtains the signaling service provided by the signaling server based on the room identifier, and the signaling service refers to that the signaling server provides a message forwarding service for terminal devices with the same room identifier;
  • the second terminal device establishes a secure data channel with the first terminal device through the signaling service.
  • a signaling service including:
  • the second terminal device receives the first message sent by the first terminal device through the signaling service.
  • the first message includes the first network port information, the first user name, the first password, and the first certificate verification fingerprint of the first certificate.
  • a username and a first password are randomly generated by the first terminal device;
  • the second terminal device sends a second message to the first terminal device through the signaling service.
  • the second message includes the second network port information, the second user name, the second password, and the second certificate verification fingerprint of the second certificate.
  • the second The user name and the second password are randomly generated by the second terminal device;
  • the second terminal device establishes a secure data channel according to the first message and the second message.
  • the security data channel includes a first security data channel and a second security data channel, the first security data channel is used for transmitting control data, and the second security data channel is used for transmitting video data;
  • the second terminal device establishes a secure data channel according to the first message and the second message, including:
  • the second terminal device verifies the fingerprint based on the second user name, the second password and the first certificate, and establishes a second secure data channel between the second network port and the first network port.
  • the second terminal device sends video data to the first terminal device through the secure data channel, including:
  • the second terminal device encrypts the video data by using a preset encryption key
  • the second terminal device sends the encrypted video data to the first terminal device through the second secure data channel, and the first terminal device is configured to decrypt the received video data by using a preset encryption key.
  • FIG. 4 shows a flowchart of a remote control method provided by another exemplary embodiment of the present application.
  • Step 401 in response to the control operation in the restored screen, the first terminal device generates control data according to operation information of the control operation, where the operation information at least includes the operation type and the operation position.
  • the first terminal device when the first terminal device has a touch function, if a touch operation located on the restored screen is received, the first terminal device obtains touch information corresponding to the touch operation, so as to obtain the touch information based on the touch operation.
  • the touch information generates control data.
  • the touch information includes a touch type and a touch position
  • the touch type includes at least one of click, double-click, long-press, slide, drag or press
  • the touch position includes a touch starting point Position, touch end position (for swipe, drag), or at least one mid-touch position (for swipe, drag).
  • the touch position is represented by the coordinates of the touch operation in the restored screen image.
  • the first terminal device 51 receives the video data (obtained by encoding the captured screen image 521 ) sent by the second terminal device 52 , the restored screen image is displayed in the window 511 .
  • the first terminal device 51 acquires the click position (x, y) of the click operation, and determines the touch-type drag.
  • the first terminal device and the second terminal device belong to different platforms (such as using a smartphone to control a smart TV or wearable device), or belong to the same platform but have different models, or the user adjusts the display of the screen at the first terminal device When the size of the screen image at the first terminal device is different from the image size of the screen image at the second terminal device. At this time, if the second terminal device directly responds to the touch according to the touch position included in the control data, a response error will occur.
  • the first picture size of the restored screen image (that is, the display size at the first terminal device) is the same as the second size of the screen image at the second terminal device.
  • the first terminal device When the screen sizes are different, the first terminal device generates control data according to the touch information of the touch operation and the first screen size, so that the second terminal device maps the touch position at the first terminal device to the screen image of the second terminal device middle.
  • the second picture size is obtained by parsing from the video data by the first terminal device.
  • the first terminal device may also send a notification message to the second terminal device through the secure data channel when detecting that the size of the first screen changes (which may be caused by the user resizing the window).
  • the embodiment does not limit this.
  • the size of the window 511 at the first terminal device 51 is a ⁇ b (the size of the first screen is also a ⁇ b), and the control data sent to the second terminal device 52 includes the first Screen size a ⁇ b, touch type “1” (indicating click operation), and touch position (x, y).
  • the first terminal device may perform control according to an external input and output device (such as a mouse and keyboard) to control the screen image, which will not be repeated in this embodiment.
  • an external input and output device such as a mouse and keyboard
  • Step 402 the first terminal device sends control data to the second terminal device through the secure data channel.
  • Step 403 the second terminal device receives the control data sent by the first terminal device through the secure data channel.
  • Step 404 the second terminal device generates a touch event according to the operation type and operation position in the control data.
  • the second terminal device After receiving the control data, the second terminal device parses and obtains the operation type and the operation position contained in the control data, thereby generating an operation event.
  • the operation event is a touch event
  • the operation event is an event that is simulated (reported) to the operating system.
  • the second terminal device when it is parsed that the control data includes not only the operation type and the operation position, but also the first screen size, when generating the operation position in the operation event, the second terminal device according to the first screen The size of the screen image and the second screen size of the screen image at the second terminal device are used to map the operation position, so as to generate operation events according to the operation type and the mapped operation position, so as to ensure the accuracy of the control operation.
  • Step 405 the second terminal device reports the operation event to the operating system, and the operating system is used to respond to the operation event.
  • the second terminal device reports the operation event to the operating system, simulating that the second terminal device receives the control operation, and the operating system responds to the operation event.
  • the second terminal device 52 clicks the control arrow 522 on the No. Icons in the second row and the third column.
  • the first terminal device sends the first screen size of the local screen to the second terminal device, so that the second terminal device performs position mapping on the touch position based on the first screen size and the second screen size, so as to avoid
  • the abnormal control problem caused by the inconsistent screen size of the terminal equipment improves the control accuracy of the remote control.
  • the first terminal device and the second terminal device need to obtain the same room ID from the device online server, and use the information provided by the signaling server based on the room ID. order service, thereby establishing a secure data channel. Exemplary embodiments are used for description below.
  • the method may include the following steps.
  • Step 601 the first terminal device sends a remote control request to the second terminal device in an online state through the device online server.
  • the first terminal device displays the device identifier of at least one candidate terminal device, and the candidate terminal device has performed remote control with the first terminal device and is in an online state
  • the history terminal equipment in response to the selection operation of the second terminal equipment in the at least one candidate terminal equipment, send a remote control request to the second terminal equipment through the equipment online server.
  • the first terminal device After the first terminal device is connected to the device online server, it sends a device online status query request to the device online server, requesting the device online server to query the device status (including online status and offline status) of historical terminal devices (which have been remotely controlled). condition). After the device online server feeds back the queried device state to the first terminal device, the first terminal device displays the historical terminal devices that are in the online state, so that the user can choose from them.
  • the first terminal device when the first terminal device performs remote control with the second terminal device for the first time, the first terminal device obtains the input target device identifier, and the target device identifier is the device identifier of the second terminal device; Send a remote control request containing the target device ID to the device online server.
  • the device online server queries the device state of the second terminal device according to the target device identifier, and sends a remote control request to the second terminal device when the second terminal device is in the online state.
  • the first terminal device may also use other methods (such as clicking on the remote control link) to trigger the sending of the remote control request, which is not limited in this embodiment of the present application.
  • Step 602 the second terminal device receives the remote control request sent by the first terminal device through the device online server.
  • Step 603 in response to the request accepting operation for the remote control request, the second terminal device acquires the room identifier generated by the device online server, and the first terminal device and the second terminal device have the same room identifier.
  • the second terminal device when the first terminal device is allowed to perform remote control, the second terminal device sends a request-accept response to the device online server, and after the device online server receives the request-accept response, it is the first terminal device and the device online server.
  • the second terminal device is assigned the same and unique room ID.
  • the device online server obtains the room identifier from the resource pool and assigns it to the first terminal device and the second terminal device, so as to ensure that different terminal devices performing different remote controls at the same time have different room identifiers.
  • Step 604 in response to the second terminal device accepting the remote control request, the first terminal device obtains the room identifier generated by the device online server, and the first terminal device and the second terminal device have the same room identifier.
  • the first terminal device receives the room identifier allocated by the device online server, it determines that the second terminal device has accepted the remote control.
  • Step 605 the first terminal device obtains the signaling service provided by the signaling server based on the room identifier.
  • Step 606 the second terminal device acquires the signaling service provided by the signaling server based on the room identifier.
  • the signaling server only provides signaling services for terminal devices with the same room identifier, where the signaling service refers to the signaling server for the terminal devices with the same room identifier.
  • Terminal devices provide message forwarding services, that is, only terminal devices with the same room ID can establish a secure data channel through signaling services.
  • the first terminal device after acquiring the room identifier, connects to the signaling server through the room identifier, and the second terminal device also connects to the signaling server through the room identifier.
  • the signaling server verifies the room identifiers of the first terminal device and the second terminal device, and provides signaling services for the first terminal device and the second terminal device when the two are consistent.
  • both the first terminal device 71 and the second terminal device 72 log in to the device online server 73 .
  • the first terminal device 71 sends a remote control request to the second terminal device 72 through the device online server 73.
  • the device online server 73 is the first terminal device 71 and the second terminal.
  • Device 72 assigns the same room identification. After acquiring the room identifier, the first terminal device 71 and the second terminal device 72 further establish a secure data channel between the devices through the signaling service provided by the signaling server 74 based on the room identifier.
  • Step 607 The first terminal device sends a first message to the second terminal device through the signaling service, where the first message includes the first network port information, the first user name, the first password, and the first certificate verification fingerprint of the first certificate , the first user name and the first password are randomly generated by the first terminal device.
  • the first terminal device opens a first network port (IP port) , and send a first message including the first network port information, the first user name, the first password, and the first certificate verification fingerprint corresponding to the first certificate through the signaling service.
  • IP port IP port
  • the first user name and the first password are randomly generated by the first terminal device, that is, each time the remote control is performed, the first user name and the first password in the first message sent by the first terminal device are different. Even if the user name and password used by the first terminal device for remote control this time are leaked, the user name and password cannot be used for the next remote control, which improves the security of remote control.
  • the first certificate is used for a secure transport layer protocol (Transport Layer Security, TLS) connection.
  • the first message sent by the first terminal device 71 to the second terminal device 72 through the signaling service includes the IP port a, the user name user1 , the password pwd1 , and the fingerprint fingerprint1 of the certificate cer1 .
  • Step 608 the second terminal device receives the first message sent by the first terminal device through the signaling service.
  • the signaling server forwards the first message sent by the first terminal device to the second terminal device with the same room identifier, and correspondingly, the second terminal device receives the first message forwarded by the signaling server .
  • Step 609 the second terminal device sends a second message to the first terminal device through the signaling service, where the second message includes the second network port information, the second user name, the second password, and the second certificate verification fingerprint of the second certificate , the second user name and the second password are randomly generated by the second terminal device.
  • the second terminal device first opens the second network port (IP port), and sends information including the second network port information, the second user name, the second password and the The second certificate corresponds to the second message of the second certificate verification fingerprint.
  • IP port IP port
  • the second user name and the second password are randomly generated by the second terminal device, that is, each time the remote control is performed, the second user name and the second password in the second message sent by the second terminal device are different, so as to avoid using fixed The security risks caused when the username and password are leaked.
  • the second certificate is also used for a secure transport layer protocol (Transport Layer Security, TLS) connection.
  • the second message sent by the second terminal device 72 to the first terminal device 71 through the signaling service includes the IP port b, the user name user2, the password pwd2, and the fingerprint fingerprint2 of the certificate cer2.
  • first message and the second message only takes the above information contained in the first message and the second message as an example for description.
  • the first message and the second message may also include other information used when establishing a secure data channel.
  • the required information is not limited in this embodiment.
  • Step 610 the first terminal device receives the second message sent by the second terminal device through the signaling service.
  • the signaling server forwards the second message sent by the second terminal device to the first terminal device with the same room identifier, and correspondingly, the first terminal device receives the second message forwarded by the signaling server .
  • the first terminal device establishes a secure data channel with the second terminal device through the signaling server according to the first message and the second message; the second terminal device communicates with the first terminal device through the signaling server according to the first message and the second message.
  • the terminal device establishes a secure data channel.
  • a first secure data channel for transmitting video data and a second secure data channel for transmitting control data are established between the first terminal device and the second terminal device.
  • Step 611 the first terminal device verifies the fingerprint based on the first user name, the first password and the second certificate, and establishes a first secure data channel between the first network port and the second network port.
  • the first terminal device when the first terminal device requests to establish a first secure data channel with the second terminal device through the signaling service, it sends a message including the first user name, the first password and the second certificate to the second terminal device Validate the channel establishment request for the fingerprint.
  • the second terminal device detects whether the first user name and the first password are consistent with the user name and password included in the first message, and the second certificate verifies whether the fingerprint matches the second certificate. If the user name and the password are the same and the certificate verification fingerprint matches, a first secure data channel is established between the first network port of the first terminal device and the second network port of the second terminal device.
  • the first secure data channel is a TLS secure connection.
  • the first terminal device 71 establishes a first secure data channel between the IP port a and the IP port b based on user1 , pwd1 and fingerprint2 .
  • the first terminal device sends the control data to the second terminal device through the first secure data connection.
  • Step 612 the second terminal device verifies the fingerprint based on the second user name, the second password and the first certificate, and establishes a second secure data channel between the second network port and the first network port.
  • the second terminal device requests to establish a second secure data channel with the first
  • Two user name, second password and first certificate verify the channel establishment request of fingerprint.
  • the first terminal device detects whether the second user name and the second password are consistent with the user name and password included in the second message, and the first certificate verifies whether the fingerprint matches the first certificate. If the user name and the password are consistent and the certificate verification fingerprint matches, a second secure data channel is established between the second network port of the second terminal device and the first network port of the first terminal device.
  • the second terminal device 72 establishes a second secure data channel between IP port b and IP port a based on user2 , pwd2 and fingerprint1 .
  • the second terminal device sends the video data to the first terminal device through the second secure data connection.
  • the first terminal device and the second terminal device exchange control rights.
  • the first terminal device and the second terminal device when establishing a secure data channel, the first terminal device and the second terminal device The second terminal device also establishes a third secure data channel and a fourth secure data channel, wherein the first terminal device is used to transmit video data to the second terminal device through the third secure data channel, and the second terminal device is used to transmit video data to the second terminal device through the fourth The secure data channel transmits control data to the first terminal device.
  • the third secure data channel is established by the first terminal device based on the first user name, the first password and the second certificate verification fingerprint
  • the fourth secure data channel is established by the second terminal device based on the second user name, the second password And the first certificate verification fingerprint is established.
  • the first terminal device and the second terminal device switch to use the third secure data channel and the fourth secure data channel for data transmission.
  • the second terminal device and the first terminal device are based on the second user name, second The password and the first certificate verify the fingerprint, and after establishing the second secure data channel with the first terminal device, a preset encryption key is generated, and the preset encryption key is transmitted to the first terminal device through the second secure data channel.
  • the video data is encrypted by using the preset encryption key, and the encrypted video data is transmitted on the second secure data channel based on the Secure Real-time Transport Protocol (SRTP) protocol.
  • SRTP Secure Real-time Transport Protocol
  • the video data transmission process between the first terminal device and the second terminal device may include the following steps.
  • Step 901 the second terminal device encrypts the video data by using a preset encryption key.
  • the second terminal device Before transmitting the video data to the first terminal device, the second terminal device encrypts the video data using the preset encryption key set in the secure data channel establishment stage to obtain encrypted video data.
  • Step 902 the second terminal device sends the encrypted video data to the first terminal device through the second secure data channel.
  • the second terminal device sends the encrypted video data to the first terminal device through the second secure data channel with the first terminal device.
  • the encrypted video data is transmitted based on the SRTP secure transmission protocol, which reduces data transmission delay while ensuring data transmission security.
  • Step 903 the first terminal device decrypts the video data by using a preset encryption key.
  • the first terminal device after receiving the video data through the second secure data channel, the first terminal device first decrypts the video data by using the preset encryption key to obtain the decrypted video data.
  • Step 904 the first terminal device decodes the decrypted video data, and restores and displays the screen image at the second terminal device.
  • the process of decoding the decrypted video data by the first terminal device and restoring the displayed screen image may refer to the above-mentioned embodiment, and details are not described herein again in this embodiment.
  • the equipment online server allocates the same room ID to both remote control parties, so that both remote control parties obtain the signaling service provided by the signaling server based on the unified room ID, thus ensuring the security of the remote control process;
  • the control dual-transmission establishes a secure data channel based on the randomly generated user name and password, avoids the potential security risks caused by the leakage of the user name and password when using a fixed user name and password, and further improves the security of the remote control process.
  • a first secure data channel for transmitting video data and a second secure data channel for transmitting control data are established between the first terminal device and the second terminal device, so as to avoid video during the remote control process.
  • the mutual influence caused by the simultaneous transmission of data and control data helps to reduce the transmission delay of video data and control data and improve the real-time performance of remote control.
  • the step of taking the first terminal device as the execution body can be independently implemented as a remote control method on the side of the first terminal device, and the step of using the second terminal device as the execution body can be independently realized and called. It is a remote control method on the side of the second terminal device, which is not repeated in this embodiment of the present application.
  • FIG. 10 shows a structural block diagram of a remote control apparatus provided by an embodiment of the present application.
  • the apparatus can be implemented as all or a part of the first terminal device through software, hardware or a combination of the two.
  • the device includes:
  • a request sending module 1001 configured to send a remote control request to the second terminal device in an online state through the device online server;
  • a first channel establishment module 1002 configured to establish a secure data channel with the second terminal device through a signaling server in response to the second terminal device accepting the remote control request, and the secure data channel is established based on WebRTC;
  • a video data receiving module 1003 configured to receive video data sent by the second terminal device through the secure data channel, where the video data is obtained by encoding the captured screen images by the second terminal device;
  • a display module 1004 configured to decode the video data and restore and display the screen image at the second terminal device
  • a control data sending module 1005 configured to send control data to the second terminal device through the secure data channel in response to a control operation in the screen after restoration, and the second terminal device is configured to send control data according to the control data Data is responsive to the control operation within the screen.
  • control data sending module 1005 includes:
  • control data generating unit configured to generate the control data according to the operation information of the control operation in response to the control operation in the screen after restoration, the operation information at least including the operation type and the operation position;
  • control data sending unit configured to send the control data to the second terminal device through the secure data channel, and the second terminal device is configured to respond to the control operation in the screen according to the control data .
  • control data generating unit is used for:
  • the second terminal device In response to the restored first screen size of the screen image being different from the second screen size of the screen image at the second terminal device, according to the operation information of the control operation and the first screen size
  • the control data is generated, and the second terminal device is configured to perform position mapping on the operation position according to the first screen size and the second screen size.
  • the first channel establishment module 1002 includes:
  • a room identifier acquiring unit configured to acquire the room identifier generated by the device online server in response to the second terminal device accepting the remote control request, the first terminal device and the second terminal device have the same room identifier ;
  • a signaling service acquiring unit configured to acquire the signaling service provided by the signaling server based on the room identifier, where the signaling service refers to that the signaling server provides a message forwarding service for terminal devices with the same room identifier;
  • a first channel establishing unit configured to establish the secure data channel with the second terminal device through the signaling service.
  • the first channel establishment unit is used for:
  • the first user name and the first password are randomly generated by the first terminal device;
  • a second message sent by the second terminal device is received through the signaling service, where the second message includes second network port information, a second user name, a second password, and a second certificate verification fingerprint of the second certificate , the second user name and the second password are randomly generated by the second terminal device;
  • the secure data channel is established based on the first message and the second message.
  • the secure data channel includes a first secure data channel and a second secure data channel, the first secure data channel is used to transmit the control data, and the second secure data channel is used to transmit the video data;
  • the first channel establishment unit is also used for:
  • the first secure data channel is established between the first network port and the second network port based on the first username, the first password and the second certificate verification fingerprint.
  • the display module 1004 is used for:
  • the preset encryption key is set by the second terminal device when the second secure data channel is established;
  • the decrypted video data is decoded, and the screen image at the second terminal device is restored and displayed.
  • the request sending module 1001 is used for:
  • the candidate terminal device is a historical terminal device that has been remotely controlled with the first terminal device and is in the online state; in response to the at least one candidate terminal device For the selection operation of the second terminal device, the remote control request is sent to the second terminal device through the device online server;
  • Obtain the input target device identification which is the device identification of the second terminal device; send the remote control request including the target device identification to the device online server, and the device online server is used for Query the device status of the second terminal device according to the target device identifier, and send the remote control request to the second terminal device when the second terminal device is in the online state.
  • FIG. 11 shows a structural block diagram of a remote control apparatus provided by another embodiment of the present application.
  • the apparatus may be implemented by software, hardware or a combination of the two to become all or a part of the second terminal device.
  • the device includes:
  • the second channel establishment module 1102 is configured to, in response to the request accepting operation for the remote control request, establish a secure data channel with the first terminal device through a signaling server, and the secure data channel is established based on WebRTC;
  • the video data sending module 1103 is configured to send video data to the first terminal device through the secure data channel, where the video data is obtained by encoding the captured screen images, and the first terminal device is used to send the video data to the first terminal device.
  • the video data is decoded, and the screen picture at the second terminal device is restored and displayed;
  • the control data receiving module 1104 is configured to receive the control data sent by the first terminal device through the secure data channel, and the control data is received by the first terminal device when the control operation in the restored screen is received generate;
  • the control response module 1105 is configured to respond to the control operation in the screen according to the control data.
  • the control response module 1105 includes:
  • an event generation unit configured to generate an operation event according to the operation type and operation position in the control data
  • An event reporting unit configured to report the operation event to the operating system, and the operating system is configured to respond to the operation event.
  • control data further includes a first screen size, where the first screen size is the size of the screen after restoration;
  • the event generating unit is used for:
  • the operation event is generated according to the operation type and the mapped operation position.
  • the second channel establishment module 1102 includes:
  • a room identifier acquiring unit configured to acquire the room identifier generated by the device online server in response to the request accepting operation for the remote control request, the first terminal device and the second terminal device have the same room identifier ;
  • a signaling service acquiring unit configured to acquire the signaling service provided by the signaling server based on the room identifier, where the signaling service refers to that the signaling server provides a message forwarding service for terminal devices with the same room identifier;
  • a second channel establishing unit configured to establish the secure data channel with the first terminal device through the signaling service.
  • the second channel establishment unit is used for:
  • the first message includes the first network port information, the first user name, the first password, and the first certificate verification fingerprint of the first certificate , the first user name and the first password are randomly generated by the first terminal device;
  • the second user name and the second password are randomly generated by the second terminal device;
  • the secure data channel is established based on the first message and the second message.
  • the secure data channel includes a first secure data channel and a second secure data channel, the first secure data channel is used to transmit the control data, and the second secure data channel is used to transmit the video data;
  • the second channel establishment unit is used for:
  • the second secure data channel is established between the second network port and the first network port based on the second username, the second password and the first certificate verification fingerprint.
  • the video data sending module 1103 is used for:
  • FIG. 12 shows a structural block diagram of a terminal device provided by an exemplary embodiment of the present application. It can be implemented as the first terminal device or the second terminal device in the above embodiments, and the terminal device in this application can include one or more of the following components: a processor 1210 and a memory 1220 .
  • Processor 1210 may include one or more processing cores.
  • the processor 1210 uses various interfaces and lines to connect various parts in the entire electronic device, and executes the electronic device by running or executing the instructions, programs, code sets or instruction sets stored in the memory 1220, and calling the data stored in the memory 1220.
  • Various functions of the device and processing data may adopt at least one of a digital signal processing (Digital Signal Processing, DSP), a Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and a Programmable Logic Array (Programmable Logic Array, PLA).
  • DSP Digital Signal Processing
  • FPGA Field-Programmable Gate Array
  • PLA Programmable Logic Array
  • the processor 1210 may integrate one or more of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), a neural network processor (Neural-network Processing Unit, NPU), and a modem, etc.
  • a central processing unit Central Processing Unit, CPU
  • a graphics processor Graphics Processing Unit, GPU
  • a neural network processor Neural-network Processing Unit, NPU
  • modem etc.
  • the CPU mainly handles the operating system, user interface, and applications
  • the GPU is used to render and draw the content that needs to be displayed on the touch screen
  • the NPU is used to implement artificial intelligence (AI) functions
  • the modem is used to process Wireless communication. It can be understood that, the above-mentioned modem may not be integrated into the processor 1210, but is implemented by a single chip.
  • the memory 1220 may include random access memory (Random Access Memory, RAM), or may include read-only memory (Read-Only Memory, ROM).
  • the memory 1220 includes a non-transitory computer-readable storage medium.
  • Memory 1220 may be used to store instructions, programs, codes, sets of codes, or sets of instructions.
  • the memory 1220 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), Instructions and the like used to implement the following various method embodiments; the storage data area may store data (such as audio data, phone book) and the like created according to the use of the terminal device.
  • the terminal device in this embodiment of the present application further includes a communication component 1230 .
  • the communication component 1230 may be a Bluetooth component, a WiFi component, an NFC component, etc., for communicating with an external device (a server or other terminal device) through a wired or wireless network.
  • the structure of the terminal device shown in the above drawings does not constitute a limitation on the terminal device, and the terminal device may include more or less components than those shown in the drawings, or a combination of certain components may be included. some components, or a different arrangement of components.
  • the terminal device also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a speaker, a microphone, and a power supply, which will not be repeated here.
  • Embodiments of the present application further provide a computer-readable storage medium, where at least one instruction is stored in the storage medium, and at least one instruction is used to be executed by a processor to implement the remote control method described in the foregoing embodiments.
  • Embodiments of the present application provide a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the remote control methods provided in the above embodiments.
  • Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.
  • a storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.

Abstract

本申请实施例公开了一种远程控制方法、装置、设备、存储介质及系统，属于终端技术领域。包括：通过设备在线服务器向处于在线状态的第二终端设备发送远程控制请求；响应于第二终端设备接受远程控制请求，通过信令服务器与第二终端设备建立安全数据通道，安全数据通道基于WebRTC建立；通过安全数据通道接收第二终端设备发送的视频数据；对视频数据进行解码，还原显示第二终端设备处的屏幕画面；响应于还原后屏幕画面内的控制操作，通过安全数据通道向第二终端设备发送控制数据，第二终端设备用于根据控制数据响应屏幕画面内的控制操作。本申请实施例基于WebRTC实现不同平台终端设备间的屏幕分享及远程控制，且提高了远程控制的实时性。

Description

远程控制方法、装置、设备、存储介质及系统
本申请要求于2020年09月23日提交的申请号为202011009850.X、发明名称为“远程控制方法、装置、设备、存储介质及系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及终端技术领域，特别涉及一种远程控制方法、装置、设备、存储介质及系统。
背景技术
随着终端技术的不断发展，终端设备的功能越来越丰富，但是功能不断丰富的同时也增加了用户使用终端设备的学习成本。
比如，由于老人学习使用终端设备的能力较差，因此当老人在使用终端设备的过程中遇到问题时，需要通过电话等方式求助子女，或者由子女亲自操作终端设备以解决问题。
发明内容
本申请实施例提供了一种远程控制方法、装置、设备、存储介质及系统。所述技术方案如下：
一方面，本申请实施例提供了一种远程控制方法，应用于第一终端设备，所述方法包括：
通过设备在线服务器向处于在线状态的第二终端设备发送远程控制请求；
响应于所述第二终端设备接受所述远程控制请求，通过信令服务器与所述第二终端设备建立安全数据通道，所述安全数据通道基于网页实时通信(Web Real-Time Communication，WebRTC)建立；
通过所述安全数据通道接收所述第二终端设备发送的视频数据，所述视频数据由所述第二终端设备对采集到的屏幕画面进行编码得到；
对所述视频数据进行解码，还原显示所述第二终端设备处的所述屏幕画面；
响应于还原后所述屏幕画面内的控制操作，通过所述安全数据通道向所述第二终端设备发送控制数据，所述第二终端设备用于根据所述控制数据响应所述屏幕画面内的所述控制操作。
另一方面，本申请实施例提供了一种远程控制方法，应用于第二终端设备，所述方法包括：
通过设备在线服务器接收第一终端设备发送的远程控制请求；
响应于对所述远程控制请求的请求接受操作，通过信令服务器与所述第一终端设备建立安全数据通道，所述安全数据通道基于WebRTC建立；
通过所述安全数据通道向所述第一终端设备发送视频数据，所述视频数据通过对采集到的屏幕画面进行编码得到，所述第一终端设备用于对所述视频数据进行解码，并还原显示所述第二终端设备处的所述屏幕画面；
通过所述安全数据通道接收所述第一终端设备发送的控制数据，所述控制数据由所述第一终端设备接收到还原后所述屏幕画面内的控制操作时生成；
根据所述控制数据响应所述屏幕画面内的所述控制操作。
另一方面，本申请实施例提供了一种远程控制装置，应用于第一终端设备，所述装置包括：
请求发送模块，用于通过设备在线服务器向处于在线状态的第二终端设备发送远程控制 请求；
第一通道建立模块，用于响应于所述第二终端设备接受所述远程控制请求，通过信令服务器与所述第二终端设备建立安全数据通道，所述安全数据通道基于WebRTC建立；
视频数据接收模块，用于通过所述安全数据通道接收所述第二终端设备发送的视频数据，所述视频数据由所述第二终端设备对采集到的屏幕画面进行编码得到；
显示模块，用于对所述视频数据进行解码，还原显示所述第二终端设备处的所述屏幕画面；
控制数据发送模块，用于响应于还原后所述屏幕画面内的控制操作，通过所述安全数据通道向所述第二终端设备发送控制数据，所述第二终端设备用于根据所述控制数据响应所述屏幕画面内的所述控制操作。
另一方面，本申请实施例提供了一种远程控制装置，应用于第二终端设备，所述装置包括：
请求接收模块，用于通过设备在线服务器接收第一终端设备发送的远程控制请求；
第二通道建立模块，用于响应于对所述远程控制请求的请求接受操作，通过信令服务器与所述第一终端设备建立安全数据通道，所述安全数据通道基于WebRTC建立；
视频数据发送模块，用于通过所述安全数据通道向所述第一终端设备发送视频数据，所述视频数据通过对采集到的屏幕画面进行编码得到，所述第一终端设备用于对所述视频数据进行解码，并还原显示所述第二终端设备处的所述屏幕画面；
控制数据接收模块，用于通过所述安全数据通道接收所述第一终端设备发送的控制数据，所述控制数据由所述第一终端设备接收到还原后所述屏幕画面内的控制操作时生成；
控制响应模块，用于根据所述控制数据响应所述屏幕画面内的所述控制操作。
另一方面，本申请实施例提供了一种终端设备，所述终端设备包括处理器和存储器；所述存储器存储有至少一条指令，所述至少一条指令用于被所述处理器执行以实现上述方面所述的第一终端设备侧的远程控制方法，或，实现上述方面所述的第二终端设备侧的远程控制方法。
另一方面，本申请实施例提供了一种计算机可读存储介质，所述存储介质存储有至少一条指令，所述至少一条指令用于被处理器执行以实现如上述方面所述的第一终端设备侧的远程控制方法，或，实现上述方面所述的第二终端设备侧的远程控制方法。
另一方面，本申请实施例提供了一种远程控制系统，所述远程控制系统包括：第一终端设备、第二终端设备、设备在线服务器和信令服务器；
所述第一终端设备通过有线或无线网络分别与所述设备在线服务器和所述信令服务器相连；
所述第二终端设备通过有线或无线网络分别与所述设备在线服务器和所述信令服务器相连；
所述第一终端设备用于实现如上述方面所述的第一终端设备侧的远程控制方法；
所述第二终端设备用于实现如上述方面所述的第二终端设备侧的远程控制方法。
另一方面，本申请实施例提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述方面提供的远程控制方法。
附图说明
图1示出了本申请一个示例性实施例提供的实施环境的示意图；
图2示出了本申请一个示例性实施例提供的远程控制方法的流程图；
图3是本申请一个示例性实施例提供的远程控制方法实施过程的实施示意图；
图4示出了本申请另一个示例性实施例提供的远程控制方法的流程图；
图5是本申请一个示例性实施例示出的触控位置映射过程的实施示意图；
图6示出了本申请另一个示例性实施例提供的远程控制方法的流程图；
图7是本申请一个示例性实施例示出的信令服务获取过程的实施示意图；
图8是本申请一个示例性实施例示出的安全数据通道建立过程的实施示意图；
图9示出了本申请另一个示例性实施例提供的远程控制方法的流程图；
图10示出了本申请一个实施例提供的远程控制装置的结构框图；
图11示出了本申请另一个实施例提供的远程控制装置的结构框图；
图12示出了本申请一个示例性实施例提供的终端设备的结构方框图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
在本文中提及的“多个”是指两个或两个以上。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。字符“/”一般表示前后关联对象是一种“或”的关系。
请参考图1，其示出了本申请一个示例性实施例提供的实施环境的示意图，该实施环境中包括第一终端设备110、第二终端设备120、设备在线服务器130以及信令服务器140。
第一终端设备110和第二终端设备120是具有远程控制功能的电子设备，该电子设备可以包括智能手机、平板电脑、智能电视、个人计算机或可穿戴式设备等等，本实施例对终端设备的类型并不进行限定。并且，该终端设备的远程控制功能可以通过安装的远程控制应用程序实现，或者，通过网页实现。
远程控制场景下，第一终端设备110和第二终端设备120之一为控制终端，而另一终端设备则为被控终端。可选的，控制终端为发起远程控制的终端设备，而被控终端为接受远程控制的终端设备。可选的，在远程控制过程中，控制终端和被控终端可以进行切换。
在一些实施例中，进行远程控制的双方可以为同一平台(或类型)的终端设备，也可以是不同平台的终端设备，即可以实现跨平台的远程控制。比如，用户可以使用智能手机对其他智能手机进行远程控制；或者，用户可以使用智能手机对智能电视或者可穿戴式设备进行远程控制。本申请实施例并不对远程控制双方的平台进行限定。
设备在线服务器130是用于提供设备发现以及消息通知服务的服务器，其可以是一台服务器、若干台服务器构成的服务器集群或云计算中心。当第一终端设备110和第二终端设备120需要进行远程控制时，均需要连接设备在线服务器130，其中，终端设备可以通过有线或无线网络与设备在线服务器130建立连接。
信令服务器140是用于为远程控制双方提供信息交换服务，从而协助远程控制双方建立安全数据通道的服务器，其可以是一台服务器、若干台服务器构成的服务器集群或云计算中心。
进行远程控制前，第一终端设备110和第二终端设备120均需要连接信令服务器140，并通过信令服务器140向对端发送消息，以此在第一终端设备110和第二终端设备120之间建立安全数据通道。其中，终端设备可以通过有线或无线网络与信令服务器140建立连接。
可选的，安全数据通道基于WebRTC建立，用于传输被控终端发送的视频数据，以及控制终端发送的控制数据，且能够实现低延时数据传输，提高远程控制的实时性。
在一种可能的应用场景下，第一终端设备110和第二终端设备120位于不同场所。如图1所示，第一终端设备110位于办公场所，而第二终端设备120则位于家庭住所，当需要使用第一终端设备110对第二终端设备120进行远程控制时，第一终端设备110开启远程控制应用程序，以连接至设备在线服务器130。
连接至设备在线服务器130后，第一终端设备110通过设备在线服务器130向第二终端设备120发送远程控制请求。当第二终端设备120当前连接至设备在线服务器130，并接受远程控制时，第一终端设备110和第二终端设备120则进一步通过信令服务器140建立安全数据通道150。建立起安全数据通道150后，第二终端设备120即对屏幕画面(以及第二终端设备120侧的音频)进行编码，并通过安全数据通道150将编码得到的视频数据发送至第一终端设备110，由第一终端设备110进行解码显示，使用户可以通过第一终端设备110查看到第二终端设备120的实时显示内容。进一步的，用户可以在第一终端设备110显示的内容中进行控制操作，由第一终端设备110将控制操作对应的控制数据(通过安全数据通道150)传输至第二终端设备120，以便第二终端设备120根据控制数据进行控制操作响应，改变当前显示内容，并通过安全数据通道150更新至第一终端设备110。
除了应用于上述场景外，本申请实施例提供的远程控制方法还可以应用于远程维护场景，即售后工程师可以通过终端设备对售出的终端设备进行配置或调试，本申请实施例对具体的应用场景并不进行限定。
请参考图2，其示出了本申请一个示例性实施例提供的远程控制方法的流程图，本实施例以该方法应用图1所示的实施环境为例进行说明，该方法可以包括如下步骤。
步骤201，第一终端设备通过设备在线服务器向处于在线状态的第二终端设备发送远程控制请求。
本申请实施例中，第一终端设备为控制终端，第二终端设备为被控终端。在一种可能的实施方式中，当需要通过第一终端设备对第二终端设备进行控制时，第一终端设备首先连接设备在线服务器，然后向设备在线服务器发送远程控制请求，该远程控制请求中包含第二终端设备的设备标识。可选的，不同终端对应不同设备标识，且该设备标识由设备在线服务器分配。
由于第二终端设备可能并未连接至设备在线服务器，即处于离线状态，而设备在线服务器无法向处于离线状态的终端设备发送远程控制请求，因此在一种可能的实施方式中，设备在线服务器接收到远程控制请求后，根据其中包含的第二终端设备的设备标识，查询该第二终端设备是否处于在线状态。若处于在线状态，则向第二终端设备发送远程控制请求；若处于离线状态，则向第一终端设备发送设备离线消息，以便第一终端设备使用者提示第二终端设备使用者将第二终端设备连接至设备在线服务器。
示意性的，如图3所示，第一终端设备31需要对第二终端设备32进行远程控制时，通过开启远程控制应用程序连接设备在线服务器。当接收到对设备选择界面311中第二终端设备32对应设备标识的选择操作时，第一终端设备31即通过设备在线服务器向第二终端设备32发送远程控制请求。
步骤202，第二终端设备通过设备在线服务器接收第一终端设备发送的远程控制请求。
在一种可能的实施方式中，当接收到设备在线服务器发送的远程控制请求时，第二终端设备显示远程控制提示窗，并通过远程控制提示窗接收远程控制响应(接受或拒绝响应)。
示意性的，如图3所示，处于在线状态的第二终端设备32接收到远程控制请求后，显示远程控制提示窗321。
步骤203，响应于对远程控制请求的请求接受操作，第二终端设备通过信令服务器与第一终端设备建立安全数据通道，安全数据通道基于WebRTC建立。
可选的，当接收到对远程控制请求的请求接受操作时，第二终端设备通过网页套接字安全(Web Socket Secure，WSS)协议连接至信令服务器，从而通过信令服务器与第一终端设备进行信息交互，进而与第一终端设备建立安全数据通道。
可选的，为了实现低延迟数据传输，本申请实施例中，第一终端设备和第二终端设备基于WebRTC建立安全数据通道。
示意性的，如图3所示，当接收到对远程控制提示窗321中接受控件的点击操作时，第二终端设备32即连接信令服务器，与第一终端设备31建立安全数据通道。
步骤204，响应于第二终端设备接受远程控制请求，第一终端设备通过信令服务器与第二终端设备建立安全数据通道，安全数据通道基于WebRTC建立。
在一种可能的实施方式中，第二终端设备接受远程控制请求后，向设备在线服务器发送请求接受响应，设备在线服务器接收到该请求接受响应后即通知第一终端设备，以便第一终端设备连接至信令服务器，进而通过信令服务器提供的信令服务与第二终端设备建立安全数据通道。
步骤205，第二终端设备通过安全数据通道向第一终端设备发送视频数据，视频数据通过对采集到的屏幕画面进行编码得到。
建立安全数据通道后，为了能够在第一终端设备处显示第二终端设备侧的屏幕画面，以便后续用户通过第一终端设备进行远程控制，第二终端设备采集屏幕画面，并对屏幕画面进行编码得到视频数据，从而将视频数据以视频流的形式发送至第一终端设备(通过安全数据通道)。
可选的，第二终端设备在采集屏幕画面的同时进行音频采集(包括系统音频或麦克风采集的外部音频，并进行混音处理)，并将采集到的音频进行编码得到音频数据，从而将音频数据和视频数据同步发送至第一终端设备。
示意性的，如图3所示，第二终端设备32对当前采集到的屏幕画面322进行编码得到视频数据，并通过安全数据通道将视频数据传输至第一终端设备31。
步骤206，第一终端设备通过安全数据通道接收第二终端设备发送的视频数据。
可选的，第一终端设备通过安全数据通道接收第二终端设备发送的音频数据。
步骤207，第一终端设备对视频数据进行解码，还原显示第二终端设备处的屏幕画面。
第一终端设备对接收到的视频数据进行解码，得到连续的视频帧画面，从而对该视频帧画面进行显示，达到在第一终端设备侧还原显示第二终端设备屏幕画面的效果。
可选的，第一终端设备全屏显示解码还原后的屏幕画面，或者，第一终端设备在窗口中显示显示解码还原后的屏幕画面，且该窗口的尺寸可调。
可选的，当接收到音频数据时，第一终端设备对音频数据进行解码播放，还原第二终端设备采集到的音频。
示意性的，如图3所示，第一终端设备31接收到视频数据后，对视频数据进行解码，并显示还原后的屏幕画面312。
步骤208，响应于还原后屏幕画面内的控制操作，第一终端设备通过安全数据通道向第二终端设备发送控制数据。
借助安全数据通道进行视频数据传输，第一终端设备与第二终端设备显示相同画面，由于第一终端设备处显示的是解码得到的视频画面，而实际对屏幕显示内容的操作权限仍旧在第二终端设备处，因此当用户对第一终端设备显示的屏幕画面进行控制操作时，第一终端设备需要将控制操作对应的控制数据发送至第二终端设备，由第二终端设备根据控制数据对控制操作进行响应。
可选的，该控制操作包括触控操作、外设输入操作(比如通过控制鼠标移动点击、通过键盘进行文字输入)、语音控制操作或手势操作中的至少一种。
示意性的，如图3所述，当接收到用户对还原后屏幕画面312中的字体大小调节控件进行触控操作时(将字体大小由标准调整为特大)，第一终端设备31生成触控操作对应的控制数据，并将该控制数据发送至第二终端设备32。
步骤209，第二终端设备通过安全数据通道接收第一终端设备发送的控制数据。
步骤210，第二终端设备根据控制数据响应屏幕画面内的控制操作。
进一步的，第二终端设备对接收到的控制数据进行解析，并根据解析结果在第二终端设 备侧还原第一终端设备侧接收到的控制操作，进而对该控制操作进行响应(响应后屏幕画面可能会发生变化)。需要说明的是，第二终端设备响应该控制操作过程中，仍旧通过安全数据通道向第一终端设备发送视频数据，保证第一终端设备与第二终端设备处显示画面的一致性。
示意性的，如图3所示，第二终端设备32根据接收到的控制数据，对屏幕画面322中的字体大小调节控件进行调节，还原出第一终端设备31侧用户的触控操作。
采用图3所示的实施例，当家中老人的手机字体过小时，子女可以使用手机对老人使用的手机进行远程控制，提高老人使用手机的字体大小，无需实地操作老人使用的手机。
综上所述，本申请实施例中，第一终端设备通过设备在线服务器向第二终端设备发起远程控制请求，并借助信令服务器与第二终端设备建立起基于WebRTC的安全数据通道，从而利用该安全数据通道接收第二终端设备发送的视频数据，对第二终端设备处的屏幕画面进行还原，并根据接收到的控制操作，通过安全数据通道向第二终端设备发送控制数据，实现对第二终端设备的远程控制；本申请实施例基于WebRTC实现终端设备之间的屏幕分享及远程控制，适用于不同平台终端设备，且基于WebRTC实现数据的低时延传输，有助于提高远程控制的实时性。
可选的，控制操作包括触控操作；
响应于还原后屏幕画面内的控制操作，第一终端设备通过安全数据通道向第二终端设备发送控制数据，包括：
响应于还原后屏幕画面内的控制操作，第一终端设备根据控制操作的操作信息生成控制数据，操作信息至少包括操作类型和操作位置；
第一终端设备通过安全数据通道向第二终端设备发送控制数据，第二终端设备用于根据控制数据响应屏幕画面内的控制操作。
可选的，第一终端设备根据控制操作的操作信息生成控制数据，包括：
响应于还原后屏幕画面的第一画面尺寸与屏幕画面在第二终端设备处的第二画面尺寸不同，根据控制操作的操作信息以及第一画面尺寸生成控制数据，第二终端设备用于根据第一画面尺寸以及第二画面尺寸，对操作位置进行位置映射。
可选的，响应于第二终端设备接受远程控制请求，第一终端设备通过信令服务器与第二终端设备建立安全数据通道，包括：
响应于第二终端设备接受远程控制请求，获取设备在线服务器生成的房间标识，第一终端设备和第二终端设备具有相同房间标识；
第一终端设备基于房间标识获取信令服务器提供的信令服务，信令服务指信令服务器为具有相同房间标识的终端设备提供消息转发服务；
第一终端设备通过信令服务与第二终端设备建立安全数据通道。
可选的，第一终端设备通过信令服务与第二终端设备建立安全数据通道，包括：
第一终端设备通过信令服务向第二终端设备发送第一消息，第一消息中包含第一网络端口信息、第一用户名、第一密码以及第一证书的第一证书验证指纹，第一用户名和第一密码由所述第一终端设备随机生成；
第一终端设备通过信令服务接收第二终端设备发送的第二消息，第二消息中包含第二网络端口信息、第二用户名、第二密码以及第二证书的第二证书验证指纹，第二用户名和第二密码由第二终端设备随机生成；
第一终端设备根据第一消息和第二消息建立安全数据通道。
可选的，安全数据通道包括第一安全数据通道和第二安全数据通道，第一安全数据通道用于传输控制数据，第二安全数据通道用于传输视频数据；
第一终端设备根据第一消息和第二消息建立安全数据通道，包括：
基于第一用户名、第一密码和第二证书验证指纹，在第一网络端口和第二网络端口之间建立第一安全数据通道。
可选的，第一终端设备对视频数据进行解码，还原显示第二终端设备处的屏幕画面，包括：
第一终端设备通过预设加密密钥对视频数据进行解密，预设加密密钥是在建立第二安全数据通道时由第二终端设备设置；
第一终端设备对解密后的视频数据进行解码，还原显示第二终端设备处的屏幕画面。
可选的，第一终端设备通过设备在线服务器向处于在线状态的第二终端设备发送远程控制请求，包括：
第一终端设备显示至少一个候选终端设备的设备标识，候选终端设备是与第一终端设备进行过远程控制，且处于在线状态的历史终端设备；响应于对至少一个候选终端设备中第二终端设备的选择操作，通过设备在线服务器向第二终端设备发送远程控制请求；
或，
第一终端设备获取输入的目标设备标识，目标设备标识为第二终端设备的设备标识；向设备在线服务器发送包含目标设备标识的远程控制请求，设备在线服务器用于根据目标设备标识查询第二终端设备的设备状态，并在第二终端设备处于在线状态时，向第二终端设备发送远程控制请求。
可选的，第二终端设备根据控制数据响应屏幕画面内的控制操作，包括：
第二终端设备根据控制数据中的操作类型和操作位置生成操作事件；
第二终端设备向操作系统上报操作事件，操作系统用于对操作事件进行响应。
可选的，控制数据中还包括第一画面尺寸，第一画面尺寸为还原后屏幕画面的尺寸；
第二终端设备根据控制数据中的操作类型和操作位置生成触控事件，包括：
第二终端设备根据第一画面尺寸以及屏幕画面在第二终端设备处的第二画面尺寸，对操作位置进行位置映射；
第二终端设备根据操作类型和映射后的操作位置生成操作事件。
可选的，响应于对远程控制请求的请求接受操作，第二终端设备通过信令服务器与第二终端设备建立安全数据通道，包括：
响应于对远程控制请求的请求接受操作，第二终端设备获取设备在线服务器生成的房间标识，第一终端设备和第二终端设备具有相同房间标识；
第二终端设备基于房间标识获取信令服务器提供的信令服务，信令服务指信令服务器为具有相同房间标识的终端设备提供消息转发服务；
第二终端设备通过信令服务与第一终端设备建立安全数据通道。
可选的，通过信令服务与第一终端设备建立安全数据通道，包括：
第二终端设备通过信令服务接收第一终端设备发送的第一消息，第一消息中包含第一网络端口信息、第一用户名、第一密码以及第一证书的第一证书验证指纹，第一用户名和第一密码由第一终端设备随机生成；
第二终端设备通过信令服务向第一终端设备发送第二消息，第二消息中包含第二网络端口信息、第二用户名、第二密码以及第二证书的第二证书验证指纹，第二用户名和第二密码由第二终端设备随机生成；
第二终端设备根据第一消息和第二消息建立安全数据通道。
可选的，安全数据通道包括第一安全数据通道和第二安全数据通道，第一安全数据通道用于传输控制数据，第二安全数据通道用于传输视频数据；
第二终端设备根据第一消息和第二消息建立安全数据通道，包括：
第二终端设备基于第二用户名、第二密码和第一证书验证指纹，在第二网络端口和第一网络端口之间建立第二安全数据通道。
可选的，第二终端设备通过安全数据通道向第一终端设备发送视频数据，包括：
第二终端设备通过预设加密密钥对视频数据进行加密；
第二终端设备通过第二安全数据通道向第一终端设备发送加密后的视频数据，第一终端设备用于通过预设加密密钥对接收到的视频数据进行解密。
如图4所示，其示出了本申请另一个示例性实施例提供的远程控制方法的流程图。
步骤401，响应于还原后屏幕画面内的控制操作，第一终端设备根据控制操作的操作信息生成控制数据，操作信息至少包括操作类型和操作位置。
在一种可能的实施方式中，当第一终端设备具有触控功能时，若接收到位于还原后屏幕画面内的触控操作，第一终端设备获取触控操作对应的触控信息，从而基于该触控信息生成控制数据。
其中，该触控信息包括触控类型和触控位置，且触控类型包括单击、双击、长按、滑动、拖动或按压中的至少一种，相应的，触控位置包括触控起点位置、触控终点位置(针对滑动、拖动)或至少一个触控中间位置(针对滑动、拖动)。
可选的，该触控位置采用触控操作在还原后屏幕画面内的坐标表示。
示意性的，如图5所示，第一终端设备51接收到第二终端设备52发送的视频数据(对采集到的屏幕画面521进行编码得到)后，在窗口511内显示还原后的屏幕画面。当接收到点击操作(通过控制箭头522进行点击)时，第一终端设备51获取点击操作的点击位置(x，y)，并确定触控类型的拖动。
当第一终端设备和第二终端设备属于不同平台(比如使用智能手机控制智能电视或者可穿戴式设备)，或者，属于同一平台但型号不同，或者，用户调整第一终端设备处屏幕画面的显示尺寸时，第一终端设备处屏幕画面的画面尺寸与第二终端设备处屏幕画面的画面尺寸可能并不相同。此时，若第二终端设备直接根据控制数据中包含的触控位置进行触控响应，将会出现响应错误。
为了提高控制操作的准确性，在一种可能的实施方式中，当还原后屏幕画面的第一画面尺寸(即第一终端设备处的显示尺寸)与屏幕画面在第二终端设备处的第二画面尺寸不同时，第一终端设备根据触控操作的触控信息以及第一画面尺寸生成控制数据，以便第二终端设备将第一终端设备处的触控位置映射至第二终端设备的屏幕画面中。可选的，第二画面尺寸由第一终端设备从视频数据中解析得到。
在其他可能的实施方式中，第一终端设备也可以在检测到第一画面尺寸发生变化时(可能因用户调整窗口大小导致尺寸变化)，通过安全数据通道向第二终端设备发送通知消息，本实施例对此不作限定。
示意性的，如图5所示，第一终端设备51处窗口511的尺寸为a×b(第一画面尺寸同样为a×b)，向第二终端设备52发送的控制数据中包含第一画面尺寸a×b，触控类型“1”(指示点击操作)以及触控位置(x，y)。
当然，在其他可能的实施方式中，第一终端设备可以根据外部输入输出设备(比如鼠标键盘)进行控制，对屏幕画面进行控制，本实施例在此不作赘述。
步骤402，第一终端设备通过安全数据通道向第二终端设备发送控制数据。
步骤403，第二终端设备通过安全数据通道接收第一终端设备发送的控制数据。
步骤404，第二终端设备根据控制数据中的操作类型和操作位置生成触控事件。
第二终端设备接收到控制数据后，解析得到控制数据中包含的操作类型以及操作位置，从而生成操作事件。可选的，当第二终端设备具有触控屏时，该操作事件即为触控事件，该操作事件为模拟向操作系统注入(上报)的事件。
在一种可能的实施方式中，当解析到控制数据中除了包含操作类型和操作位置外，还包括第一画面尺寸时，在生成操作事件中的操作位置时，第二终端设备根据第一画面尺寸以及屏幕画面在第二终端设备处的第二画面尺寸，对操作位置进行位置映射，从而根据操作类型 和映射后的操作位置生成操作事件，保证控制操作的准确性。
示意性的，如图5所示，当第一画面尺寸为(a，b)，第二画面尺寸为(A，B)，且触控位置为(x，y)时，映射得到的触控位置为(x×A/a，y×B/b)。
步骤405，第二终端设备向操作系统上报操作事件，操作系统用于对操作事件进行响应。
进一步的，第二终端设备将操作事件上报至操作系统，模拟出第二终端设备接收到控制操作，由操作系统对该操作事件进行响应。示意性的，如图5所示，当用户在第一终端设备51处执行单击操作时，第二终端设备52根据第一终端设备51发送的控制数据，控制箭头522点击屏幕画面521中第二行第三列的图标。
本实施例中，第一终端设备通过向第二终端设备发送本端屏幕画面的第一画面尺寸，使第二终端设备基于第一画面尺寸和第二画面尺寸对触控位置进行位置映射，避免因终端设备画面尺寸不一致导致的控制异常问题，提高了远程控制的控制准确性。
为了提高远程控制的安全性，在一种可能的实施方式中，第一终端设备和第二终端设备需要从设备在线服务器处获取相同的房间标识，并基于该房间标识使用信令服务器提供的信令服务，从而建立安全数据通道。下面采用示例性的实施例进行说明。
如图6所示，其示出了本申请另一个示例性实施例提供的远程控制方法的流程图，该方法可以包括如下步骤。
步骤601，第一终端设备通过设备在线服务器向处于在线状态的第二终端设备发送远程控制请求。
关于发送远程控制请求的方式，在一种可能的实施方式中，第一终端设备显示至少一个候选终端设备的设备标识，该候选终端设备是与第一终端设备进行过远程控制，且处于在线状态的历史终端设备；响应于对至少一个候选终端设备中第二终端设备的选择操作，通过设备在线服务器向第二终端设备发送远程控制请求。
可选的，第一终端设备连接至设备在线服务器后，即向设备在线服务器发送设备在线状态查询请求，请求设备在线服务器查询历史终端设备(进行过远程控制)的设备状态(包括在线状态和离线状态)。设备在线服务器向第一终端设备反馈查询到的设备状态后，第一终端设备对处于在线状态的历史终端设备进行显示，以便用户从中进行选择。
在另一种可能的实施方式中，当第一终端设备首次与第二终端设备进行远程控制时，第一终端设备获取输入的目标设备标识，该目标设备标识为第二终端设备的设备标识；向设备在线服务器发送包含目标设备标识的远程控制请求。相应的，设备在线服务器根据目标设备标识查询第二终端设备的设备状态，并在第二终端设备处于在线状态时，向第二终端设备发送远程控制请求。
当然，除了上述两种触发发送远程控制请求的方式外，第一终端设备还可以采用其他方式(比如点击远程控制链接)触发发送远程控制请求，本申请实施例并不对此构成限定。
步骤602，第二终端设备通过设备在线服务器接收第一终端设备发送的远程控制请求。
步骤603，响应于对远程控制请求的请求接受操作，第二终端设备获取设备在线服务器生成的房间标识，第一终端设备和第二终端设备具有相同房间标识。
在一种可能的实施方式中，当允许第一终端设备进行远程控制时，第二终端设备向设备在线服务器发送请求接受响应，设备在线服务器接收到请求接受响应后，即为第一终端设备和第二终端设备分配相同且唯一的房间标识(roomid)。可选的，设备在线服务器从资源池中获取房间标识并分配给第一终端设备和第二终端设备，保证同一时刻下进行不同远程控制的不同终端设备具有不同的房间标识。
步骤604，响应于第二终端设备接受远程控制请求，第一终端设备获取设备在线服务器生成的房间标识，第一终端设备和第二终端设备具有相同房间标识。
相应的，第一终端设备接收到设备在线服务器分配的房间标识后，确定第二终端设备已 接受远程控制。
步骤605，第一终端设备基于房间标识获取信令服务器提供的信令服务。
步骤606，第二终端设备基于房间标识获取信令服务器提供的信令服务。
在一种可能的实施方式中，为了提高远程控制建立过程的安全性，信令服务器仅为具有相同房间标识的终端设备提供信令服务，该信令服务指信令服务器为具有相同房间标识的终端设备提供消息转发服务，即仅具有相同房间标识的终端设备之间才能够通过信令服务建立安全数据通道。
可选的，第一终端设备获取到房间标识后，通过该房间标识连接信令服务器，第二终端设备同样通过该房间标识连接信令服务器。信令服务器对第一终端设备和第二终端设备的房间标识进行验证，并在两者一致时，为第一终端设备和第二终端设备提供信令服务。
示意性的，如图7所示，当需要通过第一终端设备71控制第二终端设备72时，第一终端设备71和第二终端设备72均登录设备在线服务器73。第一终端设备71通过设备在线服务器73向第二终端设备72发送远程控制请求，当接收到第二终端设备72发送的请求接受响应时，设备在线服务器73为第一终端设备71和第二终端设备72分配相同的房间标识。第一终端设备71和第二终端设备72获取到该房间标识后，进一步基于房间标识，通过信令服务器74提供的信令服务在设备间建立安全数据通道。
步骤607，第一终端设备通过信令服务向第二终端设备发送第一消息，第一消息中包含第一网络端口信息、第一用户名、第一密码以及第一证书的第一证书验证指纹，第一用户名和第一密码由第一终端设备随机生成。
终端设备间建立安全数据通道时，需要通过信令服务进行信息交换，为了提高建立安全数据通道的安全性，在一种可能的实施方式中，第一终端设备开放第一网络端口(IP端口)，并通过信令服务发送包含第一网络端口信息、第一用户名、第一密码以及第一证书对应第一证书验证指纹(fingerprint)的第一消息。
其中，第一用户名和第一密码由第一终端设备随机生成，即每次进行远程控制时，第一终端设备发送的第一消息中第一用户名和第一密码均不相同。即便第一终端设备本次进行远程控制时使用的用户名和密码被泄露，也无法使用该用户名和密码进行下一次远程控制，提高远程控制的安全性。第一证书用于进行安全传输层协议(Transport Layer Security，TLS)连接。
示意性的，如图8所示，第一终端设备71通过信令服务向第二终端设备72发送的第一消息中包含IP端口a，用户名user1，密码pwd1，证书cer1的指纹fingerprint1。
步骤608，第二终端设备通过信令服务接收第一终端设备发送的第一消息。
在一种可能的实施方式中，信令服务器将第一终端设备发送的第一消息转发至具有相同房间标识的第二终端设备，相应的，第二终端设备接收信令服务器转发的第一消息。
步骤609，第二终端设备通过信令服务向第一终端设备发送第二消息，第二消息中包含第二网络端口信息、第二用户名、第二密码以及第二证书的第二证书验证指纹，第二用户名和第二密码由第二终端设备随机生成。
与第一终端设备发送的第一消息类似的，第二终端设备第开放第二网络端口(IP端口)，并通过信令服务发送包含第二网络端口信息、第二用户名、第二密码以及第二证书对应第二证书验证指纹(fingerprint)的第二消息。
其中，第二用户名和第二密码由第二终端设备随机生成，即每次进行远程控制时，第二终端设备发送的第二消息中第二用户名和第二密码均不相同，从而避免使用固定用户名和密码被泄露时造成的安全隐患。第二证书同样用于进行安全传输层协议(Transport Layer Security，TLS)连接。
示意性的，如图8所示，第二终端设备72通过信令服务向第一终端设备71发送的第二消息中包含IP端口b，用户名user2，密码pwd2，证书cer2的指纹fingerprint2。
需要说明的是，本实施例仅以第一消息和第二消息中包含上述信息为例进行说明，在实际应用过程中，第一消息和第二消息中还可以包括其他建立安全数据通道时所需的信息，本实施例对此不作限定。
步骤610，第一终端设备通过信令服务接收第二终端设备发送的第二消息。
在一种可能的实施方式中，信令服务器将第二终端设备发送的第二消息转发至具有相同房间标识的第一终端设备，相应的，第一终端设备接收信令服务器转发的第二消息。
进一步的，第一终端设备根据第一消息和第二消息，通过信令服务器与第二终端设备建立安全数据通道；第二终端设备根据第一消息和第二消息，通过信令服务器与第一终端设备建立安全数据通道。
由于视频数据的数据传输量较大且对传输时延要求较高，而控制数据的数据传输量较小且对传输时延要求较高，且两种数据传输可能同时进行，因此本申请实施例中，第一终端设备和第二终端设备之间建立有用于传输视频数据的第一安全数据通道，以及用于传输控制数据的第二安全数据通道。
步骤611，第一终端设备基于第一用户名、第一密码和第二证书验证指纹，在第一网络端口和第二网络端口之间建立第一安全数据通道。
在一种可能的实施方式中，第一终端设备通过信令服务请求与第二终端设备建立第一安全数据通道时，向第二终端设备发送包含第一用户名、第一密码以及第二证书验证指纹的通道建立请求。相应的，第二终端设备检测第一用户名和第一密码与第一消息中包含的用户名和密码是否一致，且第二证书验证指纹与第二证书是否匹配。若用户名和密码一致且证书验证指纹匹配，则在第一终端设备的第一网络端口和第二终端设备的第二网路端口之间建立第一安全数据通道。可选的，该第一安全数据通道为TLS安全连接。
示意性的，如图8所示，第一终端设备71基于user1、pwd1和fingerprint2，在IP端口a和IP端口b之间建立第一安全数据通道。
后续远程控制过程中，第一终端设备即通过第一安全数据连接将控制数据发送至第二终端设备。
步骤612，第二终端设备基于第二用户名、第二密码和第一证书验证指纹，在第二网络端口和第一网络端口之间建立第二安全数据通道。
与第一安全数据通道建立过程类似的，在一种可能的实施方式中，第二终端设备通过信令服务请求与第一终端设备建立第二安全数据通道时，向第一终端设备发送包含第二用户名、第二密码以及第一证书验证指纹的通道建立请求。相应的，第一终端设备检测第二用户名和第二密码与第二消息中包含的用户名和密码是否一致，且第一证书验证指纹与第一证书是否匹配。若用户名和密码一致且证书验证指纹匹配，则在第二终端设备的第二网络端口和第一终端设备的第一网路端口之间建立第二安全数据通道。
示意性的，如图8所示，第二终端设备72基于user2、pwd2和fingerprint1，在IP端口b和IP端口a之间建立第二安全数据通道。
后续远程控制过程中，第二终端设备即通过第二安全数据连接将视频数据发送至第一终端设备。
在一些可能的应用场景下，为了方便远程控制过程中，第一终端设备和第二终端设备交换控制权，在一种可能的实施方式中，在建立安全数据通道时，第一终端设备与第二终端设备还建立有第三安全数据通道和第四安全数据通道，其中，第一终端设备用于通过第三安全数据通道向第二终端设备传输视频数据，第二终端设备用于通过第四安全数据通道向第一终端设备传输控制数据。
可选的，第三安全数据通道由第一终端设备基于第一用户名、第一密码和第二证书验证指纹建立，第四安全数据通道由第二终端设备基于第二用户名、第二密码和第一证书验证指纹建立。
可选的，当接收到第一终端设备和/或第二终端设备触发的控制权交换操作时，第一终端设备和第二终端设备即切换使用第三安全数据通道和第四安全数据通道进行数据传输。
在一种可能的实施方式中，由于视频数据对传输时延要求较高，且数据量较大，因此为了降低传输时延，第二终端设备与第一终端设备基于第二用户名、第二密码和第一证书验证指纹，与第一终端设备建立第二安全数据通道后，生成预设加密密钥，并通过第二安全数据通道将预设加密密钥传输至第一终端设备，后续即使用该预设加密密钥对视频数据进行加密，并基于安全实时传输协议(Secure Real-time Transport Protocol，SRTP)协议在第二安全数据通道上传输加密后的视频数据。
如图9所示，第一终端设备和第二终端设备间进行视频数据传输过程可以包括如下步骤。
步骤901，第二终端设备通过预设加密密钥对视频数据进行加密。
第二终端设备向第一终端设备传输视频数据前，使用安全数据通道建立阶段设置的预设加密密钥对视频数据进行加密，得到加密后的视频数据。
步骤902，第二终端设备通过第二安全数据通道向第一终端设备发送加密后的视频数据。
进一步的，第二终端设备通过与第一终端设备之间的第二安全数据通道，向第一终端设备发送加密后的视频数据。可选的，加密后的视频数据基于SRTP安全传输协议传输，在保证数据传输安全性的同时，降低数据传输时延。
步骤903，第一终端设备通过预设加密密钥对视频数据进行解密。
相应的，第一终端设备通过第二安全数据通道接收到视频数据后，首先使用预设加密密钥对该视频数据进行解密，得到解密后的视频数据。
步骤904，第一终端设备对解密后的视频数据进行解码，还原显示第二终端设备处的屏幕画面。
其中，第一终端设备对解密后视频数据进行解码以及还原显示屏幕画面的过程可以参考上述实施例，本实施例在此不再赘述。
本实施例中，设备在线服务器通过为远程控制双方分配相同的房间标识，使远程控制双方基于统一的房间标识获取信令服务器提供的信令服务，保证了远程控制过程的安全性；并且，远程控制双发基于随机生成的用户名和密码建立安全数据通道，避免使用固定用户名和密码时，因用户名和密码泄露造成的安全隐患，进一步提高了远程控制过程的安全性。
此外，本实施例中，第一终端设备和第二终端设备之间建立用于传输视频数据的第一安全数据通道，以及用于传输控制数据的第二安全数据通道，避免远程控制过程中视频数据和控制数据同时传输造成的相互影响，有助于降低视频数据和控制数据的传输时延，提高远程控制的实时性。
需要说明的是，上述各个实施例中，以第一终端设备为执行主体的步骤可以单独实现称为第一终端设备侧的远程控制方法，以第二终端设备为执行主体的步骤可以单独实现称为第二终端设备侧的远程控制方法，本申请实施例在此不再赘述。
请参考图10，其示出了本申请一个实施例提供的远程控制装置的结构框图。该装置可以通过软件、硬件或者两者的结合实现成为第一终端设备的全部或一部分。该装置包括：
请求发送模块1001，用于通过设备在线服务器向处于在线状态的第二终端设备发送远程控制请求；
第一通道建立模块1002，用于响应于所述第二终端设备接受所述远程控制请求，通过信令服务器与所述第二终端设备建立安全数据通道，所述安全数据通道基于WebRTC建立；
视频数据接收模块1003，用于通过所述安全数据通道接收所述第二终端设备发送的视频数据，所述视频数据由所述第二终端设备对采集到的屏幕画面进行编码得到；
显示模块1004，用于对所述视频数据进行解码，还原显示所述第二终端设备处的所述屏幕画面；
控制数据发送模块1005，用于响应于还原后所述屏幕画面内的控制操作，通过所述安全数据通道向所述第二终端设备发送控制数据，所述第二终端设备用于根据所述控制数据响应所述屏幕画面内的所述控制操作。
可选的，所述控制数据发送模块1005，包括：
控制数据生成单元，用于响应于还原后所述屏幕画面内的所述控制操作，根据所述控制操作的操作信息生成所述控制数据，所述操作信息至少包括操作类型和操作位置；
控制数据发送单元，用于通过所述安全数据通道向所述第二终端设备发送所述控制数据，所述第二终端设备用于根据所述控制数据响应所述屏幕画面内的所述控制操作。
可选的，所述控制数据生成单元，用于：
响应于还原后所述屏幕画面的第一画面尺寸与所述屏幕画面在所述第二终端设备处的第二画面尺寸不同，根据所述控制操作的所述操作信息以及所述第一画面尺寸生成所述控制数据，所述第二终端设备用于根据所述第一画面尺寸以及所述第二画面尺寸，对所述操作位置进行位置映射。
可选的呢，所述第一通道建立模块1002，包括：
房间标识获取单元，用于响应于所述第二终端设备接受所述远程控制请求，获取所述设备在线服务器生成的房间标识，所述第一终端设备和所述第二终端设备具有相同房间标识；
信令服务获取单元，用于基于所述房间标识获取所述信令服务器提供的信令服务，所述信令服务指所述信令服务器为具有相同房间标识的终端设备提供消息转发服务；
第一通道建立单元，用于通过所述信令服务与所述第二终端设备建立所述安全数据通道。
可选的，所述第一通道建立单元，用于：
通过所述信令服务向所述第二终端设备发送第一消息，所述第一消息中包含第一网络端口信息、第一用户名、第一密码以及第一证书的第一证书验证指纹，所述第一用户名和所述第一密码由所述第一终端设备随机生成；
通过所述信令服务接收所述第二终端设备发送的第二消息，所述第二消息中包含第二网络端口信息、第二用户名、第二密码以及第二证书的第二证书验证指纹，所述第二用户名和所述第二密码由所述第二终端设备随机生成；
根据所述第一消息和所述第二消息建立所述安全数据通道。
可选的，所述安全数据通道包括第一安全数据通道和第二安全数据通道，所述第一安全数据通道用于传输所述控制数据，所述第二安全数据通道用于传输所述视频数据；
所述第一通道建立单元，还用于：
基于所述第一用户名、所述第一密码和所述第二证书验证指纹，在第一网络端口和第二网络端口之间建立所述第一安全数据通道。
可选的，所述显示模块1004，用于：
通过预设加密密钥对所述视频数据进行解密，所述预设加密密钥是在建立所述第二安全数据通道时由所述第二终端设备设置；
对解密后的所述视频数据进行解码，还原显示所述第二终端设备处的所述屏幕画面。
可选的，所述请求发送模块1001，用于：
显示至少一个候选终端设备的设备标识，所述候选终端设备是与所述第一终端设备进行过远程控制，且处于所述在线状态的历史终端设备；响应于对至少一个所述候选终端设备中所述第二终端设备的选择操作，通过所述设备在线服务器向所述第二终端设备发送所述远程控制请求；
或，
获取输入的目标设备标识，所述目标设备标识为所述第二终端设备的设备标识；向所述设备在线服务器发送包含所述目标设备标识的所述远程控制请求，所述设备在线服务器用于根据所述目标设备标识查询所述第二终端设备的设备状态，并在所述第二终端设备处于所述 在线状态时，向所述第二终端设备发送所述远程控制请求。
请参考图11，其示出了本申请另一个实施例提供的远程控制装置的结构框图。该装置可以通过软件、硬件或者两者的结合实现成为第二终端设备的全部或一部分。该装置包括：
请求接收模块1101，用于通过设备在线服务器接收第一终端设备发送的远程控制请求；
第二通道建立模块1102，用于响应于对所述远程控制请求的请求接受操作，通过信令服务器与所述第一终端设备建立安全数据通道，所述安全数据通道基于WebRTC建立；
视频数据发送模块1103，用于通过所述安全数据通道向所述第一终端设备发送视频数据，所述视频数据通过对采集到的屏幕画面进行编码得到，所述第一终端设备用于对所述视频数据进行解码，并还原显示所述第二终端设备处的所述屏幕画面；
控制数据接收模块1104，用于通过所述安全数据通道接收所述第一终端设备发送的控制数据，所述控制数据由所述第一终端设备接收到还原后所述屏幕画面内的控制操作时生成；
控制响应模块1105，用于根据所述控制数据响应所述屏幕画面内的所述控制操作。
可选的，
所述控制响应模块1105，包括：
事件生成单元，用于根据所述控制数据中的所述操作类型和操作位置生成操作事件；
事件上报单元，用于向操作系统上报所述操作事件，所述操作系统用于对所述操作事件进行响应。
可选的，所述控制数据中还包括第一画面尺寸，所述第一画面尺寸为还原后所述屏幕画面的尺寸；
所述事件生成单元，用于：
根据所述第一画面尺寸以及所述屏幕画面在所述第二终端设备处的第二画面尺寸，对所述操作位置进行位置映射；
根据所述操作类型和映射后的所述操作位置生成所述操作事件。
可选的，所述第二通道建立模块1102，包括：
房间标识获取单元，用于响应于对所述远程控制请求的所述请求接受操作，获取所述设备在线服务器生成的房间标识，所述第一终端设备和所述第二终端设备具有相同房间标识；
信令服务获取单元，用于基于所述房间标识获取所述信令服务器提供的信令服务，所述信令服务指所述信令服务器为具有相同房间标识的终端设备提供消息转发服务；
第二通道建立单元，用于通过所述信令服务与所述第一终端设备建立所述安全数据通道。
可选的，所述第二通道建立单元，用于：
通过所述信令服务接收所述第一终端设备发送的第一消息，所述第一消息中包含第一网络端口信息、第一用户名、第一密码以及第一证书的第一证书验证指纹，所述第一用户名和所述第一密码由所述第一终端设备随机生成；
通过所述信令服务向所述第一终端设备发送第二消息，所述第二消息中包含第二网络端口信息、第二用户名、第二密码以及第二证书的第二证书验证指纹，所述第二用户名和所述第二密码由所述第二终端设备随机生成；
根据所述第一消息和所述第二消息建立所述安全数据通道。
可选的，所述安全数据通道包括第一安全数据通道和第二安全数据通道，所述第一安全数据通道用于传输所述控制数据，所述第二安全数据通道用于传输所述视频数据；
所述第二通道建立单元，用于：
基于所述第二用户名、所述第二密码和所述第一证书验证指纹，在第二网络端口和第一网络端口之间建立所述第二安全数据通道。
可选的，所述视频数据发送模块1103，用于：
通过预设加密密钥对所述视频数据进行加密；
通过所述第二安全数据通道向所述第一终端设备发送加密后的所述视频数据，所述第一终端设备用于通过所述预设加密密钥对接收到的所述视频数据进行解密。
请参考图12，其示出了本申请一个示例性实施例提供的终端设备的结构方框图。其可以实现成为上述实施例中的第一终端设备或第二终端设备，本申请中的终端设备可以包括一个或多个如下部件：处理器1210和存储器1220。
处理器1210可以包括一个或者多个处理核心。处理器1210利用各种接口和线路连接整个电子设备内的各个部分，通过运行或执行存储在存储器1220内的指令、程序、代码集或指令集，以及调用存储在存储器1220内的数据，执行电子设备的各种功能和处理数据。可选地，处理器1210可以采用数字信号处理(Digital Signal Processing，DSP)、现场可编程门阵列(Field－Programmable Gate Array，FPGA)、可编程逻辑阵列(Programmable Logic Array，PLA)中的至少一种硬件形式来实现。处理器1210可集成中央处理器(Central Processing Unit，CPU)、图像处理器(Graphics Processing Unit，GPU)、神经网络处理器(Neural-network Processing Unit，NPU)和调制解调器等中的一种或几种的组合。其中，CPU主要处理操作系统、用户界面和应用程序等；GPU用于负责触摸显示屏所需要显示的内容的渲染和绘制；NPU用于实现人工智能(Artificial Intelligence，AI)功能；调制解调器用于处理无线通信。可以理解的是，上述调制解调器也可以不集成到处理器1210中，单独通过一块芯片进行实现。
存储器1220可以包括随机存储器(Random Access Memory，RAM)，也可以包括只读存储器(Read-Only Memory，ROM)。可选地，该存储器1220包括非瞬时性计算机可读介质(non-transitory computer-readable storage medium)。存储器1220可用于存储指令、程序、代码、代码集或指令集。存储器1220可包括存储程序区和存储数据区，其中，存储程序区可存储用于实现操作系统的指令、用于至少一个功能的指令(比如触控功能、声音播放功能、图像播放功能等)、用于实现下述各个方法实施例的指令等；存储数据区可存储根据终端设备的使用所创建的数据(比如音频数据、电话本)等。
本申请实施例中的终端设备还包括通信组件1230。其中，通信组件1230可以为蓝牙组件、WiFi组件、NFC组件等等，用于通过有线或无线网络与外部设备(服务器或其他终端设备)进行通信。
除此之外，本领域技术人员可以理解，上述附图所示出的终端设备的结构并不构成对终端设备的限定，终端设备可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。比如，终端设备中还包括射频电路、输入单元、传感器、音频电路、扬声器、麦克风、电源等部件，在此不再赘述。
本申请实施例还提供了一种计算机可读存储介质，该存储介质存储有至少一条指令，至少一条指令用于被处理器执行以实现如上述实施例所述的远程控制方法。
本申请实施例提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述实施例提供的远程控制方法。
本领域技术人员应该可以意识到，在上述一个或多个示例中，本申请实施例所描述的功能可以用硬件、软件、固件或它们的任意组合来实现。当使用软件实现时，可以将这些功能存储在计算机可读介质中或者作为计算机可读介质上的一个或多个指令或代码进行传输。计算机可读介质包括计算机存储介质和通信介质，其中通信介质包括便于从一个地方向另一个地方传送计算机程序的任何介质。存储介质可以是通用或专用计算机能够存取的任何可用介质。
以上所述仅为本申请的可选实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (20)

  1. 一种远程控制方法，应用于第一终端设备，所述方法包括：
    通过设备在线服务器向处于在线状态的第二终端设备发送远程控制请求；
    响应于所述第二终端设备接受所述远程控制请求，通过信令服务器与所述第二终端设备建立安全数据通道，所述安全数据通道基于WebRTC建立；
    通过所述安全数据通道接收所述第二终端设备发送的视频数据，所述视频数据由所述第二终端设备对采集到的屏幕画面进行编码得到；
    对所述视频数据进行解码，还原显示所述第二终端设备处的所述屏幕画面；
    响应于还原后所述屏幕画面内的控制操作，通过所述安全数据通道向所述第二终端设备发送控制数据，所述第二终端设备用于根据所述控制数据响应所述屏幕画面内的所述控制操作。
  2. 根据权利要求1所述的方法，其中，所述响应于还原后所述屏幕画面内的控制操作，通过所述安全数据通道向所述第二终端设备发送控制数据，包括：
    响应于还原后所述屏幕画面内的所述控制操作，根据所述控制操作的操作信息生成所述控制数据，所述操作信息至少包括操作类型和操作位置；
    通过所述安全数据通道向所述第二终端设备发送所述控制数据，所述第二终端设备用于根据所述控制数据响应所述屏幕画面内的所述控制操作。
  3. 根据权利要求2所述的方法，其中，所述根据所述控制操作的操作信息生成所述控制数据，包括：
    响应于还原后所述屏幕画面的第一画面尺寸与所述屏幕画面在所述第二终端设备处的第二画面尺寸不同，根据所述控制操作的所述操作信息以及所述第一画面尺寸生成所述控制数据，所述第二终端设备用于根据所述第一画面尺寸以及所述第二画面尺寸，对所述操作位置进行位置映射。
  4. 根据权利要求1至3任一所述的方法，其中，所述响应于所述第二终端设备接受所述远程控制请求，通过信令服务器与所述第二终端设备建立安全数据通道，包括：
    响应于所述第二终端设备接受所述远程控制请求，获取所述设备在线服务器生成的房间标识，所述第一终端设备和所述第二终端设备具有相同房间标识；
    基于所述房间标识获取所述信令服务器提供的信令服务，所述信令服务指所述信令服务器为具有相同房间标识的终端设备提供消息转发服务；
    通过所述信令服务与所述第二终端设备建立所述安全数据通道。
  5. 根据权利要求4所述的方法，其中，所述通过所述信令服务与所述第二终端设备建立所述安全数据通道，包括：
    通过所述信令服务向所述第二终端设备发送第一消息，所述第一消息中包含第一网络端口信息、第一用户名、第一密码以及第一证书的第一证书验证指纹，所述第一用户名和所述第一密码由所述第一终端设备随机生成；
    通过所述信令服务接收所述第二终端设备发送的第二消息，所述第二消息中包含第二网络端口信息、第二用户名、第二密码以及第二证书的第二证书验证指纹，所述第二用户名和所述第二密码由所述第二终端设备随机生成；
    根据所述第一消息和所述第二消息建立所述安全数据通道。
  6. 根据权利要求5所述的方法，其中，所述安全数据通道包括第一安全数据通道和第二安全数据通道，所述第一安全数据通道用于传输所述控制数据，所述第二安全数据通道用于传输所述视频数据；
    所述根据所述第一消息和所述第二消息建立所述安全数据通道，包括：
    基于所述第一用户名、所述第一密码和所述第二证书验证指纹，在第一网络端口和第二网络端口之间建立所述第一安全数据通道。
  7. 根据权利要求6所述的方法，其中，所述对所述视频数据进行解码，还原显示所述第二终端设备处的所述屏幕画面，包括：
    通过预设加密密钥对所述视频数据进行解密，所述预设加密密钥是在建立所述第二安全数据通道时由所述第二终端设备设置；
    对解密后的所述视频数据进行解码，还原显示所述第二终端设备处的所述屏幕画面。
  8. 根据权利要求1至3任一所述的方法，其中，所述通过设备在线服务器向处于在线状态的第二终端设备发送远程控制请求，包括：
    显示至少一个候选终端设备的设备标识，所述候选终端设备是与所述第一终端设备进行过远程控制，且处于所述在线状态的历史终端设备；响应于对至少一个所述候选终端设备中所述第二终端设备的选择操作，通过所述设备在线服务器向所述第二终端设备发送所述远程控制请求；
    或，
    获取输入的目标设备标识，所述目标设备标识为所述第二终端设备的设备标识；向所述设备在线服务器发送包含所述目标设备标识的所述远程控制请求，所述设备在线服务器用于根据所述目标设备标识查询所述第二终端设备的设备状态，并在所述第二终端设备处于所述在线状态时，向所述第二终端设备发送所述远程控制请求。
  9. 一种远程控制方法，应用于第二终端设备，所述方法包括：
    通过设备在线服务器接收第一终端设备发送的远程控制请求；
    响应于对所述远程控制请求的请求接受操作，通过信令服务器与所述第一终端设备建立安全数据通道，所述安全数据通道基于WebRTC建立；
    通过所述安全数据通道向所述第一终端设备发送视频数据，所述视频数据通过对采集到的屏幕画面进行编码得到，所述第一终端设备用于对所述视频数据进行解码，并还原显示所述第二终端设备处的所述屏幕画面；
    通过所述安全数据通道接收所述第一终端设备发送的控制数据，所述控制数据由所述第一终端设备接收到还原后所述屏幕画面内的控制操作时生成；
    根据所述控制数据响应所述屏幕画面内的所述控制操作。
  10. 根据权利要求9所述的方法，其中，所述根据所述控制数据响应所述屏幕画面内的所述控制操作，包括：
    根据所述控制数据中的所述操作类型和操作位置生成操作事件；
    向操作系统上报所述操作事件，所述操作系统用于对所述操作事件进行响应。
  11. 根据权利要求10所述的方法，其中，所述控制数据中还包括第一画面尺寸，所述第一画面尺寸为还原后所述屏幕画面的尺寸；
    所述根据所述控制数据中的所述操作类型和操作位置生成触控事件，包括：
    根据所述第一画面尺寸以及所述屏幕画面在所述第二终端设备处的第二画面尺寸，对所述操作位置进行位置映射；
    根据所述操作类型和映射后的所述操作位置生成所述操作事件。
  12. 根据权利要求9至11任一所述的方法，其中，所述响应于对所述远程控制请求的请求接受操作，通过信令服务器与所述第二终端设备建立安全数据通道，包括：
    响应于对所述远程控制请求的所述请求接受操作，获取所述设备在线服务器生成的房间标识，所述第一终端设备和所述第二终端设备具有相同房间标识；
    基于所述房间标识获取所述信令服务器提供的信令服务，所述信令服务指所述信令服务器为具有相同房间标识的终端设备提供消息转发服务；
    通过所述信令服务与所述第一终端设备建立所述安全数据通道。
  13. 根据权利要求12所述的方法，其中，所述通过所述信令服务与所述第一终端设备建立所述安全数据通道，包括：
    通过所述信令服务接收所述第一终端设备发送的第一消息，所述第一消息中包含第一网络端口信息、第一用户名、第一密码以及第一证书的第一证书验证指纹，所述第一用户名和所述第一密码由所述第一终端设备随机生成；
    通过所述信令服务向所述第一终端设备发送第二消息，所述第二消息中包含第二网络端口信息、第二用户名、第二密码以及第二证书的第二证书验证指纹，所述第二用户名和所述第二密码由所述第二终端设备随机生成；
    根据所述第一消息和所述第二消息建立所述安全数据通道。
  14. 根据权利要求13所述的方法，其中，所述安全数据通道包括第一安全数据通道和第二安全数据通道，所述第一安全数据通道用于传输所述控制数据，所述第二安全数据通道用于传输所述视频数据；
    所述根据所述第一消息和所述第二消息建立所述安全数据通道，包括：
    基于所述第二用户名、所述第二密码和所述第一证书验证指纹，在第二网络端口和第一网络端口之间建立所述第二安全数据通道。
  15. 根据权利要求14所述的方法，其中，所述通过所述安全数据通道向所述第一终端设备发送视频数据，包括：
    通过预设加密密钥对所述视频数据进行加密；
    通过所述第二安全数据通道向所述第一终端设备发送加密后的所述视频数据，所述第一终端设备用于通过所述预设加密密钥对接收到的所述视频数据进行解密。
  16. 一种远程控制装置，应用于第一终端设备，所述装置包括：
    请求发送模块，用于通过设备在线服务器向处于在线状态的第二终端设备发送远程控制请求；
    第一通道建立模块，用于响应于所述第二终端设备接受所述远程控制请求，通过信令服务器与所述第二终端设备建立安全数据通道，所述安全数据通道基于WebRTC建立；
    视频数据接收模块，用于通过所述安全数据通道接收所述第二终端设备发送的视频数据，所述视频数据由所述第二终端设备对采集到的屏幕画面进行编码得到；
    显示模块，用于对所述视频数据进行解码，还原显示所述第二终端设备处的所述屏幕画面；
    控制数据发送模块，用于响应于还原后所述屏幕画面内的控制操作，通过所述安全数据通道向所述第二终端设备发送控制数据，所述第二终端设备用于根据所述控制数据响应所述屏幕画面内的所述控制操作。
  17. 一种远程控制装置，应用于第二终端设备，所述装置包括：
    请求接收模块，用于通过设备在线服务器接收第一终端设备发送的远程控制请求；
    第二通道建立模块，用于响应于对所述远程控制请求的请求接受操作，通过信令服务器与所述第一终端设备建立安全数据通道，所述安全数据通道基于WebRTC建立；
    视频数据发送模块，用于通过所述安全数据通道向所述第一终端设备发送视频数据，所述视频数据通过对采集到的屏幕画面进行编码得到，所述第一终端设备用于对所述视频数据进行解码，并还原显示所述第二终端设备处的所述屏幕画面；
    控制数据接收模块，用于通过所述安全数据通道接收所述第一终端设备发送的控制数据，所述控制数据由所述第一终端设备接收到还原后所述屏幕画面内的控制操作时生成；
    控制响应模块，用于根据所述控制数据响应所述屏幕画面内的所述控制操作。
  18. 一种终端设备，所述终端设备包括处理器和存储器；所述存储器存储有至少一条指令，所述至少一条指令用于被所述处理器执行以实现如权利要求1至8任一所述的远程控制方法，或，实现如权利要求9至15任一所述的远程控制方法。
  19. 一种计算机可读存储介质，所述存储介质存储有至少一条指令，所述至少一条指令用于被处理器执行以实现如权利要求1至8任一所述的远程控制方法，或，实现如权利要求9至15任一所述的远程控制方法。
  20. 一种远程控制系统，其特征在于，所述远程控制系统包括：第一终端设备、第二终端设备、设备在线服务器和信令服务器；
    所述第一终端设备通过有线或无线网络分别与所述设备在线服务器和所述信令服务器相连；
    所述第二终端设备通过有线或无线网络分别与所述设备在线服务器和所述信令服务器相连；
    所述第一终端设备用于实现如权利要求1至8任一所述的远程控制方法；
    所述第二终端设备用于实现如权利要求9至15任一所述的远程控制方法。
PCT/CN2021/107920 2020-09-23 2021-07-22 远程控制方法、装置、设备、存储介质及系统 WO2022062623A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011009850.XA CN112153140B (zh) 2020-09-23 2020-09-23 远程控制方法、装置、设备、存储介质及系统
CN202011009850.X 2020-09-23

Publications (1)

Publication Number Publication Date
WO2022062623A1 true WO2022062623A1 (zh) 2022-03-31

Family

ID=73896132

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/107920 WO2022062623A1 (zh) 2020-09-23 2021-07-22 远程控制方法、装置、设备、存储介质及系统

Country Status (2)

Country Link
CN (1) CN112153140B (zh)
WO (1) WO2022062623A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114884926A (zh) * 2022-06-14 2022-08-09 贵州大学 一种用于远程驾驶的点对点视频传输方法及系统
CN115002175A (zh) * 2022-05-27 2022-09-02 中国联合网络通信集团有限公司 一种远程控制方法、装置及存储介质
CN116208797A (zh) * 2023-03-06 2023-06-02 北京数码视讯软件技术发展有限公司 远程可视化的控制方法和系统
CN117579679A (zh) * 2024-01-15 2024-02-20 海马云（天津）信息技术有限公司 信令交互方法和装置、电子设备及存储介质

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110650199B (zh) * 2019-09-25 2022-08-12 湖南快乐阳光互动娱乐传媒有限公司 远程连接方法及装置
CN112153140B (zh) * 2020-09-23 2022-12-06 Oppo广东移动通信有限公司 远程控制方法、装置、设备、存储介质及系统
CN113515255B (zh) * 2021-05-17 2023-02-07 英华达（上海）科技有限公司 音频播放控制方法、系统、电子设备及存储介质
CN113286190A (zh) * 2021-05-19 2021-08-20 湖南快乐阳光互动娱乐传媒有限公司 一种跨网络同屏控制方法、装置和跨网络同屏系统
CN113891128A (zh) * 2021-09-18 2022-01-04 上海豹云网络信息服务有限公司 屏幕共享方法、装置以及电子设备
CN114257633A (zh) * 2021-12-20 2022-03-29 北京深思数盾科技股份有限公司 远程视频对讲方法及电子设备
CN116761212B (zh) * 2023-07-27 2024-04-23 北京小米机器人技术有限公司 图像传输控制方法、装置、终端设备及存储介质
CN117499163B (zh) * 2024-01-03 2024-03-01 成都数之联科技股份有限公司 一种基于WebRTC的服务器远程维护方法、系统及设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105357240A (zh) * 2014-08-21 2016-02-24 中兴通讯股份有限公司 远程协助的控制方法及装置
CN106161571A (zh) * 2015-04-27 2016-11-23 Tcl集团股份有限公司 一种基于WebRTC的远程操作方法及系统
CN106302646A (zh) * 2016-07-29 2017-01-04 北京小米移动软件有限公司 远程操作方法及装置
US20170104704A1 (en) * 2014-04-30 2017-04-13 Samsung Electronics Co., Ltd Method, electronic device, and storage medium for providing service
CN109408168A (zh) * 2018-09-25 2019-03-01 维沃移动通信有限公司 一种远程交互方法和终端设备
CN112153140A (zh) * 2020-09-23 2020-12-29 Oppo广东移动通信有限公司 远程控制方法、装置、设备、存储介质及系统

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102014012355A1 (de) * 2014-08-25 2016-02-25 Unify Gmbh & Co. Kg Verfahren zur Steuerung einer Multimedia-Anwendung, Softwareprodukt und Vorrichtung
CN105407369A (zh) * 2015-11-17 2016-03-16 青岛海信电器股份有限公司 一种基于Web应用的终端通信方法与装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170104704A1 (en) * 2014-04-30 2017-04-13 Samsung Electronics Co., Ltd Method, electronic device, and storage medium for providing service
CN105357240A (zh) * 2014-08-21 2016-02-24 中兴通讯股份有限公司 远程协助的控制方法及装置
CN106161571A (zh) * 2015-04-27 2016-11-23 Tcl集团股份有限公司 一种基于WebRTC的远程操作方法及系统
CN106302646A (zh) * 2016-07-29 2017-01-04 北京小米移动软件有限公司 远程操作方法及装置
CN109408168A (zh) * 2018-09-25 2019-03-01 维沃移动通信有限公司 一种远程交互方法和终端设备
CN112153140A (zh) * 2020-09-23 2020-12-29 Oppo广东移动通信有限公司 远程控制方法、装置、设备、存储介质及系统

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115002175A (zh) * 2022-05-27 2022-09-02 中国联合网络通信集团有限公司 一种远程控制方法、装置及存储介质
CN115002175B (zh) * 2022-05-27 2024-02-06 中国联合网络通信集团有限公司 一种远程控制方法、装置及存储介质
CN114884926A (zh) * 2022-06-14 2022-08-09 贵州大学 一种用于远程驾驶的点对点视频传输方法及系统
CN116208797A (zh) * 2023-03-06 2023-06-02 北京数码视讯软件技术发展有限公司 远程可视化的控制方法和系统
CN117579679A (zh) * 2024-01-15 2024-02-20 海马云（天津）信息技术有限公司 信令交互方法和装置、电子设备及存储介质
CN117579679B (zh) * 2024-01-15 2024-04-19 海马云（天津）信息技术有限公司 信令交互方法和装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN112153140B (zh) 2022-12-06
CN112153140A (zh) 2020-12-29

Similar Documents

Publication Publication Date Title
WO2022062623A1 (zh) 远程控制方法、装置、设备、存储介质及系统
US10805380B2 (en) Data transmission method and device
CN111459433B (zh) 一种传屏方法、设备及存储介质
WO2022188595A1 (zh) 应用画面的显示方法、装置、终端、投屏系统及介质
EP3163972B1 (en) Terminal, server, and terminal control method
WO2019090902A1 (zh) 屏幕共享的方法、装置、电子设备及存储介质
CN102859480A (zh) 屏幕共享
KR20150032674A (ko) 통신 시스템
CN103650458A (zh) 媒体流的传输方法、装置与系统
US9801146B2 (en) Terminal and synchronization control method among terminals
CN109104416B (zh) 一种基于pon局域网的多设备协作云服务平台
US20170171496A1 (en) Method and Electronic Device for Screen Projection
EP3217596B1 (en) Communication terminal, communication system, and data transmission method
CN112511892B (zh) 屏幕共享方法、装置、服务器及存储介质
EP3151481B1 (en) Communication terminal, communication system, and output method
JP2018505572A (ja) ビデオ通信終了方法および装置
WO2023241613A1 (zh) 通话建立方法、装置、终端、系统及可读存储介质
WO2018010222A1 (zh) 信息共享方法、信息共享装置和终端
CN109104774B (zh) 数据传输方法和系统
JPWO2017094774A1 (ja) 制御システム、通信制御方法、及びプログラム
JP2017069936A (ja) 通信端末、通信システム、出力方法、及びプログラム
TW201328243A (zh) 遠端監控系統及方法
JP2016517644A (ja) オーディオ−ビデオ用のワイヤレスドッキングシステム
CN109922100A (zh) 一种信息处理方法、终端及服务器
US11281599B2 (en) Shared peripheral devices

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21871001

Country of ref document: EP

Kind code of ref document: A1