WO2021109745A1 - 一种远程协助的方法、电子设备和系统 - Google Patents

一种远程协助的方法、电子设备和系统 Download PDF

Info

Publication number
WO2021109745A1
WO2021109745A1 PCT/CN2020/122844 CN2020122844W WO2021109745A1 WO 2021109745 A1 WO2021109745 A1 WO 2021109745A1 CN 2020122844 W CN2020122844 W CN 2020122844W WO 2021109745 A1 WO2021109745 A1 WO 2021109745A1
Authority
WO
WIPO (PCT)
Prior art keywords
terminal
interface
target device
operation interface
control
Prior art date
Application number
PCT/CN2020/122844
Other languages
English (en)
French (fr)
Inventor
于冬阳
司马经华
张海波
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to US17/782,016 priority Critical patent/US12126856B2/en
Priority to EP20897195.2A priority patent/EP4047902B1/en
Publication of WO2021109745A1 publication Critical patent/WO2021109745A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/43615—Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40—Support for services or applications
    • H04L65/403—Arrangements for multi-party communication, e.g. for conferences
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/414—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
    • H04N21/41407—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/4223—Cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G06F9/453—Help systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
    • H04N7/142—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
    • H04N2007/145—Handheld terminals

Definitions

  • the present invention relates to the field of augmented reality technology, in particular to a remote assistance method, electronic equipment and system.
  • AR augmented reality
  • the embodiments of the present application provide a remote assistance method, electronic device, and system.
  • this application provides a remote assistance method, which is executed by the first terminal during a video call between the first terminal and the second terminal, and the method includes: Send to the server; receive the device information corresponding to the target device sent by the server; virtualize the operation interface according to the received device information, and display the operation interface on the current video call interface; the operation interface is all The control panel of the target device or the operation interface of the remote control; according to the data from the second terminal, on the current video call interface, the user of the second terminal displays the operation interface on the second terminal One or more operations.
  • the virtual control interface is presented in the video in the first terminal through AR technology, so that the first terminal 20 According to the operation information sent from the second terminal, the operation actions for operating the operation interface are analyzed, and then the first terminal restores these operation actions on the video interface according to the operation information, and the first corresponding user watches these restored operation actions.
  • the method further includes: binding the operation interface with the target device. This is to prevent the virtual operation interface of the previous device from being used on the terminal after the device in the video stream changes.
  • the displaying the operation interface on the current video call interface includes: displaying the operation interface on or near the target device in the current video call interface, wherein , The operation interface is the operation interface of the remote control of the target device; or the operation interface is overlaid on the control panel of the target device in the current video call interface or replaces the control panel of the target device, wherein , The operation interface is the operation interface of the control panel of the target device.
  • the method further includes: receiving at least one first operation sent by the second terminal, and sending a control instruction to the target device, where the control instruction is used to control the target device.
  • the controller operates through a virtual interface in the video, and then sends a control instruction for one operation to the first terminal.
  • the first terminal has the function of directly controlling the target device
  • the first terminal directly Send control instructions to the target device, so that the assisting party can directly control the target device, thereby helping the assisted party more effectively.
  • the method further includes: receiving at least one second operation of the user on the operation interface; determining the visual effect corresponding to the at least one second operation, and displaying all the items on the current video call interface.
  • the visual effect; the visual effect is synchronized to the second terminal.
  • the effect corresponding to the operation is sent to the assisting party terminal, so that the assisting party knows whether the assisted party's operation is correct.
  • the method further includes: determining a control instruction corresponding to the at least one second operation according to the received at least one second operation, and sending the control instruction to the target device , The control instruction is used to control the target device.
  • an embodiment of the present application provides a terminal device, including: a transceiver, a processor, and a memory; the transceiver is used to receive and send data; the memory stores one or more programs, and the one or The multiple programs include instructions, and when the instructions are executed by the processor, the electronic device executes the methods described in each possible implementation of the first aspect.
  • an embodiment of the present application provides a remote assistance method, which is executed by the second terminal during a video call between the second terminal and the first terminal, and the method includes: Device information, the target device is the device displayed in the current video call interface; the operating interface is virtualized according to the received device information, and the operating interface is displayed on the current video call interface; the operating interface is the The control panel of the target device or the operation interface of the remote control; receiving at least one third operation of the user on the operation interface on the current video call interface; displaying the visual effects corresponding to one or more operations on the operation interface, and The visual effect is synchronized with the first terminal.
  • a video call interface including a virtual operation interface is presented on the second terminal, and the user corresponding to the second terminal operates the virtual operation interface on the second terminal, so that the first terminal synchronously presents the user’s virtual operation interface.
  • the operation of the operation interface is used to assist the user corresponding to the first terminal in adjusting and controlling the target device.
  • the method further includes: binding the operation interface with the target device.
  • the displaying the operation interface on the current video call interface includes: displaying the operation interface on the target device or around the target device in the current video call interface, wherein The operation interface is the control interface of the remote control of the target device; or the operation interface is overlaid on the control panel of the target device in the current video call interface or replaces the control panel of the target device, wherein , The operation interface is a control interface of a control panel of the target device.
  • the method further includes: according to the data from the first terminal, on the current video call interface, displaying that the user of the first terminal interacts with the virtual operation interface on the first terminal One or more operations.
  • an embodiment of the present application provides a terminal device, including: a transceiver, a processor, and a memory; the transceiver is used to receive and send data; the memory stores one or more programs, and the one or The multiple programs include instructions, and when the instructions are executed by the processor, the electronic device executes the methods described in each possible implementation of the third aspect.
  • the embodiments of the present application provide a remote assistance system, including: a target device that needs to be controlled, a cloud server, a terminal device that executes the methods described in the first aspect, and a terminal device that executes the methods described in the third aspect Each terminal device that may implement the described method.
  • the embodiments of the present application provide a readable storage medium for storing instructions.
  • the instructions When the instructions are executed, the methods described in the first aspect may be executed and the methods described in the third aspect are executed.
  • the various possible implementations of the described methods are implemented.
  • the embodiments of the present application provide a computer program device containing instructions that, when run on a terminal, enable the terminal to execute the methods described in the first aspect and execute the methods as in the third aspect.
  • a computer program device containing instructions that, when run on a terminal, enable the terminal to execute the methods described in the first aspect and execute the methods as in the third aspect.
  • FIG. 1 is a schematic structural diagram of a remote assistance system provided by an embodiment of this application.
  • FIG. 2 is a schematic diagram of a virtual operation interface displayed on a terminal according to an embodiment of the application
  • FIG. 3 is a schematic diagram of a virtual operation interface displayed on a terminal according to an embodiment of the application.
  • FIG. 4 is a flowchart of a remote assistance method provided by an embodiment of the application.
  • FIG. 5 is a flowchart of another remote assistance method provided by an embodiment of the application.
  • FIG. 6 is a schematic flowchart of a communication process between a first terminal and a second terminal according to an embodiment of the application
  • FIG. 7 is a schematic diagram of a working process among various devices of a remote assistance system provided by an embodiment of the application.
  • FIG. 8 is a schematic structural diagram of a terminal device provided by an embodiment of the present invention.
  • FIG. 1 is a schematic structural diagram of a remote assistance system provided by an embodiment of this application. As shown in FIG. 1, the system includes: a target device 10, a terminal 20, a terminal 30, and a cloud server 40.
  • the target device 10 in the embodiment of the present application is a device (target device) that needs to perform control operations, including but not limited to electronic equipment such as air conditioners, televisions, mobile phones, computers, mechanical equipment, vehicles, and other types of equipment; or the above equipment Remote control.
  • electronic equipment such as air conditioners, televisions, mobile phones, computers, mechanical equipment, vehicles, and other types of equipment; or the above equipment Remote control.
  • the embodiments of the present application will take an air conditioner as an example to describe the technical solution.
  • the terminal 20 and the terminal 30 in the embodiment of the present application include, but are not limited to, electronic devices such as mobile phones, tablets, and laptop computers.
  • the terminal 20 shoots the target device 10 through a camera and acquires an image of the target device 10 to generate a video stream including the image of the target device 10.
  • the terminal 20 sends the acquired video stream including the image of the target device 10 to the terminal 30, and also sends the video stream to the cloud server 40.
  • the cloud server 40 recognizes the target device 10 in the video stream in real time, and according to the identified target device 10. Match the data model and controllable profile data corresponding to the target device 10, and then send the data model and controllable profile data corresponding to the target device 10 to the terminal 20 and the terminal 30; the terminal 20 and the terminal 30 are based on the received
  • the data model of, a virtual operation interface is created in the video.
  • This operation interface is an image corresponding to the real remote control device or control panel that controls the target device 10, and is presented on the periphery and directly above the target device 10 in the video; And according to the received controllable configuration file, the virtual buttons and knobs on the operation interface are associated with their corresponding control instructions, so that when the user operates the buttons, knobs and other controls on the operation interface, the terminal 20 can determine each A control instruction corresponding to an operation.
  • the terminal 20 shoots the air conditioner, it sends a video including the air conditioner or an image of the air conditioner to the server, and the server recognizes the brand of the air conditioner through, for example, artificial intelligence (AI).
  • AI artificial intelligence
  • the configuration file is to execute the instruction to turn off or turn on the air conditioner after pressing the "ON/OFF” button; then the remote control model and the controllable configuration file are sent to the terminal 20, and the terminal 20 virtualizes a corresponding remote control after receiving it Monitor image and displayed on the current video interface.
  • the video interface with the virtual remote control image can be synchronized to the terminal 30 via audio and video synchronization technology.
  • the server can also send the remote controller model and controllable configuration files to the terminal 30, but the former is more in implementation. Simple and economical.
  • the terminal 30 determines the instruction corresponding to the virtual button (different operations on the same button can correspond to different instructions, such as long press And short press corresponding to different instructions), and the information corresponding to the instruction, such as operation position, operation type, time stamp, etc., is sent to the opposite terminal 20, and the operation is displayed on the virtual remote control image in the current video interface of the terminal 30
  • Corresponding visual effects for example, the button dents downwards as the user presses it, and recovers when the user raises his hand.
  • the terminal determines the corresponding visual effect according to the current user operation and the controllable configuration file; the visual effect is also synchronized to the terminal 20.
  • the user of the terminal 20 can intuitively see how to control the remote control; the terminal 20 can also send a control instruction for controlling the target device according to the received instruction to realize remote control of the target device.
  • the terminal 20 does not have the function of controlling the target device 10 (for example, the terminal 20 does not have an infrared emission function, is not connected to the target device, etc.), the user does not need the terminal 20 to directly control the target device 10, etc., the server 40.
  • the controllable configuration file may not be sent, or the sent controllable configuration file does not contain control instructions (instructions to directly control the target device).
  • the user can follow the operation of the remote assistance party displayed on the terminal 20 on the remote control model. , After learning to control the operation of the target device 10, you can directly operate on the target device 10 or on the physical remote control to realize the control of the target device 10.
  • the terminal 20 or the terminal 30 When the user operates the virtual operation interface on the terminal 20 or the terminal 30, the terminal 20 or the terminal 30 will record the operation information during the operation, such as coordinate points, timestamps, times, duration and other information, to determine the corresponding visual effects , And displayed on the current video call interface, synchronized to the opposite end of the video call.
  • the operation information such as coordinate points, timestamps, times, duration and other information
  • the user B corresponding to the terminal 30 assists the user A to control the air conditioner, and the user A tells the user B through audio and video that he wants: “Turn on the air conditioner, and Adjust the temperature to 21 degrees (assuming the startup temperature is 17 degrees)", user B operates the "ON/OFF" virtual button on the virtual remote control interface displayed in the terminal 30 to turn on the air conditioner.
  • the terminal 30 Record the operation information such as the coordinate point, operation time and number of clicks of the sliding operation during the operation; then user B operates the temperature virtual button "+" in the virtual remote control displayed on the terminal 30, and by clicking four times, The temperature is adjusted to "21 degrees".
  • the terminal 30 records operation information such as the coordinate points, operation time, and number of clicks of the sliding operation during the operation.
  • the terminal 20 determines and presents the visual effects of these operations according to the received input. For example, when the key of the virtual remote control is pressed, the display window displays the corresponding icons, numbers, etc. These visual effects are displayed in the current state of operation. On the ongoing video call interface, it is synchronized to the opposite end of the video call.
  • the terminal 20 presents the visual effects of the user's operation, and teaches the user A how to operate the terminal 20 in an intuitive manner to operate the air conditioner, turn on the air conditioner, and adjust the temperature to 21 degrees.
  • Fig. 2 is a schematic diagram of a virtual operation interface displayed on a terminal according to an embodiment of the application.
  • the terminal 20 and the terminal 30 are using a mobile phone to make a video call, and the user of the terminal 20 points the camera of the mobile phone to the air conditioner, so that an image of the air conditioner is displayed on the video call interface, as shown in FIG. 2.
  • the terminal 20 has transmission functions such as infrared, WI-FI, Bluetooth, and data flow, or has the function of directly controlling the target device.
  • the terminal 20 receives the data model (and controllable configuration file) corresponding to the air conditioner, it virtualizes a virtual control interface, the control interface includes some virtual buttons such as "ON/OFF", temperature virtual keys, humidity virtual keys, etc.; The virtual control interface is presented near the air conditioner in the video stream.
  • the terminal 30 When the user B of the terminal 30 operates the virtual buttons on the virtual control interface, the terminal 30 presents the visual effects of these operations on its interface, and synchronizes to the opposite end of the video call, the terminal 20, which corresponds to the obtained operation information
  • the control information is also sent to the terminal 20.
  • the user A of the terminal 20 can intuitively learn how to operate on the control interface by watching.
  • the terminal 20 sends the control instructions corresponding to each virtual button to the air conditioner through transmission methods such as infrared, WI-FI, Bluetooth, and data flow.
  • FIG. 3 is a schematic diagram of a virtual operation interface displayed on a terminal according to an embodiment of the application.
  • the terminal 20 after the terminal 20 receives the data model corresponding to the air conditioner, it constructs a virtual air conditioner three-dimensional structure diagram of the same size as the air conditioner in the video stream, and then covers the virtual air conditioner three-dimensional structure diagram On the air conditioner in the video stream, or replace the air conditioner in the video stream.
  • This virtual air conditioner is also synchronized to the terminal 30 by the audio and video call synchronization technology.
  • the terminal 30 When the user B operates the operable buttons on the three-dimensional structure diagram of the virtual air conditioner on the terminal 30, the terminal 30 will be virtual in the current video interface.
  • the visual effects corresponding to the operations on the three-dimensional structure diagram of the air conditioner are synchronized to the terminal 20 in real time.
  • the terminal 20 presents the visual effect.
  • the user A can understand the method of operating and controlling the air conditioner by watching these restored operation actions, and then directly operate on the control panel of the air conditioner to realize the control of the air conditioner. .
  • the terminal After the terminal receives the user's operation on the virtual operation interface, it needs to determine the object of the operation based on the coordinates, and determine the display effect based on information such as time, pressure, direction, distance, and controllable configuration files. For example, if the user presses a touch button or a virtual button, a cursor or shadow is displayed on the button, indicating that the user has selected it, or an animation effect such as an aperture or special symbol is displayed on the button, indicating that the user long presses, clicks, etc. Operation. There is a corresponding relationship between operation information and operation objects and display effects, which need to be pre-stored locally or in the cloud.
  • the terminal 20 can be a controllable configuration file, or the terminal can obtain the brand and model of the target device from the network or other devices. obtain.
  • the terminal 20 obtains the two-dimensional or three-dimensional data of the control interface such as the control device and the control panel corresponding to the target device 10, and presents the virtual data corresponding to the data in the video call interface of the terminal 30.
  • the virtual control interface is also synchronized to the terminal 30.
  • the terminal 30 receives the operation of the assisting party (user B) on the virtual control interface presented on the terminal 30, and then sends the operation information to the terminal 20, and the terminal 20 receives
  • the obtained operation information restores these operation actions on the video interface, or the terminal 30 determines and presents the user operation according to the user operation, and the same is synchronized to the opposite end, so that the user A corresponding to the terminal 20 can understand the operation and the operation by watching the operation action.
  • the method of controlling the target device can be directly operated on the control panel on the terminal 20 or on the real remote control device, because the assisted party user A can see on the real control panel or remote control on his terminal
  • the specific operation is very intuitive, just like the assisting party is teaching on the spot, which improves the efficiency of communication between the two parties.
  • Fig. 4 is a flowchart of a remote assistance method provided by an embodiment of the application.
  • the embodiment of the present application provides a remote assistance method, which is executed by the terminal 20 corresponding to the assisted person.
  • the method implements the steps as follows:
  • Step S401 Send the image including the target device to the server.
  • a communication connection is established between the terminal 20 and the terminal 30 for the transmission of data such as video, device information, and operation information.
  • the terminal 20 and the terminal 30 can directly establish a communication connection, such as D2D communication technology; it is also possible to establish a connection between the terminal 20 and the terminal 30 through a video server, such as during an existing WeChat video call. , Let the video server send the video stream of the terminal 20 to the terminal 30 for presentation.
  • a communication connection such as D2D communication technology
  • a video server such as during an existing WeChat video call.
  • the communication connection between the terminal 20 and the terminal 30 may be initiated by the terminal 20.
  • the user A corresponding to the terminal 20 cannot operate the current target device 10
  • the user B corresponding to the terminal 30 is designated to assist himself in the target device 10.
  • a video connection is made by initiating a request to the terminal 30.
  • the camera of the terminal 20 shoots the target device 10, acquires an image of the target device 10, generates a video stream including the image of the target device 10, and sends it in real time ⁇ Terminal 30.
  • the terminal 20 sends the video stream to the cloud server 40.
  • what the terminal 20 sends to the cloud server 40 may also include pictures, short videos, etc. of the target device 10.
  • the terminal 20 sets the acquired video stream, for example, intercepts N frames of pictures every few seconds or minutes, and then sends the intercepted pictures to the cloud server 40, and the cloud server 40 recognizes in real time Picture of the target device 10.
  • the operation burden of the terminal 20 is reduced, and the traffic is reduced.
  • Step S403 Receive device information corresponding to the target device sent by the server.
  • the cloud server 40 receives the video stream containing the target device 10 sent by the terminal 20, it performs AI recognition, and after identifying the brand and model of the target device 10, it determines what type of device the target device 10 is. After the device type is determined, the device information corresponding to the device is matched.
  • the device information includes the data model of the target device 10 and may also include controllable configuration file data.
  • the data model may be a real remote controller for controlling the target device 10 as shown in FIG. 1, a virtual operation interface for controlling the target device 10 as shown in FIG. 2, a control panel of the target device 10 as shown in FIG. 3, and so on.
  • the controllable configuration file can be "start”, “close”, “upward”, “downward”, “channel” and other control commands.
  • the terminal 20 can determine the control instructions corresponding to each operation.
  • the controllable profile corresponding to the “close” button is to execute an instruction to shut down the target device 10 after the “close” button is pressed.
  • step S405 an operation interface is virtualized according to the received device information, and the operation interface is displayed on the current video call interface.
  • the terminal 20 does not have the function of controlling the target device 10 (for example, the terminal 20 does not have an infrared emission function, is not connected to the target device, etc.), the user does not need the terminal 20 to directly control the target device 10, etc., the user can choose to connect to For the data content requested or received by the server, for example, by providing menu options, voice prompts, etc., the cloud server 40 may only send the data model corresponding to the target device 10. After receiving the data model, the terminal 20 presents images of physical objects such as a real remote control device, a real control panel on the target device 10, or a control device corresponding to the target device 10 in the current video interface.
  • the terminal 20 receives the data model of the air conditioner ( And controllable profile data), an air conditioner remote control matching the air conditioner can be selected to be virtualized on the terminal 20 (the control interface displayed on the video interface can be selected by the user or is default).
  • the virtual air conditioner remote control is exactly the same as the real air conditioner remote control, including the "ON/OFF" virtual buttons used to control the start or close of the air conditioner, the "-" and “+” virtual buttons to control the temperature of the air conditioner, etc. button.
  • the terminal 20 synchronizes the displayed air conditioner remote control to the terminal 30, and the user B corresponding to the terminal 30 can operate the keys of the air conditioner remote control displayed on the video interface.
  • the terminal 20 receives the data model and controllable configuration file of the air conditioner, and selects a virtual one for controlling the air conditioner on the terminal 20.
  • Virtual operation interface includes the "ON/OFF" virtual buttons used to control the startup or shutdown of the air conditioner, and the up and down virtual buttons used to control the temperature of the air conditioner.
  • the terminal 20 sends the displayed virtual operation interface to the terminal 30, and the user B corresponding to the terminal 30 can perform operations on the keys in the virtual operation interface.
  • the terminal 20 Since the terminal 20 receives the controllable configuration file of the air conditioner, if user A operates a button on the virtual operation interface on the video interface, the terminal 20 determines the corresponding control configuration file for the air conditioner according to the controllable configuration file corresponding to the button. Commands are sent to the air conditioner through transmission methods such as infrared, WI-FI, Bluetooth, and data flow to directly control the air conditioner.
  • the terminal 20 needs to bind the receiving device information with the target device 10 in the video stream, so that the virtual operation interface displayed by the terminal 20 according to the device information corresponds to the target device 10 to prevent fraud
  • the terminal 20 still uses the virtual operation interface of the previous device. Specifically, the terminal 20 continuously or intermittently sends the image of the target device 10 to the cloud server 40. If the device image recognized by the cloud server 40 does not change, the device information of the target device 10 that has been sent to the terminal 20 will not be changed; If the server cannot recognize the image of the target device 10, it will send a notification message to the terminal 20 so that the terminal 20 will stop displaying the virtual operation interface on the video.
  • Step S407 Display the operation of the second terminal user on the virtual operation interface on the current video call interface.
  • the terminal 20 After receiving the device information sent by the cloud server 40, the terminal 20 presents a virtual operation interface on the video stream. At the same time, the device information is sent to the terminal 30 (synchronized to the opposite terminal using an audio and video synchronization algorithm or sent separately), so that the terminal 30 synchronizes the video call interface including the virtual operation interface. This step is to ensure that the contents displayed on the terminal 20 and the terminal 30 are exactly the same, so that the terminal 30 also presents a virtual operation interface to assist the user A in controlling the device.
  • the cloud server 40 can directly deliver the device information to the terminal 20 and the terminal 30, and the terminal 20 does not need to forward the device information, so as to reduce the workload of the terminal 20.
  • the way in which the terminal 20 synchronizes the virtual operation interface is simpler to implement.
  • the terminal 30 After the user B corresponding to the terminal 30 operates the virtual operation interface, the terminal 30 records and sends the operation information such as the coordinate points of the sliding operation, the operation time and the number of clicks, and the pressing time during the operation to the terminal 20.
  • the visual effect corresponding to the operation is presented on the virtual operation interface of the current video call interface of the terminal 30.
  • the button sags downward as the user presses it, and resumes as the user raises the hand, and the visual effect is also synchronized.
  • the specific synchronization algorithm is the prior art, for example, the synchronization is achieved through an audio and video synchronization algorithm, such as a time stamping method, and will not be repeated here.
  • the terminal 20 establishes a smart home interconnection protocol with the target device. After the terminal 20 receives the operation information sent by the terminal 30, it parses the operation information, and then converts the operation information into instructions through the smart home interconnection protocol, and sends it to the air conditioner through transmission methods such as infrared, WI-FI, Bluetooth, and data traffic. There is no need for the user A corresponding to the terminal 20 to perform the operation again.
  • the terminal 30 can also determine the control instruction corresponding to the user's operation according to the user's operation, and send the control instruction to the terminal 20, and the terminal 20 accordingly sends the control instruction through, for example, infrared instructions, or Bluetooth instructions, or Wi-Fi instructions, or point-to-point instructions. To the target device, realize the control of the target device.
  • the terminal 30 will present the visual effect corresponding to the operation on the virtual operation interface in the current video interface. It is sent to the terminal 20 in real time, and the user corresponding to the terminal 20 can understand the method of operating and controlling the air conditioner by watching the visual effect corresponding to the operation, and then use the physical remote control, the control panel on the target device 10, and the corresponding control of the target device 10 Directly operate on physical objects such as equipment.
  • the terminal 20 will present the visual effect corresponding to the operation on the virtual operation interface in the current video interface, and send it to the terminal 30 in real time.
  • the corresponding user B can determine whether the operation of the user A corresponding to the terminal 20 is correct by viewing the visual effect corresponding to the operation, and can provide guidance in time, for example, by voice, and/or by operating a virtual operation interface.
  • the operation information needs to be sent to the terminal 30.
  • user A draws a marked circle for a certain area on the screen of the terminal 20, and the terminal 20 needs to record a series of coordinate points, timestamps and times, pressing duration, etc. on the screen at that time Operation information, showing corresponding visual effects.
  • synchronize to the opposite end for example, match the video sent by the terminal 20 through a timestamp to ensure that the content seen by the two parties on the interface will be consistent.
  • This application provides a remote assistance method executed by the terminal 20.
  • the virtual device is presented in the video of the terminal 20 through AR technology.
  • the control interface through the audio and video synchronization technology, makes the video call interface of the terminals of the communication parties present the virtual control interface, and the visual effect of the operation of the virtual control interface by the user of any terminal will be synchronized to all the video calls.
  • users who need assistance such as user A corresponding to the terminal 20, can learn about the method of operating and controlling the target device by watching these operations, and can directly operate on the control panel on the terminal 20 or on the real remote control device. Since user A of the assisted party can see the specific operations on the real control panel or remote control on his terminal, it is very intuitive, just like the assisting party is teaching on-site, which improves the efficiency of communication between the two parties. Reduce the difficulty of remote assistance.
  • Fig. 5 is a flowchart of another remote assistance method provided by an embodiment of the application.
  • the embodiment of the present application provides another remote assistance method, which is executed by the terminal 30 corresponding to the assisted person.
  • the method is implemented Proceed as follows:
  • Step S501 Receive device information of the target device.
  • a communication connection between the terminal 20 and the terminal 30 is established.
  • the communication connection between the terminal 20 and the terminal 30 can be initiated by the terminal 30.
  • the terminal 20 initiates a request. Video connection. Of course, it can also be initiated by the terminal 20.
  • the device information acquired by the terminal 30 may not be sent through the terminal 20.
  • the cloud server 40 identifies the target device 10 and obtains the device information corresponding to the device, it directly delivers the device information to the terminal 20 and the terminal 30, and the terminal 20 does not need to forward the device information, so as to reduce the workload of the terminal 20.
  • step S503 a virtual operation interface is displayed on the current video call interface.
  • the terminal 30 can virtualize the operation interface according to the received device information, if the terminal 30 receives only the data model corresponding to the target device 10 in the device information. After receiving the data model, the terminal 30 presents a virtual operation interface in the current video interface.
  • the operation interface may correspond to a real remote control device, a real control panel on the target device 10, or a control device corresponding to the target device 10, etc. Images of physical objects.
  • an air conditioner remote controller matching the air conditioner is virtualized on the terminal 30.
  • the virtual air conditioner remote control is exactly the same as the real air conditioner remote control, including the "ON/OFF" virtual buttons used to control the start or close of the air conditioner, the "-" and “+” virtual buttons to control the temperature of the air conditioner, etc. button.
  • the terminal 20 sends the displayed air conditioner remote control to the terminal 30, and the user B corresponding to the terminal 30 can operate on the keys of the air conditioner remote control on the video interface.
  • the terminal 30 it needs to be determined according to the selection of the user A corresponding to the terminal 20. If the presentation selected by the user A is the air conditioner remote controller, the air conditioner remote controller is also presented on the terminal 30, which ensures that the operation interfaces of the control device images presented on the terminal 20 and the terminal 30 are the same. Therefore, when the terminal 20 sends the device information to the terminal 30, the data model in the device information is selected so that the terminal 30 can only present the operation interface with the image of the control device displayed on the terminal 20. .
  • the terminal 30 receives the controllable configuration file of the air conditioner, it generates a corresponding control command and sends it to the terminal 20 according to the user's operation.
  • the terminal 20 can send the control command through transmission methods such as infrared, WI-FI, Bluetooth, and data traffic. Give the air conditioner to directly control the air conditioner.
  • the terminal 30 needs to bind the receiving device information with the corresponding real device in the video stream, so that the virtual operation interface displayed by the terminal 30 according to the device information corresponds to the device, so as to prevent video streaming After the device in the device changes, the virtual operation interface of the previous device is still used on the terminal 30.
  • the receiving device information with the corresponding real device in the video stream
  • the virtual operation interface displayed by the terminal 30 according to the device information corresponds to the device, so as to prevent video streaming
  • the virtual operation interface of the previous device is still used on the terminal 30.
  • Step S505 Display the user's operation on the virtual operation interface on the current video call interface.
  • the user B corresponding to the terminal 30 operates the virtual operation interface, and the terminal 30 records and sends operation information such as the coordinate points of the sliding operation, the operation time and the number of clicks, and the pressing time during the operation, and sends the operation information to the terminal 20.
  • the visual effect corresponding to the operation is presented on the virtual operation interface of the current video interface of the terminal 30.
  • the button sags downwards as the user presses it and recovers as the user raises the hand.
  • the visual effect is also synchronized to On the terminal 20, so that the user of the terminal 20 can intuitively and synchronously see the operation on the remote control.
  • the operation of the user B on the virtual operation interface on the terminal 30 may also be sent to the terminal 20 through other methods or algorithms, so as to ensure that the content seen by both parties on the interface will be consistent.
  • This application provides a remote assistance method executed by the terminal 30.
  • a video call interface including a virtual operation interface is presented on the terminal 30.
  • the user B corresponding to the terminal 30 operates the virtual operation interface on the terminal 30 to make the terminal 20 synchronously present
  • the operation of the user B on the virtual operation interface realizes the control of the target device 10 by the user A corresponding to the assist terminal 20.
  • FIG. 6 is a schematic flowchart of a communication process between a first terminal and a second terminal provided by an embodiment of the application. As shown in Figure 6, the communication process between the first terminal and the second terminal is specifically as follows:
  • step S601 a communication connection is established between the terminal 20 and the terminal 30.
  • the purpose of establishing a communication connection is to transmit data such as video, device information, and operation information.
  • the communication connection between the terminal 20 and the terminal 30 may be initiated by the terminal 20.
  • the user B corresponding to the terminal 30 is designated to assist Operate the target device 10 by himself, and initiate a request to the terminal 30 to make a video connection.
  • the communication connection between the terminal 20 and the terminal 30 may be initiated by the terminal 30.
  • the terminal 30 When the user B corresponding to the terminal 30 learns that the user A corresponding to the terminal 20 cannot operate a certain device, the The terminal 20 initiates a request to make a video connection.
  • the terminal 20 and the terminal 30 can directly establish a communication connection, such as D2D communication technology; it is also possible to establish a connection between the terminal 20 and the terminal 30 through a video server.
  • a communication connection such as D2D communication technology
  • the video server sends the video stream of the terminal 20 to the terminal 30 for presentation.
  • the specific implementation manner is not limited in this application.
  • step S602 the camera of the terminal 20 is activated to obtain a video stream including the target device 10.
  • the camera of the terminal 20 is activated to shoot the target device 10, obtain an image of the target device 10, and generate a video stream including the image of the target device 10.
  • step S603 the terminal 20 sends the acquired video stream to the terminal 30 in real time.
  • the video stream of the terminal 20 can be sent to the terminal 30 in real time for presentation, thereby achieving the same content displayed between the terminal 0 and the terminal 30.
  • step S604 the terminal 20 sends the acquired video stream to the cloud server 40 in real time.
  • what the terminal 20 sends to the cloud server 40 may also include pictures, short videos, etc. of the target device 10.
  • the terminal 20 sets the acquired video stream, for example, intercepts N frames of pictures every few seconds or minutes, and then sends the intercepted pictures to the cloud server 40, and the cloud server 40 recognizes in real time Picture of the target device 10.
  • the operation burden of the terminal 20 is reduced, and the traffic is reduced.
  • step S605 the cloud server 40 recognizes the target device 10 in the image according to the received image including the target device 10, and matches the device information corresponding to the device according to the target device 10.
  • the cloud server 40 receives the video stream containing the target device 10 sent by the terminal 20, it performs AI recognition, and after identifying the brand and model of the target device 10, it determines what type of device the target device 10 is. After the device type is determined, the device information corresponding to the device is matched.
  • the device information includes the data model of the target device 10 and may also include controllable configuration file data.
  • the data model may be a real remote controller for controlling the target device 10 as shown in FIG. 1, a virtual operation interface for controlling the target device 10 as shown in FIG. 2, a control panel of the target device 10 as shown in FIG. 3, and so on.
  • the controllable configuration file can be "start”, “close”, “upward”, “downward”, “channel” and other control commands.
  • Each control command in the controllable configuration file can be controlled with the buttons on the operation interface in the data model , Knobs, etc. are associated, so that when the user operates the buttons, knobs, and other controls on the operation interface, the terminal 20 can determine the control instructions corresponding to each operation.
  • the controllable profile corresponding to the “close” button is to execute an instruction to shut down the target device 10 after the “close” button is pressed.
  • step S606 the cloud server 40 sends the device information to the terminal 20.
  • the cloud server 40 can only send The data model corresponding to the target device 10. If the terminal 20 has transmission functions such as infrared, WI-FI, Bluetooth, data flow, etc., or has the function of directly controlling the target device, the cloud server 40 sends the data model and controllable configuration file corresponding to the target device 10.
  • step S607 the terminal 20 virtualizes an operation interface on the video stream according to the device information.
  • the terminal 20 After receiving the data model, the terminal 20 presents images of physical objects such as the real remote control device, the real control panel on the target device 10, and the control device corresponding to the target device 10 in the current video interface.
  • the terminal 20 receives the data model of the air conditioner , after the terminal 20 receives the data model of the air conditioner , choose to virtually create an air conditioner remote control matching the air conditioner on the terminal 20 (the control interface displayed on the video interface can be selected by the user, or is the default).
  • the virtual air conditioner remote control is exactly the same as the real air conditioner remote control, including the "ON/OFF” virtual buttons used to control the start or close of the air conditioner, the "-" and “+” virtual buttons to control the temperature of the air conditioner, etc. button.
  • the terminal 20 sends the displayed air conditioner remote control to the terminal 30, and the user B corresponding to the terminal 30 can operate on the keys of the air conditioner remote control on the video interface.
  • the terminal 20 and the terminal 30 do not have controllable configuration files, when the user operates the button of the air conditioner remote control in the video interface, the terminal still presents the visual effect corresponding to the operation, and then sends it to the opposite terminal in real time for the user A can watch the operation effect and learn to control the operation of the target device 10, and at the same time, the user B can see the operation of the user A, which can provide real-time guidance and assistance to the user A.
  • the current video interface can not only display the real remote control device, the real control panel on the target device 10, the real control device corresponding to the target device 10, etc.
  • the image can also present a virtual operation interface as shown in Figure 2.
  • the terminal 20 receives the data model and controllable configuration file of the air conditioner, and selects a virtual one for controlling the air conditioner on the terminal 20.
  • Virtual operation interface includes the "ON/OFF" virtual buttons used to control the startup or shutdown of the air conditioner, and the up and down virtual buttons used to control the temperature of the air conditioner.
  • the terminal 20 sends the displayed virtual operation interface to the terminal 30, and the user B corresponding to the terminal 30 can perform operations on the keys in the virtual operation interface.
  • the terminal 20 Since the terminal 20 receives the controllable configuration file of the air conditioner, if the user A operates a button on the virtual operation interface on the video interface, the terminal 20 controls the air conditioner according to the command of the controllable configuration file corresponding to the button, such as infrared rays, WI-FI, Bluetooth, data flow and other transmission methods are sent to the air conditioner to directly control the air conditioner.
  • the button such as infrared rays, WI-FI, Bluetooth, data flow and other transmission methods are sent to the air conditioner to directly control the air conditioner.
  • the terminal 20 needs to bind the receiving device information with the target device 10 in the video stream, so that the virtual operation interface displayed by the terminal 20 according to the device information corresponds to the target device 10 to prevent fraud
  • the terminal 20 still uses the virtual operation interface of the previous device. Specifically, the terminal 20 continuously or intermittently sends the image of the target device 10 to the cloud server 40. If the device image recognized by the cloud server 40 does not change, the device information of the target device 10 that has been sent to the terminal 20 will not be changed; If the server cannot recognize the image of the target device 10, it will send a notification message to the terminal 20 so that the terminal 20 will stop displaying the virtual operation interface on the video.
  • step S608 the terminal 20 sends the device information to the terminal 30.
  • the device information acquired by the terminal 30 may not be sent through the terminal 20.
  • the cloud server 40 recognizes the device and obtains the device information corresponding to the device, it directly delivers the device information to the terminal 20 and the terminal 30, and the terminal 20 does not need to forward the device information, so as to reduce the workload of the terminal 20.
  • step S609 the terminal 30 virtualizes an operation interface on the video stream according to the device information.
  • the terminal 30 receives only the data model corresponding to the target device 10 in the device information. After receiving the data model, the terminal 30 presents images of physical objects such as the real remote control device, the real control panel on the target device 10, and the control device corresponding to the target device 10 in the current video interface.
  • an air conditioner remote control matching the air conditioner is virtualized on the terminal 30 Device.
  • the virtual air conditioner remote control is exactly the same as the real air conditioner remote control, including the "ON/OFF" virtual buttons used to control the start or close of the air conditioner, the "-" and “+” virtual buttons to control the temperature of the air conditioner, etc. button.
  • the terminal 20 sends the displayed air conditioner remote control to the terminal 30, and the user B corresponding to the terminal 30 can operate on the keys of the air conditioner remote control on the video interface.
  • the visual effect corresponding to the operation is presented on the terminal 30, which is sent to the terminal 20 in real time, so that the user A can watch the operation effect and learn to control the operation of the target device 10 .
  • the terminal 30 it needs to be determined according to the selection of the user A corresponding to the terminal 20. If the presentation selected by the user A is the air conditioner remote controller, the air conditioner remote controller is also presented on the terminal 30, which ensures that the operation interfaces of the control device images presented on the terminal 20 and the terminal 30 are the same. Therefore, when the terminal 20 sends the device information to the terminal 30, the data model in the device information is selected so that the terminal 30 can only present the operation interface with the image of the control device displayed on the terminal 20. .
  • the terminal 30 receives a data model and a controllable configuration file corresponding to the target device 10 in the device information. After receiving the data model and the controllable configuration file, the terminal 30 can not only display the real remote control device, the real control panel on the target device 10, the real control device corresponding to the target device 10, and other physical objects in the current video interface.
  • the image can also present a virtual operation interface as shown in Figure 2.
  • the terminal 30 virtualizes it on the terminal 30 according to the selection of the terminal 20.
  • a virtual operation interface for controlling the air conditioner is presented.
  • the virtual operation interface includes the "ON/OFF" virtual buttons used to control the startup or shutdown of the air conditioner, and the up and down virtual buttons used to control the temperature of the air conditioner.
  • the terminal 20 sends the displayed virtual operation interface to the terminal 30, and the user B corresponding to the terminal 30 can perform operations on the keys in the virtual operation interface.
  • the terminal 30 Since the terminal 30 receives the controllable configuration file of the air conditioner, if the user B operates a button on the virtual operation interface on the video interface, after the generated operation information is sent to the terminal 20, the terminal 20 will control the configuration according to the button.
  • the instructions for controlling the air conditioner in the file are sent to the air conditioner through transmission methods such as infrared, WI-FI, Bluetooth, and data flow to directly control the air conditioner.
  • the terminal 30 needs to bind the receiving device information with the real device in the video stream, so that the virtual operation interface displayed by the terminal 30 according to the device information corresponds to the device, so as to prevent the video stream from appearing in the video stream. After the device is changed, the virtual operation interface of the previous device is still used on the terminal 30.
  • step S610 the terminal 30 records the operation information of the user operating on the virtual operation interface.
  • the user B corresponding to the terminal 30 operates the virtual operation interface, and the terminal 30 records operation information such as coordinate points, operation time, number of clicks, and pressing duration of the sliding operation during the operation. After determining that the user B has completed the operation in this control process, the terminal 30 sends the recorded operation information to the terminal 20.
  • the visual effect corresponding to the operation is presented on the virtual operation interface of the current video interface of the terminal 30. For example, the button sags downwards as the user presses it and recovers as the user raises the hand.
  • the visual effect is also synchronized to On the terminal 20, so that the user of the terminal 20 can intuitively see how to manipulate the remote control.
  • step S611 the terminal 30 sends operation information and/or video to the terminal 20.
  • step S612 the terminal 20 analyzes the operation information, and converts the analyzed result into an operation action for operating the operation interface.
  • the terminal 20 After receiving the operation information, the terminal 20 parses the operation information, and tells the user A how to operate on the terminal 20 by means of video demonstrations, etc., so as to control the target device 10.
  • the visual effect corresponding to the operation is presented on the virtual operation interface of the current video interface of the terminal 30. For example, the button sags downwards as the user presses it and recovers as the user raises the hand.
  • the visual effect is also synchronized to On the terminal 20, so that the user of the terminal 20 can intuitively see how to manipulate the remote control.
  • the terminal 20 establishes a smart home interconnection protocol with the target device. After the terminal 20 receives the operation information sent by the terminal 30, it parses the operation information, and then converts the operation information into instructions through the smart home interconnection protocol, and sends it to the air conditioner through transmission methods such as infrared, WI-FI, Bluetooth, and data traffic. There is no need for the user A corresponding to the terminal 20 to perform the operation again.
  • the terminal 30 will present the visual effect corresponding to the operation on the virtual operation interface in the current video interface. It is sent to the terminal 20 in real time, and the user corresponding to the terminal 20 can understand the method of operating and controlling the air conditioner by watching the visual effect corresponding to the operation, and then use the physical remote control, the control panel on the target device 10, and the corresponding control of the target device 10 Directly operate on physical objects such as equipment.
  • step S613 the terminal 20 sends operation information and/or video to the terminal 30.
  • step S614 the terminal 30 presents an operation interface on the terminal 20.
  • the terminal 20 After user A operates the video stream on the terminal 20, the terminal 20 will present the visual effect corresponding to the operation on the virtual operation interface in the current video interface, and send it to the terminal 30 in real time, and the user B corresponding to the terminal 30 can watch the The visual effect corresponding to the operation is used to determine whether the operation of the user A corresponding to the terminal 20 is correct.
  • the operation information needs to be sent to the terminal 30.
  • the user A draws a marked circle for a certain area on the screen of the terminal 20, and the terminal 20 needs to record a series of operation information such as coordinate points, timestamps, and times on the screen at that time.
  • the operation information is sent to the terminal 30 through a dedicated communication channel, and is matched with the video sent by the terminal 20 through a timestamp, so as to ensure that the content seen by both parties on the interface will be consistent.
  • the remote assistance system provided by the embodiments of the present application is used to implement the above method embodiments. Specifically, after the terminal 20 obtains the video stream including the target device 10, the video stream is sent to the terminal 30 and the server 40, and the server 40 Obtain the two-dimensional or three-dimensional data of the real control interface such as the control device and control panel corresponding to the target device 10, and then send it to the terminal 20 and the terminal 30, and use AR technology to present the virtual control in the video in the terminal 20 and the terminal 30 Interface, the terminal 30 receives the operation of the assisting party (user B) on the virtual control interface, and then sends the operation information to the terminal 20.
  • the assisting party user B
  • the terminal 20 restores these operation actions on the video interface according to the received operation information, and the terminal 20 corresponds to By watching these restored operation actions, user A can understand how to operate and control the target device, and can directly operate on the control panel on the terminal 20 or on a real remote control device. Because the assisted user A can Seeing the specific operations on the real control panel or remote control on your own terminal is very intuitive, just like the assisting party is teaching on the spot, which improves the efficiency of communication between the two parties.
  • this application lists an embodiment to help understand: suppose that the old man at home is the user A corresponding to the terminal 20, and the son in the office corresponds to the terminal 30 For user B, both smartphones have installed the "AR Remote Assistance" APP.
  • old man A When old man A needs to turn on the sleep mode of the air conditioner, but cannot find the remote control or can not operate the remote control, old man A opens the mobile phone APP "AR Remote Assistance" and sends a request for help to his son B; Quasi air conditioner, the image of the air conditioner appears on the phone interface of son B, and the remote control of this air conditioner pops up on the interface; son B operates the corresponding button on the remote control interface, and this command is sent back to the mobile phone of old man A and watched by old man A While his son is operating the remote control on his mobile phone, old man A's mobile phone starts the sleep mode of the air conditioner through the smart home interconnection protocol after receiving the air conditioner control command.
  • this application lists an embodiment to help understand: suppose that the old man at home is the user A corresponding to the terminal 20, and the son in the office corresponds to the terminal 30 For user B, both smartphones have installed the "AR Remote Assistance” APP.
  • the remote control of the air conditioner is in old man A’s hand, old man A opens the mobile phone APP "AR Remote Assistance", and sends a call for help to son B; old man A points his phone at the remote control, son B’s The image of the remote control appears on the mobile phone interface; son B operates the corresponding button on the remote control interface, and this command is sent back to the mobile phone of old man A.
  • the phone of old man A is After receiving the air conditioner control command, the sleep mode of the air conditioner is turned on through the smart home interconnection protocol.
  • the embodiment of the present application also provides a remote control system to implement the above method embodiments. Specifically, it includes the target device 10 that needs to be controlled, the cloud server 40, the remote control device 70 executed by the terminal 20, and the remote control device 70 executed by the terminal 20. 30 implementation of the remote control device 80.
  • the system can superimpose a virtual operation interface on the video stream obtained by the terminal 20, which helps the assisting party and the assisted party to communicate, and makes it easier for the assisting party to assist the assisted party in operating the equipment.
  • FIG. 7 is a schematic diagram of a work flow between various devices of a remote assistance system provided by an embodiment of the application.
  • the remote assistance process of the system is mainly divided into a communication establishment phase, an AI recognition phase, an AR recognition phase, and a smart home interconnection phase.
  • the A party represents the terminal 20
  • the B party represents the terminal 30
  • the cloud represents the cloud server 40
  • the item represents the target device 10.
  • Step 1 Establish a communication connection between party A and party B.
  • the purpose of establishing a communication connection is to transmit data such as video, device information, and operation information.
  • the communication connection between the terminal 20 and the terminal 30 may be initiated by the terminal 20.
  • the user B corresponding to the terminal 30 is designated to assist Operate the target device 10 by himself, and initiate a request to the terminal 30 to make a video connection.
  • the communication connection between the terminal 20 and the terminal 30 may be initiated by the terminal 30.
  • the terminal 30 When the user B corresponding to the terminal 30 learns that the user A corresponding to the terminal 20 cannot operate a certain device, the The terminal 20 initiates a request to make a video connection.
  • the terminal 20 and the terminal 30 can directly establish a communication connection, such as D2D communication technology; it is also possible to establish a connection between the terminal 20 and the terminal 30 through a video server.
  • a communication connection such as D2D communication technology
  • the video server sends the video stream of the terminal 20 to the terminal 30 for presentation.
  • the specific implementation manner is not limited in this application.
  • Step 2 Party A obtains the items to be controlled through the camera.
  • the camera of the terminal 20 is activated to take a picture of the target device 10, obtain a two-dimensional or three-dimensional structure diagram of the target device 10, and generate a two-dimensional or three-dimensional structure diagram including the target device 10 Video stream.
  • Step 3 Party B obtains the video content shot by Party A in real time during the video call of Party A.
  • the video stream of the terminal 20 can be sent to the terminal 30 in real time for presentation, thereby achieving the same content displayed between the terminal 0 and the terminal 30.
  • the first step the cloud recognizes the items in the video in real time.
  • What the terminal 20 sends to the cloud server 40 may also include pictures, short videos, etc. of the target device 10.
  • the terminal 20 sets the acquired video stream, for example, intercepts N frames of pictures every few seconds or minutes, and then sends the intercepted pictures to the cloud server 40, and the cloud server 40 recognizes in real time Picture of the target device 10.
  • the operation burden of the terminal 20 is reduced, and the traffic is reduced.
  • the second step After identifying the item's information, match the item's corresponding operation information, which includes information such as on, off, up, down, channel, etc.
  • the device information includes the data model of the target device 10 and may also include controllable configuration file data.
  • the data model may be a real remote controller for controlling the target device 10 as shown in FIG. 1, a virtual operation interface for controlling the target device 10 as shown in FIG. 2, a control panel of the target device 10 as shown in FIG. 3, and so on.
  • the controllable configuration file can be "start”, “close”, “upward”, “downward”, “channel” and other control commands.
  • Each control command in the controllable configuration file can be controlled with the buttons on the operation interface in the data model , Knobs, etc. are associated, so that when the user operates the buttons, knobs, and other controls on the operation interface, the terminal 20 can determine the control instructions corresponding to each operation.
  • the controllable profile corresponding to the “close” button is to execute an instruction to shut down the target device 10 after the “close” button is pressed.
  • the first step Party A obtains the item information of the item from the cloud, then implements a virtual operation panel, and binds the operation panel to the item in the video.
  • the terminal 20 does not have the function of controlling the target device 10 (for example, the terminal 20 does not have an infrared emission function, is not connected to the target device, etc.), the user does not need the terminal 20 to directly control the target device 10, etc.
  • the cloud The server 40 may only send the data model corresponding to the target device 10.
  • the terminal 20 presents images of physical objects such as the real remote control device, the real control panel on the target device 10, and the control device corresponding to the target device 10 in the current video interface.
  • the terminal 20 presents images of physical objects such as the real remote control device, the real control panel on the target device 10, and the control device corresponding to the target device 10 in the current video interface.
  • the terminal 20 sends the displayed air conditioner remote control to the terminal 30, and the user B corresponding to the terminal 30 can operate on the keys of the air conditioner remote control on the video interface.
  • the terminal 20 and the terminal 30 do not have controllable configuration files, when the user operates the button of the air conditioner remote control in the video interface, the terminal still presents the visual effect corresponding to the operation, and then sends it to the opposite terminal in real time for the user A can watch the operation effect and learn to control the operation of the target device 10, and at the same time, the user B can see the operation of the user A, which can provide real-time guidance and assistance to the user A.
  • the cloud server 40 sends the data model and controllable corresponding to the target device 10 Configuration file.
  • the current video interface can not only display the real remote control device, the real control panel on the target device 10, the real control device corresponding to the target device 10, etc.
  • the image can also present a virtual operation interface as shown in Figure 2.
  • the terminal 20 receives the data model and controllable configuration file of the air conditioner, and selects a virtual one for controlling the air conditioner on the terminal 20.
  • the virtual operation interface includes the "ON/OFF" virtual buttons used to control the startup or shutdown of the air conditioner, and the up and down virtual buttons used to control the temperature of the air conditioner.
  • the terminal 20 sends the displayed virtual operation interface to the terminal 30, and the user B corresponding to the terminal 30 can perform operations on the keys in the virtual operation interface.
  • the terminal 20 Since the terminal 20 receives the controllable configuration file of the air conditioner, if the user A operates a button on the virtual operation interface on the video interface, the terminal 20 controls the air conditioner according to the command of the controllable configuration file corresponding to the button, such as infrared rays, WI-FI, Bluetooth, data flow and other transmission methods are sent to the air conditioner to directly control the air conditioner.
  • the button such as infrared rays, WI-FI, Bluetooth, data flow and other transmission methods are sent to the air conditioner to directly control the air conditioner.
  • the second step Party B obtains the item information sent by Party A, and also uses a virtual operation panel on the video to obtain the same content displayed on the display interface of Party A.
  • the device information acquired by the terminal 30 may not be sent through the terminal 20.
  • the cloud server 40 recognizes the device and obtains the device information corresponding to the device, it directly delivers the device information to the terminal 20 and the terminal 30, and the terminal 20 does not need to forward the device information, so as to reduce the workload of the terminal 20.
  • an air conditioner remote control matching the air conditioner is virtualized on the terminal 30.
  • the virtual air conditioner remote control is exactly the same as the real air conditioner remote control, including the "ON/OFF" virtual buttons used to control the start or close of the air conditioner, the "-" and “+” virtual buttons to control the temperature of the air conditioner, etc. button.
  • the terminal 20 sends the displayed air conditioner remote control to the terminal 30, and the user B corresponding to the terminal 30 can operate on the keys of the air conditioner remote control on the video interface.
  • a virtual operation interface for controlling the air conditioner is virtualized on the terminal 30.
  • the virtual operation interface includes the "ON/OFF" virtual buttons used to control the startup or shutdown of the air conditioner, and the up and down virtual buttons used to control the temperature of the air conditioner.
  • the terminal 20 sends the displayed virtual operation interface to the terminal 30, and the user B corresponding to the terminal 30 can perform operations on the keys in the virtual operation interface.
  • the terminal 30 Since the terminal 30 receives the controllable configuration file of the air conditioner, if the user B operates a button on the virtual operation interface on the video interface, after the generated operation information is sent to the terminal 20, the terminal 20 will control the configuration according to the button.
  • the instructions for controlling the air conditioner in the file are sent to the air conditioner through transmission methods such as infrared, WI-FI, Bluetooth, and data flow to directly control the air conditioner.
  • Step 3 Party B conducts remote operations on the virtual operation panel and records the operation information, such as coordinate information, timestamp, number of clicks and other information.
  • the user B corresponding to the terminal 30 operates the virtual operation interface, and the terminal 30 records operation information such as coordinate points, operation time, number of clicks, and pressing duration of the sliding operation during the operation. After determining that the user B has completed the operation in this control process, the terminal 30 sends the recorded operation information to the terminal 20.
  • the visual effect corresponding to the operation is presented on the virtual operation interface of the current video interface of the terminal 30. For example, the button sags downwards as the user presses it and recovers as the user raises the hand.
  • the visual effect is also synchronized to On the terminal 20, so that the user of the terminal 20 can intuitively see how to manipulate the remote control.
  • Step 4 After Party A obtains the operation information of Party B, it parses out that the operation information is a specific key operation, such as on, off, up, down, channel and other key operations.
  • the terminal 20 After receiving the operation information sent by the terminal, the terminal 20 parses out the operation information, and tells the user A how to operate on the terminal 20 through a video demonstration or the like, so as to control the target device 10.
  • the visual effect corresponding to the operation is presented on the virtual operation interface of the current video interface of the terminal 30. For example, the button sags downwards as the user presses it and recovers as the user raises the hand.
  • the visual effect is also synchronized to On the terminal 20, so that the user of the terminal 20 can intuitively see how to manipulate the remote control.
  • the first step Party A controls the corresponding items according to the key operation.
  • the terminal 20 establishes a smart home interconnection protocol with the target device. After the terminal 20 receives the operation information sent by the terminal 30, it parses the operation information, and then converts the operation information into instructions through the smart home interconnection protocol, and sends it to the air conditioner through transmission methods such as infrared, WI-FI, Bluetooth, and data traffic. There is no need for the user A corresponding to the terminal 20 to perform the operation again.
  • the terminal 30 will present the visual effect corresponding to the operation on the virtual operation interface in the current video interface, and send it in real time.
  • the user corresponding to the terminal 20 can understand the method of operating and controlling the air conditioner by watching the visual effect corresponding to the operation, and then use the physical remote control, the control panel on the target device 10, the control device corresponding to the target device 10, etc. Direct operations on physical objects.
  • FIG. 8 is a schematic structural diagram of a terminal device provided by an embodiment of the present invention, which is used to implement the above method embodiments.
  • a terminal target device 800 shown in FIG. 8 may be the aforementioned terminal 20 and terminal 30, and includes a sensor 801, a display 802, a processor 803, a memory 804, a communication interface 805, and a bus 806.
  • the processor 803, the memory 804, and the communication interface 805 in the terminal device can establish a communication connection through the bus 806.
  • the sensor 801 is used to obtain the video stream including the device and the device information corresponding to the device.
  • the sensor 801 may include a camera or the like.
  • the display 802 is used to display processed data, such as video and a virtual operation interface.
  • the processor 803 may be a central processing unit (CPU).
  • the memory 804 may include a volatile memory (volatile memory), such as a random-access memory (random-access memory, RAM); the memory may also include a non-volatile memory (non-volatile memory), such as a read-only memory (read-only memory). Only memory, ROM), flash memory, hard disk drive (HDD) or solid state drive (SSD); the memory 804 may also include a combination of the foregoing types of memories.
  • volatile memory such as a random-access memory (random-access memory, RAM)
  • non-volatile memory such as a read-only memory (read-only memory).
  • SSD solid state drive
  • the remote control methods provided in the foregoing embodiments are all executed by the processor 803.
  • Data such as video streams, device information, and operation information will be stored in the memory 804.
  • the memory 804 will also be used to store program instructions corresponding to the remote control method described in the foregoing embodiment and the like executed by the processor 803.
  • various aspects or features of the embodiments of the present application can be implemented as methods, devices, or products using standard programming and/or engineering techniques.
  • article of manufacture used in this application encompasses a computer program accessible from any computer-readable device, carrier, or medium.
  • computer-readable media may include, but are not limited to: magnetic storage devices (for example, hard disks, floppy disks, or tapes, etc.), optical disks (for example, compact discs (CD), digital versatile discs (DVD)) Etc.), smart cards and flash memory devices (for example, erasable programmable read-only memory (EPROM), cards, sticks or key drives, etc.).
  • various storage media described herein may represent one or more devices and/or other machine-readable media for storing information.
  • the term "machine-readable medium” may include, but is not limited to, wireless channels and various other media capable of storing, containing, and/or carrying instructions and/or data.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrated with one or more available media.
  • the usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).
  • the size of the sequence number of the above-mentioned processes does not mean the order of execution.
  • the execution order of the processes should be determined by their functions and internal logic, and should not be dealt with.
  • the implementation process of the embodiments of the present application constitutes any limitation.
  • the disclosed system, device, and method can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solutions of the embodiments of the present application are essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium.
  • Including several instructions to enable a computer device (which may be a personal computer, a server, or an access network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Telephonic Communication Services (AREA)
  • Selective Calling Equipment (AREA)
  • Telephone Function (AREA)

Abstract

本申请提供了一种远程协助的方法、电子设备和系统，应用于如手机、平板、笔记本等设备上。其中，所述方法在第一终端与第二终端进行视频通话过程中，由所述第一终端执行，包括：将包括目标设备的图像发送给服务器；接收服务器发送的目标设备对应的设备信息；根据接收的设备信息虚拟出操作界面，并在当前视频通话界面上显示操作界面；操作界面为目标设备的控制面板或遥控器的操作界面；根据来自第二终端的数据，在当前视频通话界面上，显示第二终端的用户在第二终端上对所述操作界面的一个或多个操作。本申请实施例通过AR技术，在终端中的视频中呈现该虚拟控制界面，并根据操作信息，可直接在终端上的控制面板或遥控设备上进行操作。

Description

一种远程协助的方法、电子设备和系统
本申请要求于2019年12月04日提交中国专利局、申请号为201911230080.9、申请名称为“一种远程协助的方法、电子设备和系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及增强现实技术领域，尤其涉及一种远程协助的方法、电子设备和系统。
背景技术
随着增强现实(augmented reality，AR)技术的快速发展，在娱乐、教育、商业、旅游、通信、游戏等领域的应用越来越广泛。从刚开始的好玩的如脸萌类应用、AR游戏到现在AR测量、AR家居、AR通信等，AR的应用已经和移动终端的结合更加密切，给消费者带来的虚实结合的体验也更趋完善。
对于现有技术中，用户可以通过手机对家用电器的连接、管理和控制，可以实现一键控制家用电器的状态，无论是在家还是在外面，都能很方便的进行控制和管理。但是，在不会使用电器功能、不清楚如何操控等的情况时，一般只能通过电话或视频电话获得帮助，学习如何操作，但在操作比较复杂可能不容易理解，或者因为需帮助方有特定困难例如视力听力障碍的情况下，这种指导方式无法提供有效帮助。
发明内容
为了克服上述问题，本申请的实施例提供了一种远程协助的方法、电子设备和系统。
为了达到上述目的，本申请的实施例采用如下技术方案：
第一方面，本申请提供一种远程协助的方法，所述方法在第一终端与第二终端进行视频通话过程中，由所述第一终端执行，所述方法包括：将包括目标设备的图像发送给服务器；接收所述服务器发送的所述目标设备对应的设备信息；根据接收的所述设备信息虚拟出操作界面，并在当前视频通话界面上显示所述操作界面；所述操作界面为所述目标设备的控制面板或遥控器的操作界面；根据来自所述第二终端的数据，在当前视频通话界面上，显示所述第二终端的用户在所述第二终端上对所述操作界面的一个或多个操作。
本申请实施例通过获取包括目标设备的视频流和根据该视频流获取该设备的相关的设备信息后，通过AR技术，在第一终端中的视频中呈现该虚拟控制界面，使得第一终端20根据从第二终端发送的操作信息，解析出对操作界面进行操作的操作动作，然后第一终端根据操作信息在视频界面上还原这些操作动作，第一对应的用户观看了这些还原的操作动作，就了解了操作和控制目标设备的方法，即可直接在第一终端上的控制面板进行操作或在真实的遥控设备上进行操作，由于被协助方用户能在自己的终端上看到在真 实控制面板或者遥控器上的具体操作，非常直观，提高了双方交流效率。
在另一个可能的实现中，所述方法还包括：将所述操作界面与所述目标设备进行绑定。以防止当视频流中的设备发生变化后，终端上仍沿用之前设备的虚拟操作界面。
在另一个可能的实现中，所述在当前视频通话界面上显示所述操作界面，包括：将所述操作界面显示在当前视频通话界面中所述目标设备上或所述目标设备的附近，其中，所述操作界面为所述目标设备的遥控器的操作界面；或者将所述操作界面覆盖在当前视频通话界面中的所述目标设备的控制面板上或替换所述目标设备的控制面板，其中，所述操作界面为所述目标设备的控制面板的操作界面。
在另一个可能的实现中，所述方法还包括：接收所述第二终端发送的至少一个第一操作，向所述目标设备发送控制指令，所述控制指令用于控制所述目标设备。
本申请实施例，控制方通过在视频中虚拟界面进行操作，然后将该操作对一个的控制指令发送给第一终端，在第一终端有与目标设备进行直接控制的功能时，第一终端直接发送控制指令给目标设备，以实现协助方直接对目标设备进行控制，从而更加有效帮助被协助方。
在另一个可能的实现中，所述方法还包括：接收用户对所述操作界面的至少一个第二操作；确定所述至少一个第二操作对应的视觉效果，并在当前视频通话界面上显示所述视觉效果；向所述第二终端同步所述视觉效果。
本申请实施例，在被协助方对视频中虚拟界面进行操作后，将操作对应的效果发送给协助方终端上，以让协助方知晓被协助方的操作是否正确。
在另一个可能的实现中，所述方法还包括：根据接收到的所述至少一个第二操作，确定对应所述的至少一个第二操作的控制指令，向所述目标设备发送所述控制指令，所述控制指令用于控制所述目标设备。
第二方面，本申请实施例提供了一种终端设备，包括：收发器、处理器和存储器；所述收发器用于接收和发送数据；所述存储器存储有一个或多个程序，所述一个或多个程序包括指令，当所述指令被所述处理器执行时，使得所述电子设备执行如第一方面各个可能实现的所述的方法。
第三方面，本申请实施例提供了一种远程协助方法，所述方法在第二终端与第一终端进行视频通话过程中，由所述第二终端执行，所述方法包括：接收目标设备的设备信息，所述目标设备为显示在当前视频通话界面中的设备；根据接收的所述设备信息虚拟出操作界面，并在当前视频通话界面上显示所述操作界面；所述操作界面为所述目标设备的控制面板或遥控器的操作界面；接收用户对所述当前视频通话界面上的操作界面的至少一个第三操作；显示对所述操作界面的一个或多个操作对应的视觉效果，并向所述第一终端同步所述视觉效果。
本申请实施例中，在第二终端上呈现包含虚拟操作界面的视频通话界面，第二终端对应的用户通过在第二终端上对该虚拟操作界面进行操作，使得第一终端同步呈现用户对虚拟操作界面的操作，以实现协助第一终端对应的用户对目标设备的调控。
在另一个可能的实现中，所述方法还包括：将所述操作界面与所述目标设备进行绑定。
在另一个可能的实现中，所述在当前视频通话界面上显示所述操作界面，包括：将 所述操作界面显示在当前视频通话界面中所述目标设备上或所述目标设备的四周，其中，所述操作界面为所述目标设备的遥控器的控制界面；或者将所述操作界面覆盖在当前视频通话界面中的所述目标设备的控制面板上或替换所述目标设备的控制面板，其中，所述操作界面为所述目标设备的控制面板的控制界面。
在另一个可能的实现中，所述方法还包括：根据来自所述第一终端的数据，在当前视频通话界面上，显示所述第一终端的用户在所述第一终端上对虚拟操作界面的一个或多个操作。
第四方面，本申请实施例提供了一种终端设备，包括：收发器、处理器和存储器；所述收发器用于接收和发送数据；所述存储器存储有一个或多个程序，所述一个或多个程序包括指令，当所述指令被所述处理器执行时，使得所述电子设备执行如第三方面各个可能实现的所述的方法。
第五方面，本申请实施例提供了一种远程协助系统，包括：需要进行控制的目标设备、云服务器、执行如第一方面各个可能实现的所述的方法的终端设备和执行如第三方面各个可能实现的所述的方法的终端设备。
第六方面，本申请实施例提供了一种可读存储介质，用于存储指令，当所述指令被执行时，使得执行如第一方面各个可能实现的所述的方法和执行如第三方面各个可能实现的所述的方法的被实现。
第七方面，本申请实施例提供了一种包含指令的计算机程序设备，当其在终端上运行时，使得所述终端执行如第一方面各个可能实现的所述的方法和执行如第三方面各个可能实现的所述的方法。
附图说明
下面对实施例或现有技术描述中所需使用的附图作简单地介绍。
图1为本申请实施例提供的一种远程协助系统的结构示意图；
图2为本申请实施例提供的一种终端上显示的虚拟操作界面的示意图；
图3为本申请实施例提供的一种终端上显示的虚拟操作界面的示意图；
图4为本申请实施例提供的一种远程协助方法的流程图；
图5为本申请实施例提供的另一种远程协助方法的流程图；
图6为本申请实施例提供的第一终端和第二终端之间通信过程的流程示意图；
图7为本申请实施例提供的一种远程协助系统的各个设备之间工作流程示意图；
图8为本发明实施例提供的一种终端设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述。
图1为本申请实施例提供的一种远程协助系统的结构示意图。如图1所示，该系统包括：目标设备10、终端20、终端30和云服务器40。
本申请实施例中的目标设备10为需要进行控制操作的设备(目标设备)，包括但不仅限空调、电视、手机、电脑等电子设备、机械设备、车辆以及其它类型的设备；或者是以上设备的遥控器。本申请实施例将以空调为例进行讲述一下技术方案。
本申请实施例中的终端20和终端30包括但不仅限手机、平板、笔记本电脑等电子设备。
本申请实施例中，在终端20与终端30之间进行视频通话过程中，终端20通过摄像头对目标设备10进行拍摄，获取目标设备10的图像，以生成包括有目标设备10影像的视频流。终端20将获取的包括有目标设备10影像的视频流发送给终端30，同时也将视频流发送给云服务器40，云服务器40实时识别视频流中的目标设备10，并根据识别出的目标设备10，匹配出目标设备10对应的数据模型和可控制配置文件数据，然后将该目标设备10对应的数据模型和可控制配置文件数据发送给终端20和终端30；终端20和终端30根据接收到的数据模型，在视频中虚拟出一个操作界面，这个操作界面是对应于控制目标设备10的真实遥控设备或控制面板的影像，呈现在视频中的目标设备10的四周、正上方等位置上；并根据接收到的可控制配置文件，将虚拟出的操作界面上按键、旋钮等与其对应的控制指令相关联，使得用户在该操作界面上对按键、旋钮等控件操作时，终端20能确定每一个操作对应的控制指令。
在一个具体的例子中，如图1所示，终端20拍摄空调后，将包括空调的视频或者将空调的图像发送到服务器，服务器通过例如人工智能(artificial intelligence，AI)识别出该空调的品牌和型号,获取该品牌和型号的空调对应的遥控器的二维或三维图像模型，并获取该遥控器上各个按键对应的指令的可控制配置文件，例如“ON/OFF”按键对应的可控制配置文件为在按下“ON/OFF”按键后，执行关闭或开启空调的指令；然后将遥控器模型和可控制配置文件都发送给终端20，终端20接收到后，虚拟出一个对应的遥控器图像，并显示在当前视频界面上。该带有虚拟遥控器图像的视频界面可以经由音视频同步技术被同步到终端30上，当然也可以由服务器发将遥该控器模型和可控制配置文件给终端30，但前者在实现上更简单经济。在用户对该虚拟遥控器图像上的虚拟按键进行操作时，例如在终端30上操作虚拟按键，则终端30确定该虚拟按键所对应的指令(同一个按键不同操作可以对应不同指令,例如长按和短按对应不同指令)，并将该指令对应的信息，例如操作位置，操作类型，时间戳等发送给对端的终端20，同时在终端30当前的视频界面中虚拟遥控器图像上呈现该操作对应的视觉效果，例如按钮随着用户的按动而向下凹陷，随用户的抬手而恢复，例如触摸式按键按下后有对应的指示灯，LCD显示内容变化等，视觉效果可以包含在可控制配置文件中，由终端根据当前用户操作和可控制配置文件确定对应的视觉效果；该视觉效果也被同步到终端20上。这样终端20的用户能直观的看到如何操控遥控器；终端20还可以根据接收到的指令，发送控制目标设备的控制指令，实现对目标设备的远程控制。
需要说明的是，在终端20不具有控制目标设备10的功能(例如终端20不具有红外线发射功能、没有与目标设备相连接等)、用户不需要终端20直接控制目标设备10等情况时，服务器40可以不发送可控制配置文件，或者发送的可控制配置文件中不包含控制指令(直接对目标设备进行控制的指令)，用户可以根据终端20上显示的远程协助方在遥控器模型上的操作，学习到控制目标设备10的操作，可以直接在目标设备10上或实体遥控器上进行操作，来实现对目标设备10的控制。
当用户在终端20或终端30上的虚拟的操作界面进行操作时，终端20或终端30会记录操作过程中的操作信息，如坐标点、时间戳、次数、时长等信息，确定对应的视觉效果， 并呈现在当前视频通话界面上，同步到视频通话的对端。在一个可能的实施例中，当终端20对应的用户A不会操作空调时，终端30对应的用户B协助用户A控制空调，用户A通过音视频告诉用户B其希望：“将空调打开，并将温度调到21度(假设开机温度为17度)”，用户B在终端30中显示的虚拟遥控器操作界面的“ON/OFF”虚拟按键处进行操作，以将空调打开，此时终端30记录下该操作过程中滑动操作的坐标点、操作时间和点击次数等操作信息；然后用户B在终端30显示的虚拟遥控器中的温度虚拟按键“+”处进行操作，通过点击四次，将温度调到“21度”，此时终端30记录下该操作过程中滑动操作的坐标点、操作时间和点击次数等操作信息。在用户操作过程中，终端20根据接收到的输入，确定并呈现这些操作的视觉效果，例如虚拟遥控器的按键被按下，显示窗口显示对应的图标，数字等，这些视觉效果显示在当前正在进行的视频通话界面上，同步到视频通话的对端。终端20上呈现用户操作的视觉效果，以直观的方式教用户A如何在终端20进行操作，以对空调进行操作，将空调打开，并将温度调到21度。
图2为本申请实施例提供的一种终端上显示的虚拟操作界面的示意图。
在一个可能的实施例中，终端20与终端30正在用手机进行视频通话，终端20的用户将手机的摄像头对着空调，这样视频通话界面上显示空调的影像，如图2所示。终端20具有红外线、WI-FI、蓝牙、数据流量等传输功能或具有直接控制目标设备的功能。终端20接收到空调对应的数据模型(和可控制配置文件)后，虚拟出一个虚拟控制界面，该控制界面包括有一些如“ON/OFF”虚拟按键、温度虚拟按键、湿度虚拟按键等等；将该虚拟控制界面呈现在视频流中的空调附近。当终端30的用户B通过操作虚拟控制界面上的虚拟按键后，终端30在其界面上呈现这些操作的视觉效果，同步到视频通话的对端--终端20，将获取的这些操作信息对应的控制信息也发送给终端20。终端20的用户A通过观看即可直观的学习到在控制界面上如何进行操作。当用户A在终端20上的虚拟控制界面操作后，终端20将各个虚拟按键对应的控制指令通过如红外线、WI-FI、蓝牙、数据流量等传输方式发送给空调。
图3为本申请实施例提供的一种终端上显示的虚拟操作界面的示意图。
在一个可能的实施例中，如图3所示，终端20接收到空调对应的数据模型后，构建出与视频流中的空调一样大小的虚拟空调三维结构图，然后将虚拟空调三维结构图覆盖在视频流中的空调上，或替换掉视频流中的空调。这个虚拟空调也由音视频通话同步技术被同步到终端30上，当用户B在终端30上的虚拟空调三维结构图上的可操作的按键进行操作时，终端30将在当前的视频界面中虚拟空调三维结构图上操作对应的视觉效果实时同步到终端20上。终端20呈现该视觉效果，用户A通过观看这些还原的操作动作，能了解操作和控制空调的方法，然后直接在空调的控制面板上进行操作，实现对空调的控制。。
终端接收到用户对虚拟操作界面上的操作后，需要根据坐标确定操作的对象，根据时间，压力，方向，距离等信息，以及可控制配置文件确定显示效果。例如用户是按压触摸式按键或虚拟按键，则在按键上显示光标或阴影之类，表示用户选中，或者在按键上显示光圈、特殊符号之类的动画效果，表示用户长按、单击等对应的操作。操作信息和操作对象及显示效果是有一个对应关系的，需要预存在本地或云端，例如可以是可控制配置文件，也可以由终端在获取目标设备的品牌和型号后，从网络或其他设备处获得。本申请实施例提供的一种远程协助系统，终端20获取目标设备10对应的控制设备、控制 面板等控制界面的二维或三维数据，在和终端30的视频通话界面中呈现该数据对应的虚拟控制界面，该虚拟控制界面也同步到终端30，终端30接收协助方(用户B)在呈现在终端30上的虚拟控制界面上的操作，然后将该操作信息发送给终端20，终端20根据接收到的操作信息在视频界面上还原这些操作动作，或者终端30根据用户操作确定并呈现该用户操作，同样的同步到对端，这样终端20对应的用户A通过观看该操作动作，能了解操作和控制目标设备的方法，即可直接在终端20上的控制面板进行操作或在真实的遥控设备上进行操作，由于被协助方用户A能在自己的终端上看到在真实控制面板或者遥控器上的具体操作，非常直观，就像协助方在现场手把手教导一样，提高了双方交流的效率。
图4为本申请实施例提供的一种远程协助方法的流程图。如图4所示，本申请实施例提供了一种远程协助方法，该方法由被协助者所对应的终端20执行，在终端20与终端30之间进行视频通话过程中，所述方法实现步骤如下：
步骤S401，将包括目标设备的图像发送给服务器。
在步骤S401之前，终端20与终端30之间建立通信连接，用于进行视频、设备信息、操作信息等数据的传输。在本申请实施例中，终端20与终端30之间可以直接建立通信连接，如D2D通信技术；也可以通过视频服务器让终端20和终端30之间建立连接，如现有的微信视频通话过程中，让视频服务器将终端20的视频流发送到终端30上进行呈现，具体实现方式在此本申请不作限定。
其中，终端20与终端30之间的通信连接可以由终端20发起，在终端20对应的用户A对当前的目标设备10无法进行操作时，指定终端30对应的用户B协助自己对该目标设备10进行操作，通过向终端30发起请求，进行视频连接。
终端20通过与终端30之间建立通信连接后，在视频通话过程中，终端20的摄像头对目标设备10进行拍摄，获取目标设备10的图像，生成包括有目标设备10影像的视频流，实时发送给终端30。同时，终端20将该视频流发送给云服务器40。
其中，终端20发送给云服务器40的也可以为包括目标设备10的图片、短视频等等。在一个可能的实施例中，终端20对获取的视频流进行设置，例如在每隔几秒钟或几分钟截取N帧的图片，然后将截取的图片发送给云服务器40，云服务器40实时识别图片的目标设备10。相比较实时向云服务器40发送视频流的方式，减少终端20运行负担，降低流量。
步骤S403，接收服务器发送的目标设备对应的设备信息。
具体地，云服务器40接收到终端20发送的包括有目标设备10的视频流后，进行AI识别，识别出目标设备10的品牌、型号等信息后，确定目标设备10为什么类型的设备。在确定设备的类型后，再匹配出与该设备对应设备信息。其中，设备信息包括目标设备10的数据模型，还可以包括可控制配置文件数据。数据模型可以为如图1中的用于控制目标设备10的真实遥控器、如图2中的用于控制目标设备10的虚拟操作界面、如图3中的目标设备10的控制面板等等。可控制配置文件可以为“启动”、“关闭”、“调高”、“调低”、“频道”等控制指令，可控制配置文件中的各个控制指令与数据模型中的操作界面上按键、旋钮等相关联，这样当用户在该操作界面上对按键、旋钮等控件操作时，终端20能确定每一个操作对应的控制指令。例如“关闭”按键对应的可控制配置文件为在按下“关闭”按键后，执行关闭目标设备10的指令。
步骤S405，根据接收的设备信息虚拟出操作界面，并在当前视频通话界面上显示操作界面。
如果终端20不具有控制目标设备10的功能(例如终端20不具有红外线发射功能、没有与目标设备相连接等)、用户不需要终端20直接控制目标设备10等情况时，可以由用户选择接向服务器请求或接收的数据内容，例如通过提供菜单选项，语音提示等，云服务器40可以只发送目标设备10对应的数据模型。终端20在接收到数据模型后，在当前的视频界面中呈现真实的遥控设备、目标设备10上真实的控制面板、或与目标设备10对应的控制设备等实体物体的影像。
在第一个可能的实施例中，如图1中终端20所示(其实终端20与终端30显示的界面是相同的，可以参见终端30的显示界面)，终端20接收到空调的数据模型(和可控制配置文件数据)后，可以选择在终端20上虚拟出一个与该空调相匹配的空调遥控器(在视频界面上显示的控制界面可以由用户选择，或者是默认)。其中虚拟出的空调遥控器与真实空调遥控器完全相同，包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的“-”和“+”的虚拟按键等按键。终端20将显示的空调遥控器同步到终端30上，终端30对应的用户B可对显示在视频界面上的空调遥控器的按键进行操作。
即使终端20和终端30没有获取可控制配置文件，但在用户对视频界面中的空调遥控器的按键进行操作时，终端上仍呈现该操作对应的视觉效果，然后实时发送给对端的终端上，以便用户A能观看到操作效果，学习到控制目标设备10的操作，同时如果用户A在终端20的虚拟空调遥控器上操作，用户B也同样可以看到用户A的操作，这样可以对用户A进行实时指导和帮助。
在第二个可能的实施例中，如图2中终端上显示的虚拟操作界面所示，终端20接收到空调的数据模型和可控制配置文件，选择在终端20上虚拟出一个用于控制空调的虚拟操作界面。其中虚拟出的操作界面包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的向上和向下的虚拟按键等按键。终端20将显示的虚拟操作界面发送到终端30上，终端30对应的用户B可在虚拟操作界面中的按键上进行操作。
由于终端20接收到空调的可控制配置文件，如果用户A在视频界面上的虚拟操作界面上的按键进行操作时，终端20根据该按键对应的可控制配置文件，确定该操作对应的控制空调的指令，并通过如红外线、WI-FI、蓝牙、数据流量等传输方式发送给空调，以直接控制空调。
针对上述两个可能的实施例中，终端20需要将接收设备信息与视频流中的目标设备10进行绑定，使得终端20根据设备信息显示的虚拟操作界面与目标设备10相对应，以防止当视频流中的目标设备10发生变化后，终端20上仍沿用之前设备的虚拟操作界面。具体的，终端20不断或间断性向云服务器40发送目标设备10的图像，如果云服务器40识别出的设备图像没有变化，就不会更改已发送给终端20的目标设备10的设备信息；如果云服务器识别不到目标设备10的图像，就会发送通知信息给终端20，让终端20在视频上终止显示虚拟操作界面。
步骤S407，在当前视频通话界面上显示第二终端用户对虚拟操作界面的操作。
终端20接收到云服务器40发送的设备信息后，在视频流上呈现虚拟操作界面。同时将该设备信息发送给终端30(利用音视频同步算法同步给对端或者单独发送)，使终端30 上同步包含该虚拟操作界面的视频通话界面。这一步是为了保证终端20和终端30上显示的内容完全相同，这样终端30上也呈现虚拟操作界面以协助用户A对设备进行调控。
其中，云服务器40可以直接将设备信息下发到终端20和终端30，终端20就不需要对设备信息进行转发，以减少终端20的工作量。但终端20同步虚拟操作界面的方式实现更简单。
在终端30对应的用户B对虚拟操作界面进行操作后，终端30记录下该操作过程中滑动操作的坐标点、操作时间和点击次数、按压时长等操作信息并发送给终端20。同时，在终端30当前的视频通话界面中虚拟操作界面上呈现该操作对应的视觉效果，例如按钮随着用户的按动而向下凹陷，随用户的抬手而恢复，该视觉效果也被同步到终端20上，以便终端20的用户能直观的看到如何操控遥控器。具体的同步算法为现有技术，例如通过音视频同步算法，例如打时间戳的方式等实现同步，此处不再赘述。
在本申请实施例中，对目标设备10的控制方法有多种，如在一个可能的实施例中，终端20与目标设备建立智能家居互联协议。终端20接收到终端30发送的操作信息后，解析操作信息，然后通过智能家居互联协议，将操作信息转换成指令，通过如红外线、WI-FI、蓝牙、数据流量等传输方式发送给空调，这样就不需要终端20对应的用户A再次进行操作。终端30也可以根据用户的操作，确定用户的操作对应的控制指令，将控制指令发给终端20，终端20据此通过例如红外指令，或者蓝牙指令，或者Wi-Fi指令,或者点对点指令等发送给目标设备，实现对目标设备的控制。
在另一个可能的实施例中，如果终端20不具有红外线发射功能、没有与目标设备10相连接等情况下，终端30将在当前的视频界面中虚拟操作界面上呈现该操作对应的视觉效果，实时发送到终端20上，终端20对应的用户通过观看该操作对应的视觉效果，能了解操作和控制空调的方法，然后在实体遥控器、目标设备10上的控制面板、目标设备10对应的控制设备等实体物体上直接进行操作。
需要进行一步说明的是，用户A在终端20上对操作界面进行操作后，终端20将在当前的视频界面中虚拟操作界面上呈现该操作对应的视觉效果，实时发送到终端30上，终端30对应的用户B通过观看该操作对应的视觉效果，来确定终端20对应的用户A的操作是否正确，并可以及时给予指导，例如通过语音，和/或通过对虚拟操作界面的操作等。
同样，用户A在终端20上对操作界面进行操作后，需要将操作信息发送到终端30上。在一个可能的实施例中，用户A在终端20的屏幕上针对某个区域划了一个标注的圈，终端20要记录当时在屏幕上的一系列的坐标点、时间戳和次数、按压时长等操作信息，呈现对应的视觉效果。并同步到对端，例如通过时间戳同终端20发送的视频进行匹配，以保证双方在界面上看到的内容将是一致的。
本申请提供由终端20执行的远程协助方法，通过获取包括目标设备10的视频流和根据该视频流获取该设备的相关的设备信息后，通过AR技术，在终端20中的视频中呈现该虚拟控制界面，通过音视频同步技术，使得通信的各方终端的视频通话界面呈现该虚拟控制界面，任何一方终端的用户对该虚拟控制界面的操作，其视觉效果都会被同步到所有视频通话的各方，这样需要协助的用户，例如终端20对应的用户A，能通过观看这些操作动作，了解操作和控制目标设备的方法，即可直接在终端20上的控制面板进行操作或在真实的遥控设备上进行操作，由于被协助方用户A能在自己的终端上看到在真实控制面 板或者遥控器上的具体操作，非常直观，就像协助方在现场手把手教导一样，提高了双方交流的效率，降低了远程协助的难度。
图5为本申请实施例提供的另一种远程协助方法的流程图。如图5所示，本申请实施例提供了另一种远程协助方法，该方法由被协助者所对应的终端30执行，在终端20与终端30之间进行视频通话过程中，所述方法实现步骤如下：
步骤S501，接收目标设备的设备信息。
在步骤S501之前，终端20与终端30之间建立通信连接。其中，终端20与终端30之间的通信连接可以由终端30发起，在终端30对应的用户B得知终端20对应的用户A对某个设备无法进行操作时，通过向终端20发起请求，进行视频连接。当然也可以由终端20发起。
在一个可能的实施例中，终端30获取的设备信息可以不通过终端20发送。在云服务器40识别目标设备10并获取该设备对应的设备信息后，直接将设备信息下发到终端20和终端30，终端20就不需要对设备信息进行转发，以减少终端20的工作量。
步骤S503，在当前视频通话界面上显示虚拟操作界面。
终端30可以根据接收的设备信息虚拟出操作界面，如果终端30接收到设备信息中只有目标设备10对应的数据模型。终端30在接收到数据模型后，在当前的视频界面中呈现虚拟的操作界面，该操作界面可以对应真实的遥控设备、目标设备10上真实的控制面板、或与目标设备10对应的控制设备等实体物体的影像。
在第一个可能的实施例中，如图1中终端30所示，终端30接收到空调的数据模型后，在终端30上虚拟出一个与该空调相匹配的空调遥控器。其中虚拟出的空调遥控器与真实空调遥控器完全相同，包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的“-”和“+”的虚拟按键等按键。终端20将显示的空调遥控器发送到终端30上，终端30对应的用户B可在视频界面上的空调遥控器中的按键上进行操作。
也可以通过音视频同步技术，将终端20上显示的虚拟操作界面同步到终端30上。
用户B对视频界面中的虚拟空调遥控器的按键进行操作时，终端30上呈现该操作对应的视觉效果，并实时发送给终端20，以便用户A能观看到操作效果，学习到控制目标设备10的操作。
需要说明的是，至于终端30上虚拟出是空调遥控器还是空调的控制面板等其它控制设备，需要根据终端20对应的用户A选择来确定。如果用户A选择的呈现是空调遥控器，则终端30上呈现的也是空调遥控器，这样保证终端20和终端30上呈现的控制设备图像的操作界面是相同的。所以这种实现的方式，是终端20向终端30发送设备信息的时候，对设备信息中的数据模型进行了选择，使终端30上只能呈现与终端20上显示的控制设备的图像的操作界面。
如果终端30接收到空调的可控制配置文件，则根据用户的操作，生成对应的控制指令发送给终端20，终端20可以通过如红外线、WI-FI、蓝牙、数据流量等传输方式将控制指令发送给空调，以直接控制空调。
针对上述两个可能的实施例中，终端30需要将接收设备信息与视频流中对应的真实设备进行绑定，使得终端30根据设备信息显示的虚拟操作界面与设备相对应，以防止当视频流中的设备发生变化后，终端30上仍沿用之前设备的虚拟操作界面。具体的在上面 已经说明，在此不再赘述。
步骤S505，在当前视频通话界面上显示用户对虚拟操作界面的操作。
终端30对应的用户B对虚拟操作界面进行操作，终端30记录下该操作过程中滑动操作的坐标点、操作时间和点击次数、按压时长等操作信息并发送给终端20。同时，在终端30当前的视频界面中虚拟操作界面上呈现该操作对应的视觉效果，例如按钮随着用户的按动而向下凹陷，随用户的抬手而恢复，该视觉效果也被同步到终端20上，以便终端20的用户能直观的同步看到对遥控器的操作。
用户B在终端30上对虚拟操作界面的操作也可以通过其他方式或算法发送到终端20上，以保证双方在界面上看到的内容将是一致的。
本申请提供由终端30执行的远程协助方法，在终端30上呈现包含虚拟操作界面的视频通话界面，终端30对应的用户B通过在终端30上对该虚拟操作界面进行操作，使得终端20同步呈现用户B对虚拟操作界面的操作，以实现协助终端20对应的用户A对目标设备10的调控。
图6为本申请实施例提供的第一终端和第二终端之间通信过程的流程示意图。如图6所示，第一终端和第二终端之间通信过程具体如下：
步骤S601，终端20与终端30之间建立通信连接。
建立通信连接的目的是用于进行视频、设备信息、操作信息等数据的传输。
在一个可能的实施例中，终端20与终端30之间的通信连接可以由终端20发起，在终端20对应的用户A对当前的目标设备10无法进行操作时，指定终端30对应的用户B协助自己对该目标设备10进行操作，通过向终端30发起请求，进行视频连接。
在一个可能的实施例中，终端20与终端30之间的通信连接可以由终端30发起，在终端30对应的用户B得知终端20对应的用户A对某个设备无法进行操作时，通过向终端20发起请求，进行视频连接。
本申请实施例中终端20与终端30之间可以直接建立通信连接，如D2D通信技术；也可以通过视频服务器让终端20和终端30之间建立连接，如现有的微信视频通话过程中，让视频服务器将终端20的视频流发送到终端30上进行呈现，具体实现方式在此本申请不作限定。
步骤S602，启动终端20的摄像头获取包括目标设备10的视频流。
在终端20与终端30之间建立通信连接后，启动终端20的摄像头对目标设备10进行拍摄，获取目标设备10的图像，生成包括有目标设备10影像的视频流。
步骤S603，终端20将获取的视频流实时发送到终端30。
本申请实施例在终端20和终端30之间建立连接后，可以将终端20的视频流实时发送到终端30上进行呈现，从而实现终端0和终端30之间显示的内容相同。
步骤S604，终端20将获取的视频流实时发送到云服务器40。
其中，终端20发送给云服务器40的还可以为包括目标设备10的图片、短视频等等。在一个可能的实施例中，终端20对获取的视频流进行设置，例如在每隔几秒钟或几分钟截取N帧的图片，然后将截取的图片发送给云服务器40，云服务器40实时识别图片的目标设备10。相比较实时向云服务器40发送视频流的方式，减少终端20运行负担，降低流量。
步骤S605，云服务器40根据接收到的包括目标设备10的图像后，识别出图像中的目标设备10，并根据目标设备10匹配出该设备对应的设备信息。
具体地，云服务器40接收到终端20发送的包括有目标设备10的视频流后，进行AI识别，识别出目标设备10的品牌、型号等信息后，确定目标设备10为什么类型的设备。在确定设备的类型后，再匹配出与该设备对应设备信息。其中，设备信息包括目标设备10的数据模型，还可以包括可控制配置文件数据。数据模型可以为如图1中的用于控制目标设备10的真实遥控器、如图2中的用于控制目标设备10的虚拟操作界面、如图3中的目标设备10的控制面板等等。可控制配置文件可以为“启动”、“关闭”、“调高”、“调低”、“频道”等控制指令，将可控制配置文件中的各个控制指令与数据模型中的操作界面上按键、旋钮等相关联，使得用户在该操作界面上对按键、旋钮等控件操作时，终端20能确定每一个操作对应的控制指令。例如“关闭”按键对应的可控制配置文件为在按下“关闭”按键后，执行关闭目标设备10的指令。
步骤S606，云服务器40将设备信息发送给终端20。
如果终端20不具有控制目标设备10的功能(例如终端20不具有红外线发射功能、没有与目标设备相连接等)、用户不需要终端20直接控制目标设备10等情况时，云服务器40可以只发送目标设备10对应的数据模型。如果终端20在终端20具有红外线、WI-FI、蓝牙、数据流量等传输功能或具有直接控制目标设备的功能时，云服务器40发送目标设备10对应的数据模型和可控制配置文件。
步骤S607，终端20根据设备信息，在视频流上虚拟出操作界面。
终端20在接收到数据模型后，在当前的视频界面中呈现真实的遥控设备、目标设备10上真实的控制面板、与目标设备10对应的控制设备等实体物体的影像。在第一个可能的实施例中，如图1中终端20所示(其实终端20与终端30显示的界面是相同的，可以参见终端30的显示界面)，终端20接收到空调的数据模型后，选择在终端20上虚拟出一个与该空调相匹配的空调遥控器(在视频界面上显示的控制界面可以由用户选择，或者是默认)。其中虚拟出的空调遥控器与真实空调遥控器完全相同，包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的“-”和“+”的虚拟按键等按键。终端20将显示的空调遥控器发送到终端30上，终端30对应的用户B可在视频界面上的空调遥控器中的按键上进行操作。
即使终端20和终端30没有可控制配置文件，但是用户对视频界面中的空调遥控器的按键进行操作时，终端上仍呈现该操作对应的视觉效果，然后实时发送给对端的终端上，以便用户A能观看到操作效果，学习到控制目标设备10的操作，同时用户B可以看到用户A的操作，这样可以对用户A进行实时指导和帮助。
终端20在接收到数据模型和可控制配置文件后，在当前的视频界面中不仅可以呈现真实的遥控设备、目标设备10上真实的控制面板、与目标设备10对应的真实控制设备等实体物体的影像，还可以呈现如图2所示的虚拟操作界面。在第二个可能的实施例中，如图2中终端上显示的虚拟操作界面所示，终端20接收到空调的数据模型和可控制配置文件，选择在终端20上虚拟出一个用于控制空调的虚拟操作界面。其中虚拟出的操作界面包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的向上和向下的虚拟按键等按键。终端20将显示的虚拟操作界面发送到终端30上，终端30对应的用户B 可在虚拟操作界面中的按键上进行操作。
由于终端20接收到空调的可控制配置文件，如果用户A在视频界面上的虚拟操作界面上的按键进行操作时，终端20根据该按键对应的可控制配置文件控制空调的指令，通过如红外线、WI-FI、蓝牙、数据流量等传输方式发送给空调，以直接控制空调。
针对上述两个可能的实施例中，终端20需要将接收设备信息与视频流中的目标设备10进行绑定，使得终端20根据设备信息显示的虚拟操作界面与目标设备10相对应，以防止当视频流中的目标设备10发生变化后，终端20上仍沿用之前设备的虚拟操作界面。具体的，终端20不断或间断性向云服务器40发送目标设备10的图像，如果云服务器40识别出的设备图像没有变化，就不会更改已发送给终端20的目标设备10的设备信息；如果云服务器识别不到目标设备10的图像，就会发送通知信息给终端20，让终端20在视频上终止显示虚拟操作界面。
步骤S608，终端20将设备信息发送给终端30。
在一个可能的实施例中，终端30获取的设备信息可以不通过终端20发送。在云服务器40识别该设备并获取该设备对应的设备信息后，直接将设备信息下发到终端20和终端30，终端20就不需要对设备信息进行转发，以减少终端20的工作量。
步骤S609，终端30根据设备信息，在视频流上虚拟出操作界面。
如果终端30接收到设备信息中只有目标设备10对应的数据模型。终端30在接收到数据模型后，在当前的视频界面中呈现真实的遥控设备、目标设备10上真实的控制面板、与目标设备10对应的控制设备等实体物体的影像。
在第一个可能的实施例中，如图1中终端30所示，终端30接收到空调的数据模型后，根据终端20的选择，在终端30上虚拟出一个与该空调相匹配的空调遥控器。其中虚拟出的空调遥控器与真实空调遥控器完全相同，包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的“-”和“+”的虚拟按键等按键。终端20将显示的空调遥控器发送到终端30上，终端30对应的用户B可在视频界面上的空调遥控器中的按键上进行操作。
用户B对视频界面中的空调遥控器的按键进行操作时，终端30上呈现该操作对应的视觉效果，实时发送给终端20，以便用户A能观看到操作效果，学习到控制目标设备10的操作。
需要说明的是，至于终端30上虚拟出是空调遥控器还是空调的控制面板等其它控制设备，需要根据终端20对应的用户A选择来确定。如果用户A选择的呈现是空调遥控器，则终端30上呈现的也是空调遥控器，这样保证终端20和终端30上呈现的控制设备图像的操作界面是相同的。所以这种实现的方式，是终端20向终端30发送设备信息的时候，对设备信息中的数据模型进行了选择，使终端30上只能呈现与终端20上显示的控制设备的图像的操作界面。
如果终端30接收到设备信息中有目标设备10对应的数据模型和可控制配置文件。终端30在接收到数据模型和可控制配置文件后，在当前的视频界面中不仅可以呈现真实的遥控设备、目标设备10上真实的控制面板、与目标设备10对应的真实控制设备等实体物体的影像，还可以呈现如图2所示的虚拟操作界面。
在第二个可能的实施例中，如图2中终端上显示的虚拟操作界面所示，终端30接收到 空调的数据模型和可控制配置文件后，根据终端20的选择，在终端30上虚拟出一个用于控制空调的虚拟操作界面。其中虚拟出的操作界面包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的向上和向下的虚拟按键等按键。终端20将显示的虚拟操作界面发送到终端30上，终端30对应的用户B可在虚拟操作界面中的按键上进行操作。
由于终端30接收到空调的可控制配置文件，如果用户B在视频界面上的虚拟操作界面上的按键进行操作时，生成的操作信息发送给终端20后，终端20根据该按键对应的可控制配置文件中控制空调的指令，通过如红外线、WI-FI、蓝牙、数据流量等传输方式发送给空调，以直接控制空调。
针对上述两个可能的实施例中，终端30需要将接收设备信息与视频流中的真实设备进行绑定，使得终端30根据设备信息显示的虚拟操作界面与设备相对应，以防止当视频流中的设备发生变化后，终端30上仍沿用之前设备的虚拟操作界面。
步骤S610，终端30记录用户在虚拟操作界面上进行操作的操作信息。
终端30对应的用户B对虚拟操作界面进行操作，终端30记录下该操作过程中滑动操作的坐标点、操作时间、点击次数、按压时长等操作信息。在确定用户B对此次控制过程中操作完成后，终端30将记录的这些操作信息发送给终端20。同时，在终端30当前的视频界面中虚拟操作界面上呈现该操作对应的视觉效果，例如按钮随着用户的按动而向下凹陷，随用户的抬手而恢复，该视觉效果也被同步到终端20上，以便终端20的用户能直观的看到如何操控遥控器。
需要进行一步说明的是，用户B在终端30上对视频流进行操作后，需要将操作信息发送到终端20上，以保证双方在界面上看到的内容将是一致的。
步骤S611，终端30向终端20发送操作信息和/或视频。
步骤S612，终端20解析操作信息，将解析的结果转换成对操作界面进行操作的操作动作。
终端20接收到这些操作信息后，解析出这些操作信息后，通过视频演示等方式，告诉用户A如何在终端20进行操作，以实现对目标设备10进行控制。同时，在终端30当前的视频界面中虚拟操作界面上呈现该操作对应的视觉效果，例如按钮随着用户的按动而向下凹陷，随用户的抬手而恢复，该视觉效果也被同步到终端20上，以便终端20的用户能直观的看到如何操控遥控器。
在本申请实施例中，对目标设备10的控制方法有多种，如在一个可能的实施例中，终端20与目标设备建立智能家居互联协议。终端20接收到终端30发送的操作信息后，解析操作信息，然后通过智能家居互联协议，将操作信息转换成指令，通过如红外线、WI-FI、蓝牙、数据流量等传输方式发送给空调，这样就不需要终端20对应的用户A再次进行操作。
在另一个可能的实施例中，如果终端20不具有红外线发射功能、没有与目标设备10相连接等情况下，终端30将在当前的视频界面中虚拟操作界面上呈现该操作对应的视觉效果，实时发送到终端20上，终端20对应的用户通过观看该操作对应的视觉效果，能了解操作和控制空调的方法，然后在实体遥控器、目标设备10上的控制面板、目标设备10对应的控制设备等实体物体上直接进行操作。
步骤S613，终端20向终端30发送操作信息和/或视频。
步骤S614，终端30呈现终端20上的操作界面。
用户A在终端20上对视频流进行操作后，终端20将在当前的视频界面中虚拟操作界面上呈现该操作对应的视觉效果，实时发送到终端30上，终端30对应的用户B通过观看该操作对应的视觉效果，来确定终端20对应的用户A的操作是否正确。
同样，用户A在终端20上对操作界面进行操作后，需要将操作信息发送到终端30上。在一个可能的实施例中，用户A在终端20的屏幕上针对某个区域划了一个标注的圈，终端20要记录当时在屏幕上的一系列的坐标点、时间戳和次数等操作信息。然后通过专有的通信频道将操作信息发送到终端30上，并通过时间戳同终端20发送的视频进行匹配，以保证双方在界面上看到的内容将是一致的。
本申请实施例提供的一种远程协助系统，用来实现以上各方法实施例，具体的，终端20获取包括目标设备10的视频流后，将该视频流发送给终端30和服务器40，服务器40获取目标设备10对应的控制设备、控制面板等真实控制界面的二维或三维数据，然后发送给终端20和终端30，并通过AR技术，在终端20和终端30中的视频中呈现该虚拟控制界面，终端30接收协助方(用户B)在该虚拟控制界面上的操作，然后将该操作信息发送给终端20，终端20根据接收到的操作信息在视频界面上还原这些操作动作，终端20对应的用户A通过观看这些还原的操作动作，能了解操作和控制目标设备的方法，即可直接在终端20上的控制面板进行操作或在真实的遥控设备上进行操作，由于被协助方用户A能在自己的终端上看到在真实控制面板或者遥控器上的具体操作，非常直观，就像协助方在现场手把手教导一样，提高了双方交流的效率。
针对上述“第一个可能实现的实施例”的技术方案，本申请列举一个实施例以助于理解：假设身处家中的老人为终端20对应的用户A，身处办公室的儿子为终端30对应的用户B，两者的智能手机均安装了“AR远程协助”APP。当老人A需要开启空调的睡眠模式，但找不到遥控器或者不会操作遥控器时，老人A打开手机APP“AR远程协助”，给儿子B发送求助；老人A将手机(的摄像头)对准空调，儿子B的手机界面出现空调的图像，并在界面上弹出此款空调的遥控器；儿子B在遥控器界面上操作相应按钮，此命令回传给老人A的手机，在老人A看着自己手机上儿子对遥控器操作的同时，老人A的手机在接收到空调控制命令后通过智能家居互联协议开启空调的睡眠模式。
针对上述“第二个可能实现的实施例”的技术方案，本申请列举一个实施例以助于理解：假设身处家中的老人为终端20对应的用户A，身处办公室的儿子为终端30对应的用户B，两者的智能手机均安装了“AR远程协助”APP。当老人A需要开启空调的睡眠模式，空调的遥控器在老人A的手上，老人A打开手机APP“AR远程协助”，给儿子B发送求助；老人A将手机对准遥控器，儿子B的手机界面出现遥控器的图像；儿子B在遥控器界面上操作相应按钮，此命令回传给老人A的手机，在老人A看着自己手机上儿子对遥控器操作的同时，老人A的手机在接收到空调控制命令后通过智能家居互联协议开启空调的睡眠模式。
本申请实施例还提供了一种远程遥控系统，用来实现以上各方法实施例，具体的，包括需要进行控制的目标设备10、云服务器40、由终端20执行的远程遥控装置70和由终 端30执行的远程遥控装置80。所述系统通过AR技术，能够在终端20获取的视频流上叠加虚拟操作界面，有助于协助方和被协助方进行交流，更容易让协助方协助被协助方对设备进行操作。
图7为本申请实施例提供的一种远程协助系统的各个设备之间工作流程示意图。如图7所示，该系统实现远程协助过程主要分为通信建立阶段、AI识别阶段、AR识别阶段和智能家居互联阶段。其中，A方表示终端20，B方表示终端30，云端表示云服务器40和物品表示目标设备10。
在建立通信阶段：
第一步：A方与B方之间建立通信连接。
建立通信连接的目的是用于进行视频、设备信息、操作信息等数据的传输。
在一个可能的实施例中，终端20与终端30之间的通信连接可以由终端20发起，在终端20对应的用户A对当前的目标设备10无法进行操作时，指定终端30对应的用户B协助自己对该目标设备10进行操作，通过向终端30发起请求，进行视频连接。
在一个可能的实施例中，终端20与终端30之间的通信连接可以由终端30发起，在终端30对应的用户B得知终端20对应的用户A对某个设备无法进行操作时，通过向终端20发起请求，进行视频连接。
本申请实施例中终端20与终端30之间可以直接建立通信连接，如D2D通信技术；也可以通过视频服务器让终端20和终端30之间建立连接，如现有的微信视频通话过程中，让视频服务器将终端20的视频流发送到终端30上进行呈现，具体实现方式在此本申请不作限定。
第二步：A方通过摄像头获取需要控制的物品。
在终端20与终端30之间建立通信连接后，启动终端20的摄像头对目标设备10进行拍摄，获取目标设备10的二维或三维结构图，生成包括有目标设备10的二维或三维结构图的视频流。
第三步：B方在于A方进行视频通话过程中，实时获取A方拍摄的视频内容。
本申请实施例在终端20和终端30之间建立连接后，可以将终端20的视频流实时发送到终端30上进行呈现，从而实现终端0和终端30之间显示的内容相同。
在AI识别阶段：
第一步：云端实时识别视频中的物品。
终端20发送给云服务器40的还可以为包括目标设备10的图片、短视频等等。在一个可能的实施例中，终端20对获取的视频流进行设置，例如在每隔几秒钟或几分钟截取N帧的图片，然后将截取的图片发送给云服务器40，云服务器40实时识别图片的目标设备10。相比较实时向云服务器40发送视频流的方式，减少终端20运行负担，降低流量。
第二步：在识别出物品的信息后，匹配出物品对应的操作信息，该操作信息包括开、关、调高、调低、频道等信息。
终端20发送的包括有目标设备10的视频流后，进行AI识别，识别出目标设备10的品牌、型号等信息后，确定目标设备10为什么类型的设备。在确定设备的类型后，再匹配出与该设备对应设备信息。其中，设备信息包括目标设备10的数据模型，还可以包括可控制配置文件数据。数据模型可以为如图1中的用于控制目标设备10的真实遥控器、如图 2中的用于控制目标设备10的虚拟操作界面、如图3中的目标设备10的控制面板等等。可控制配置文件可以为“启动”、“关闭”、“调高”、“调低”、“频道”等控制指令，将可控制配置文件中的各个控制指令与数据模型中的操作界面上按键、旋钮等相关联，使得用户在该操作界面上对按键、旋钮等控件操作时，终端20能确定每一个操作对应的控制指令。例如“关闭”按键对应的可控制配置文件为在按下“关闭”按键后，执行关闭目标设备10的指令。
在AR识别阶段：
第一步：A方从云端获取物品的物品信息，然后实现虚拟的操作面板，并将该操作面板与视频中的物品绑定。
一种情况下，如果终端20不具有控制目标设备10的功能(例如终端20不具有红外线发射功能、没有与目标设备相连接等)、用户不需要终端20直接控制目标设备10等情况时，云服务器40可以只发送目标设备10对应的数据模型。终端20在接收到数据模型后，在当前的视频界面中呈现真实的遥控设备、目标设备10上真实的控制面板、与目标设备10对应的控制设备等实体物体的影像。在第一个可能的实施例中，如图1中终端20所示(其实终端20与终端30显示的界面是相同的，可以参见终端30的显示界面)，终端20接收到空调的数据模型后，选择在终端20上虚拟出一个与该空调相匹配的空调遥控器(在视频界面上显示的控制界面可以由用户选择，或者是默认)。其中虚拟出的空调遥控器与真实空调遥控器完全相同，包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的“-”和“+”的虚拟按键等按键。终端20将显示的空调遥控器发送到终端30上，终端30对应的用户B可在视频界面上的空调遥控器中的按键上进行操作。
即使终端20和终端30没有可控制配置文件，但是用户对视频界面中的空调遥控器的按键进行操作时，终端上仍呈现该操作对应的视觉效果，然后实时发送给对端的终端上，以便用户A能观看到操作效果，学习到控制目标设备10的操作，同时用户B可以看到用户A的操作，这样可以对用户A进行实时指导和帮助。
另一种情况下，如果终端20在终端20具有红外线、WI-FI、蓝牙、数据流量等传输功能或具有直接控制目标设备的功能时，云服务器40发送目标设备10对应的数据模型和可控制配置文件。终端20在接收到数据模型和可控制配置文件后，在当前的视频界面中不仅可以呈现真实的遥控设备、目标设备10上真实的控制面板、与目标设备10对应的真实控制设备等实体物体的影像，还可以呈现如图2所示的虚拟操作界面。在第二个可能的实施例中，如图2中终端上显示的虚拟操作界面所示，终端20接收到空调的数据模型和可控制配置文件，选择在终端20上虚拟出一个用于控制空调的虚拟操作界面。其中虚拟出的操作界面包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的向上和向下的虚拟按键等按键。终端20将显示的虚拟操作界面发送到终端30上，终端30对应的用户B可在虚拟操作界面中的按键上进行操作。
由于终端20接收到空调的可控制配置文件，如果用户A在视频界面上的虚拟操作界面上的按键进行操作时，终端20根据该按键对应的可控制配置文件控制空调的指令，通过如红外线、WI-FI、蓝牙、数据流量等传输方式发送给空调，以直接控制空调。
第二步：B方获取A方发送的物品信息，也在视频上虚拟的操作面板，以实现获取与A方显示界面显示相同的内容。
终端30获取的设备信息可以不通过终端20发送。在云服务器40识别该设备并获取该设备对应的设备信息后，直接将设备信息下发到终端20和终端30，终端20就不需要对设备信息进行转发，以减少终端20的工作量。
一种情况下，终端30接收到空调的数据模型后，根据终端20的选择，在终端30上虚拟出一个与该空调相匹配的空调遥控器。其中虚拟出的空调遥控器与真实空调遥控器完全相同，包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的“-”和“+”的虚拟按键等按键。终端20将显示的空调遥控器发送到终端30上，终端30对应的用户B可在视频界面上的空调遥控器中的按键上进行操作。
此时，用户B在视频界面中的空调遥控器的按键进行操作时，终端30上仍呈现该操作对应的视觉效果，然后实时发送给终端30，以便用户A能观看到操作效果，学习到控制目标设备10的操作。
另一种情况下，终端30接收到空调的数据模型和可控制配置文件后，根据终端20的选择，在终端30上虚拟出一个用于控制空调的虚拟操作界面。其中虚拟出的操作界面包括用于控制空调的启动或关闭的“ON/OFF”的虚拟按键、控制空调的温度的向上和向下的虚拟按键等按键。终端20将显示的虚拟操作界面发送到终端30上，终端30对应的用户B可在虚拟操作界面中的按键上进行操作。
由于终端30接收到空调的可控制配置文件，如果用户B在视频界面上的虚拟操作界面上的按键进行操作时，生成的操作信息发送给终端20后，终端20根据该按键对应的可控制配置文件中控制空调的指令，通过如红外线、WI-FI、蓝牙、数据流量等传输方式发送给空调，以直接控制空调。
第三步：B方在虚拟的操作面板进行远程操作，并记录操作信息，例如坐标信息、时间戳、点击次数等信息。
终端30对应的用户B对虚拟操作界面进行操作，终端30记录下该操作过程中滑动操作的坐标点、操作时间、点击次数、按压时长等操作信息。在确定用户B对此次控制过程中操作完成后，终端30将记录的这些操作信息发送给终端20。同时，在终端30当前的视频界面中虚拟操作界面上呈现该操作对应的视觉效果，例如按钮随着用户的按动而向下凹陷，随用户的抬手而恢复，该视觉效果也被同步到终端20上，以便终端20的用户能直观的看到如何操控遥控器。
第四步：A方获取B方的操作信息后，解析出操作信息为具体的按键操作，例如为开、关、调高、调低、频道等按键操作。
终端20接收到终端发送的操作信息后，解析出这些操作信息后，通过视频演示等方式，告诉用户A如何在终端20进行操作，以实现对目标设备10进行控制。同时，在终端30当前的视频界面中虚拟操作界面上呈现该操作对应的视觉效果，例如按钮随着用户的按动而向下凹陷，随用户的抬手而恢复，该视觉效果也被同步到终端20上，以便终端20的用户能直观的看到如何操控遥控器。
在智能家居互联阶段：
第一步：A方根据按键操作对对应的物品进行控制。
在一种情况下，终端20与目标设备建立智能家居互联协议。终端20接收到终端30发送的操作信息后，解析操作信息，然后通过智能家居互联协议，将操作信息转换成指令， 通过如红外线、WI-FI、蓝牙、数据流量等传输方式发送给空调，这样就不需要终端20对应的用户A再次进行操作。
在另一种情况下，如果终端20不具有红外线发射功能、没有与目标设备10相连接等情况下，终端30将在当前的视频界面中虚拟操作界面上呈现该操作对应的视觉效果，实时发送到终端20上，终端20对应的用户通过观看该操作对应的视觉效果，能了解操作和控制空调的方法，然后在实体遥控器、目标设备10上的控制面板、目标设备10对应的控制设备等实体物体上直接进行操作。
图8为本发明实施例提供的一种终端设备的结构示意图，用来实现以上各方法实施例。如图8所示的一种终端目标设备800，该终端目标设备800可以为上述的终端20和终端30，包括传感器801，显示器802，处理器803、存储器804、通信接口805以及总线806。终端设备中的处理器803、存储器804和通信接口805可以通过总线806建立通信连接。
传感器801，用于获取包括设备的视频流和设备对应的设备信息。传感器801可包括摄像头等。
显示器802，用于显示处理后的数据，如视频、虚拟操作界面。
处理器803可以为中央处理器(central processing unit，CPU)。
存储器804可以包括易失性存储器(volatile memory)，例如随机存取存储器(random-access memory，RAM)；存储器也可以包括非易失性存储器(non-volatile memory)，例如只读存储器(read-only memory，ROM)，快闪存储器，硬盘(hard disk drive，HDD)或固态硬盘(solid state drive，SSD)；存储器804还可以包括上述种类的存储器的组合。
上述实施例提供的远程遥控方法，均由处理器803来执行。视频流、设备信息、操作信息等数据将存储在存储器804中。另外，存储器804中还将用于存储处理器803执行的用于实现上述实施例所述的远程遥控方法对应的程序指令等等。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请实施例的范围。
此外，本申请实施例的各个方面或特征可以实现成方法、装置或使用标准编程和/或工程技术的制品。本申请中使用的术语“制品”涵盖可从任何计算机可读器件、载体或介质访问的计算机程序。例如，计算机可读介质可以包括，但不限于:磁存储器件(例如，硬盘、软盘或磁带等)，光盘(例如，压缩盘(compact disc，CD)、数字通用盘(digital versatile disc，DVD)等)，智能卡和闪存器件(例如，可擦写可编程只读存储器(erasable programmable read-only memory，EPROM)、卡、棒或钥匙驱动器等)。另外，本文描述的各种存储介质可代表用于存储信息的一个或多个设备和/或其它机器可读介质。术语“机器可读介质”可包括但不限于，无线信道和能够存储、包含和/或承载指令和/或数据的各种其它介质。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序 产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘(solid state disk，SSD))等。
应当理解的是，在本申请实施例的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者接入网设备等)执行本申请实施例各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请实施例的具体实施方式，但本申请实施例的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请实施例揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请实施例的保护范围之内。

Claims (15)

  1. 一种远程协助的方法，其特征在于，所述方法在第一终端与第二终端进行视频通话过程中，由所述第一终端执行，所述方法包括：
    将包括目标设备的图像发送给服务器；
    接收所述服务器发送的所述目标设备对应的设备信息；
    根据接收的所述设备信息虚拟出操作界面，并在当前视频通话界面上显示所述操作界面；所述操作界面为所述目标设备的控制面板或遥控器的操作界面；
    根据来自所述第二终端的数据，在当前视频通话界面上，显示所述第二终端的用户在所述第二终端上对所述操作界面的一个或多个操作。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    将所述操作界面与所述目标设备进行绑定。
  3. 根据权利要求1或2所述的方法，其特征在于，所述在当前视频通话界面上显示所述操作界面，包括：
    将所述操作界面显示在当前视频通话界面中所述目标设备上或所述目标设备的附近，其中，所述操作界面为所述目标设备的遥控器的操作界面；或者
    将所述操作界面覆盖在当前视频通话界面中的所述目标设备的控制面板上或替换所述目标设备的控制面板，其中，所述操作界面为所述目标设备的控制面板的操作界面。
  4. 根据权利要求1-3中任一项所述的方法，其特征在于，所述方法还包括：
    接收所述第二终端发送的至少一个第一操作，向所述目标设备发送控制指令，所述控制指令用于控制所述目标设备。
  5. 根据权利要求1-3中任一项所述的方法，其特征在于，所述方法还包括：
    接收用户对所述操作界面的至少一个第二操作；
    确定所述至少一个第二操作对应的视觉效果，并在当前视频通话界面上显示所述视觉效果；
    向所述第二终端同步所述视觉效果。
  6. 根据权利要求5所述的方法，其特征在于，所述方法还包括：
    根据接收到的所述至少一个第二操作，确定对应所述的至少一个第二操作的控制指令，向所述目标设备发送所述控制指令，所述控制指令用于控制所述目标设备。
  7. 一种终端设备，其特征在于，包括：收发器、处理器和存储器；
    所述收发器用于接收和发送数据；
    所述存储器存储有一个或多个程序，所述一个或多个程序包括指令，当所述指令被所述处理器执行时，使得所述电子设备执行如权利要求1-6中的任意一项所述的方法。
  8. 一种远程协助方法，其特征在于，所述方法在第二终端与第一终端进行视频通话过程中，由所述第二终端执行，所述方法包括：
    接收目标设备的设备信息，所述目标设备为显示在当前视频通话界面中的设备；
    根据接收的所述设备信息虚拟出操作界面，并在当前视频通话界面上显示所述操作界面；所述操作界面为所述目标设备的控制面板或遥控器的操作界面；
    接收用户对所述当前视频通话界面上的操作界面的至少一个第三操作；
    显示对所述操作界面的一个或多个操作对应的视觉效果，并向所述第一终端同步所 述视觉效果。
  9. 根据权利要求8所述的方法，其特征在于，所述方法还包括：
    将所述操作界面与所述目标设备进行绑定。
  10. 根据权利要求8或9所述的方法，其特征在于，所述在当前视频通话界面上显示所述操作界面，包括：
    将所述操作界面显示在当前视频通话界面中所述目标设备上或所述目标设备的四周，其中，所述操作界面为所述目标设备的遥控器的控制界面；或者
    将所述操作界面覆盖在当前视频通话界面中的所述目标设备的控制面板上或替换所述目标设备的控制面板，其中，所述操作界面为所述目标设备的控制面板的控制界面。
  11. 根据权利要求8-10中任一项所述的方法，其特征在于，所述方法还包括：
    根据来自所述第一终端的数据，在当前视频通话界面上，显示所述第一终端的用户在所述第一终端上对虚拟操作界面的一个或多个操作。
  12. 一种终端设备，其特征在于，包括：收发器、处理器和存储器；
    所述收发器用于接收和发送数据；
    所述存储器存储有一个或多个程序，所述一个或多个程序包括指令，当所述指令被所述处理器执行时，使得所述电子设备执行如权利要求8-11中的任意一项所述的方法。
  13. 一种远程协助系统，其特征在于，包括：需要进行控制的目标设备、云服务器、执行如权利要求1至6任一权利要求所述的第一终端设备和执行如权利要求8至11任一权利要求所述的第二终端设备。
  14. 一种可读存储介质，用于存储指令，当所述指令被执行时，使得如权利要求1-6中的任一项或权利要求8-11中的任一项所述的方法被实现。
  15. 一种包含指令的计算机程序设备，当其在终端上运行时，使得所述终端执行如权利要求1-6中的任一项或权利要求8-11中的任一项所述的方法。
PCT/CN2020/122844 2019-12-04 2020-10-22 一种远程协助的方法、电子设备和系统 WO2021109745A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/782,016 US12126856B2 (en) 2019-12-04 2020-10-22 Remote assistance method and system, and electronic device
EP20897195.2A EP4047902B1 (en) 2019-12-04 2020-10-22 Remote assistance method, electronic device, and system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911230080.9A CN112911190A (zh) 2019-12-04 2019-12-04 一种远程协助的方法、电子设备和系统
CN201911230080.9 2019-12-04

Publications (1)

Publication Number Publication Date
WO2021109745A1 true WO2021109745A1 (zh) 2021-06-10

Family

ID=76110804

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/122844 WO2021109745A1 (zh) 2019-12-04 2020-10-22 一种远程协助的方法、电子设备和系统

Country Status (3)

Country Link
EP (1) EP4047902B1 (zh)
CN (1) CN112911190A (zh)
WO (1) WO2021109745A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113595867A (zh) * 2021-06-22 2021-11-02 青岛海尔科技有限公司 基于远程交互的设备操作方法及装置
CN113821287A (zh) * 2021-09-06 2021-12-21 深圳太力生物技术有限责任公司 基于机器人的细胞操作任务处理方法、装置、设备和介质
CN114201645A (zh) * 2021-12-01 2022-03-18 北京百度网讯科技有限公司 对象标注方法、装置、电子设备以及存储介质

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023030518A1 (zh) * 2021-09-06 2023-03-09 深圳太力生物技术有限责任公司 基于机器人的细胞操作任务处理方法、装置、设备及介质
CN114070834A (zh) * 2021-10-26 2022-02-18 深圳市商汤科技有限公司 一种远程协助方法、装置及其相关设备和存储介质
US11678002B1 (en) * 2021-12-31 2023-06-13 Universal Electronics Inc. Systems and methods for configuration of a controlling device to enable selection of one or more media streams

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150061841A1 (en) * 2013-09-02 2015-03-05 Lg Electronics Inc. Mobile terminal and method of controlling the same
CN106254462A (zh) * 2016-08-05 2016-12-21 北京奇虎科技有限公司 一种实现移动终端的远程协助的方法、服务器和系统
CN107817701A (zh) * 2017-09-20 2018-03-20 珠海格力电器股份有限公司 设备控制方法、装置、计算机可读存储介质及终端
CN108415635A (zh) * 2017-02-10 2018-08-17 广州森成和信息技术有限公司 一种图片分享系统

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10951683B2 (en) * 2015-07-13 2021-03-16 Projector.Is, Inc. Systems and methods for remote interaction
WO2018095366A1 (zh) * 2016-11-24 2018-05-31 腾讯科技（深圳）有限公司 视频推荐确定、信息显示、基于帧同步的数据处理方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150061841A1 (en) * 2013-09-02 2015-03-05 Lg Electronics Inc. Mobile terminal and method of controlling the same
CN106254462A (zh) * 2016-08-05 2016-12-21 北京奇虎科技有限公司 一种实现移动终端的远程协助的方法、服务器和系统
CN108415635A (zh) * 2017-02-10 2018-08-17 广州森成和信息技术有限公司 一种图片分享系统
CN107817701A (zh) * 2017-09-20 2018-03-20 珠海格力电器股份有限公司 设备控制方法、装置、计算机可读存储介质及终端

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4047902A4

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113595867A (zh) * 2021-06-22 2021-11-02 青岛海尔科技有限公司 基于远程交互的设备操作方法及装置
CN113821287A (zh) * 2021-09-06 2021-12-21 深圳太力生物技术有限责任公司 基于机器人的细胞操作任务处理方法、装置、设备和介质
CN114201645A (zh) * 2021-12-01 2022-03-18 北京百度网讯科技有限公司 对象标注方法、装置、电子设备以及存储介质

Also Published As

Publication number Publication date
EP4047902A1 (en) 2022-08-24
US20230008199A1 (en) 2023-01-12
EP4047902B1 (en) 2024-09-25
CN112911190A (zh) 2021-06-04
EP4047902A4 (en) 2022-12-07

Similar Documents

Publication Publication Date Title
WO2021109745A1 (zh) 一种远程协助的方法、电子设备和系统
WO2020057327A1 (zh) 信息列表显示方法、装置及存储介质
US20170364239A1 (en) Application icon customization
WO2018040063A1 (zh) 设备控制方法及装置
WO2019072096A1 (zh) 视频直播中的互动方法、装置、系统及计算机可读存储介质
CN111343476A (zh) 视频共享方法、装置、电子设备及存储介质
US10235809B2 (en) Reality to virtual reality portal for dual presence of devices
JP2017515253A (ja) 数値移転方法、端末およびクラウドサーバ
CN106506448A (zh) 直播显示方法、装置及终端
CN112738544B (zh) 直播间互动方法、装置、电子设备及存储介质
CN107113467A (zh) 用户终端装置、系统及其控制方法
EP3276975A1 (en) Method and device for controlling playing state
JP2016522952A (ja) 画像伝送の方法、装置、プログラム及び記録媒体
WO2023093698A1 (zh) 游戏直播的交互方法、存储介质、程序产品及电子设备
WO2023098011A1 (zh) 视频播放方法及电子设备
CN112312190A (zh) 视频画面的展示方法、装置、电子设备和存储介质
CN111479158A (zh) 视频展示方法、装置、电子设备及存储介质
CN104903844A (zh) 用于呈现网络和相关联的移动设备中的数据的方法
CN113489938B (zh) 虚拟会议控制方法、智能设备及终端设备
WO2024022052A1 (zh) 显示终端的控制方法、系统、显示设备以及存储介质
TW201431364A (zh) 手持設備及控制方法
CN114461124B (zh) 投屏控制方法、装置、投屏器及计算机可读存储介质
WO2019104533A1 (zh) 一种视频播放方法及装置
EP4060965A1 (en) Method and apparatus for controlling internet of things device, and medium
US12126856B2 (en) Remote assistance method and system, and electronic device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20897195

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020897195

Country of ref document: EP

Effective date: 20220516

NENP Non-entry into the national phase

Ref country code: DE