TW202343228A - Remote device control system, method and computer readable medium for augmented reality collaboration - Google Patents

Abstract

The invention discloses a remote device control system, method and computer readable medium for augmented reality collaboration. An augmented reality module of an augmented reality apparatus of a field terminal receives a live image screen of a device, so as to overlay a first operation interface on the received live image screen of the device. Next, a remote video call module of an electronic apparatus of a support terminal is connected with the augmented reality module for video collaboration to obtain the live image screen of the device, so as to overlay a second operation interface on the obtained live image screen of the device. Then, the support terminal uses the electronic apparatus to remotely control the device at the field terminal through the second operation interface, so that the support terminal can view a remote control result of the device at the field terminal through a screen provided by the remote video calling module of the electronic apparatus.

Description

Remote device control system, method and computer-readable medium for augmented reality collaboration

The present invention relates to a remote device control technology, and in particular, to a remote device control system, method and computer-readable medium for augmented reality (Augmented Reality; AR) collaboration.

When you want to control a remote device, the device itself must have remote connection capabilities in order to connect to the Internet of Things (IoT). It also requires developers to write different applications (Apps) or control programs for different devices. It is impossible to use the same application or control program to directly control multiple different devices, and all subsequent changes in different devices, large or small, require developers to intervene to re-develop/rewrite the application or control program, which is extremely time-consuming and time-consuming.

In an existing technology, an interactive remote expert collaborative maintenance system and method based on augmented reality technology is provided, which supports experts in delineating videos to send the delineated target area images to the on-site end, and uses Augmented reality technology brings feedback information from experts into real-time The video/image collected by the customer and the status information of the current device are combined into an augmented reality video.

However, this existing technology cannot allow experts at the support end to remotely control the equipment at the user's site at the on-site end, nor can the support end assist the on-site end in solving equipment control problems, nor can it use image recognition technology to identify Different devices are used to provide associated operating interfaces, and it is impossible to enable video collaboration between the on-site end and the support end to provide secure remote control.

Therefore, how to provide an innovative remote device control technology to solve any of the above problems or provide related functions/services has become a major research topic for those skilled in the art.

The present invention provides an innovative remote device control system, method and computer-readable medium for augmented reality collaboration, which can be remotely controlled by experts at the support end at the user's device at the on-site end, or The support end assists the on-site end to solve equipment control problems, or uses image recognition technology to identify different devices and provides related operating interfaces, or it allows the on-site end's augmented reality device and the support end's electronic device to interact users can conduct video collaboration connections to provide secure remote control.

The remote equipment control system for augmented reality collaboration according to the present invention includes: an augmented reality device on the field side, which has an augmented reality module to capture or receive on-site image frames including real-time images of the equipment. , and then the augmented reality module of the on-site augmented reality device overlays the first operating interface associated with the device on the captured or received on-site image screen including the real-time image of the device; the support end The electronic device has a remote video call module to communicate with the on-site terminal. The augmented reality module of the augmented reality device performs a video connection or video collaboration connection, and the remote video call module of the supporting electronic device performs a video call or video collaboration connection from the scene during the video connection or video collaboration connection. The augmented reality module of the augmented reality device on the end obtains or receives the live image including the real-time image of the device, and then the remote video call module of the electronic device on the supporting end performs the second operation associated with the device. The interface is overlaid on the obtained or received live image screen including the real-time image of the device; and the augmented reality platform provides a first operating interface and a second operating interface associated with the device for the support terminal to use electronic The device remotely controls the equipment at the site through the second operating interface provided by the augmented reality platform, so that the support end can view the equipment at the site through the screen provided by the remote video call module of the electronic device. Remote control results.

The remote device control method for augmented reality collaboration according to the present invention includes: capturing or receiving the live image frame including the real-time image of the device by the augmented reality module of the on-site augmented reality device, to The augmented reality module of the on-site augmented reality device overlays the first operating interface associated with the device on the captured or received on-site image screen including the real-time image of the device; The remote video call module of the electronic device and the augmented reality module of the on-site augmented reality device perform a video connection or video collaboration connection, so that the remote video call module of the supporting end electronic device performs a video connection or a video collaboration connection. During the video connection or video collaboration connection, the on-site image including the real-time image of the device is obtained or received from the augmented reality module of the on-site augmented reality device, and then the remote video of the electronic device on the supporting end is The call module overlays the second operating interface associated with the device on the obtained or received live image screen including the real-time image of the device; and the supporting end uses the electronic device to provide the third operating interface through the augmented reality platform. 2. The operation interface remotely controls the equipment at the site, allowing the support end to view the remote control results of the equipment at the site through the screen provided by the remote video call module of the electronic device.

The computer-readable medium of the present invention is used in a computing device or computer and stores instructions to execute the above-mentioned remote device control method for augmented reality collaboration.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, embodiments are given below and explained in detail with reference to the accompanying drawings. Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not intended to limit the scope of the invention.

1: Remote device control system

10:Augmented reality device

11:Augmented reality module

20:Augmented reality platform

201:Augmented reality scene

21: Scene editing software tools

22: Logic design software tools

23:First operating interface

231:Button

24:First control item

25: Second operating interface

251:Remote control button

26: Second control item

27:Remote control command

28:Equipment opening building block

30: Electronic devices

31:Remote video call module

40:Remote video call server

50:Equipment

51:Remote control operation instructions

A:Onsite side

B: Live video screen

C:Support end

N:Network

S1 to S5: steps

Figure 1 is a schematic structural diagram of a remote device control system for augmented reality collaboration according to the present invention.

FIG. 2 is a schematic flowchart of a remote device control method for augmented reality collaboration according to the present invention.

3 is a schematic diagram of an embodiment of the augmented reality platform in the remote device control system and method for augmented reality collaboration according to the present invention.

4A and 4B are schematic diagrams of an embodiment of the device opening building block and the connected button of the first operating interface in the remote device control system and method for augmented reality collaboration according to the present invention.

Figure 5 is a schematic diagram of an embodiment of the interactive relationship between the augmented reality device, the augmented reality platform and the device in the remote device control system and method for augmented reality collaboration according to the present invention.

6 is a schematic diagram of an embodiment of the interactive relationship between the augmented reality device, the electronic device and the device in the remote device control system and method for augmented reality collaboration according to the present invention.

7A and 7B show the remote control command and the remote control button and device opening building block of the second operating interface connected in series in the remote device control system and method for augmented reality collaboration according to the present invention ( Schematic diagram of an embodiment of the device start command).

8 is a schematic diagram of an embodiment of the interactive relationship between the augmented reality device, the augmented reality platform, the electronic device and the device in the remote device control system and method for augmented reality collaboration according to the present invention. .

The following describes the embodiments of the present invention through specific specific embodiments. Those skilled in the art can understand other advantages and effects of the present invention from the content disclosed in this specification, and can also implement it through other different specific equivalent embodiments or Use.

FIG. 1 is a schematic structural diagram of a remote device control system 1 for augmented reality collaboration according to the present invention. FIG. 2 is a schematic flow chart of a remote device control method for augmented reality collaboration according to the present invention.

As shown in Figure 1, the remote device control system 1 for augmented reality collaboration may include an augmented reality device 10 of the field end A that is connected or communicated with each other through a network N, and an augmented reality platform. 20. At least one (eg, plural) electronic device 30 supporting terminal C and a remote video call server 40. The augmented reality device 10 of the field end A may have an augmented reality module (such as an application) 11, etc., and the augmented reality platform 20 may have a scene editing software tool 21, a logic design software tool 22, a The first operation interface 23, at least one (such as plural) first control item 24, a second operation interface 25, at least one (such as plural) second control item 26, at least one (such as plural) remote control command 27 and at least A device opens the building block 28, etc., and the electronic device 30 supporting terminal C can have a remote Terminal video call module (such as application) 31, etc.

"At least one" used in the present invention represents more than one (such as one, two or more than three), "plural" represents more than two (such as two, three, four, five or more than ten), "connection" or "communication" ” represents connection or communication through wireless or wired methods. Network N can be a wireless network or a wired network, for example, the wireless network is Bluetooth, Wireless-Fidelity (Wi-Fi), etc., and Network N can also further cover the Internet and regions. Network (LAN), wireless local area network (WLAN), wide area network (WAN), metropolitan area network (MAN), virtual private network (VPN), etc.

In one embodiment, the augmented reality device 10 of the site A and the electronic device 30 of the supporting terminal C may be located in different locations or different areas, for example, the augmented reality device 10 of the field A and the electronic device 30 of the supporting terminal C 30 are located at the site and the remote end of the device 50 respectively. The personnel on site A may represent users (site personnel) at the site of the equipment 50, such as maintenance personnel, students, learners, trainees, etc. The personnel of the support terminal C may represent the supporters (support personnel) at the remote end of the equipment 50, such as experts (such as remote experts) who are familiar with the equipment 50, professionals, operators, controllers, coaches, professors, teachers, etc. Equipment 50 is a variety of physical equipment that can be connected to the network N, such as Internet of Things (IoT) equipment, electronic equipment (such as motors/generators/electronic instruments), industrial equipment, semiconductor equipment, computer equipment, network equipment, Communication equipment, home appliances (such as electrical appliances), Internet of Vehicles equipment (such as electric vehicles), machinery and equipment (such as robots), etc.

In one embodiment, the augmented reality device 10 may be augmented reality glasses (i.e., AR glasses or AR smart glasses), or may be a smartphone, tablet, or notebook with augmented reality (AR) functions. type computers, personal digital assistants (PDA), etc. The augmented reality platform 20 may be a cloud platform composed of at least one or multiple servers, and the server may be a cloud server or a server host. The scene editing software tool 21 may be a user scene editing software tool or the like. The first operation interface 23 can be used by the field terminal A or the augmented reality device 10, and can also have a button such as a Bluetooth Low Energy (BLE) button. The second operation interface 25 can be used by the support terminal C or the electronic device 30, and can also have a button such as a remote button. The electronic device 30 may be a computer, a mobile device, a portable device, etc., such as a personal computer, a tablet computer, a notebook computer, a desktop computer, a smart phone, a personal digital assistant (PDA), etc.

As shown in FIGS. 1 and 2 , the remote device control method for augmented reality collaboration according to the present invention may include the following steps S1 to S5.

Step S1: The augmented reality module (such as an application) 11 of the augmented reality device 10 of the scene terminal A captures or receives the scene image B including the real-time image of the device 50 during its execution.

In other words, the augmented reality device 10 (such as AR glasses or tablet computer) of the on-site terminal A can provide the augmented reality module (such as an application) 11 to use the augmented reality module (such as an application) 11 During its execution, capture or receive the live image B including the real-time image of the device 50 through the camera (such as lens) of the augmented reality device 10 .

Step S2: The augmented reality module (such as an application) 11 of the augmented reality device 10 of the field terminal A uses image recognition technology to capture or receive the field image frame B including the real-time image of the device 50 The type, name or identity of the device 50 is identified, and the augmented reality module (such as an application) 11 obtains data from the database of the augmented reality platform 20 based on the identified type, name or identity of the device 50 . Dynamically obtained to provide a first operation interface 23 with a button (such as a Bluetooth low energy button) associated with the device 50, and then the augmented reality module (such as an application) 11 will use the first operation interface 23 associated with the device 50 An operating interface 23 is overlaid on a real-time image including the device 50 On the live image screen B of the image, the on-site terminal A having the augmented reality device 10 can control the device 50 through the first operating interface 23 or its button (such as a BLE button).

In other words, the scene editing software tool 21 (such as the user scene editing software tool) and the logic design software tool 22 can be provided on the augmented reality platform 20 (such as a cloud platform), and through Bluetooth or wireless fidelity (Wi-Fi). Fi) and other wireless methods to obtain the remote control operation instructions 51 (such as power, light, temperature, etc. instructions) provided by the connected device 50. At the same time, a first operation interface 23 and at least one (eg plural) first control items 24 associated with the device 50 can be added or designed on the augmented reality scene 201 (see FIG. 3 ) of the augmented reality platform 20 , for the augmented reality module (such as an application) 11 of the augmented reality device 10 of the field terminal A to use the first operation newly added or designed on the augmented reality scene 201 of the augmented reality platform 20 Interface 23 and first control item 24.

The augmented reality module (such as an application) 11 of the augmented reality device 10 of the field end A can identify the type, name or identity of the device 50 based on the scene image B including the real-time image of the device 50, so as to The augmented reality module (such as an application) 11 overlays the first operating interface 23 associated with the device 50 on the live image including the real-time image of the device 50 based on the recognized type, name or identity of the device 50 . On screen B, the field terminal A having the augmented reality device 10 controls the device 50 through the first operating interface 23 or its button (such as a BLE button).

Step S3: Provide a video call function in the augmented reality module (such as an application) 11 of the augmented reality device 10 of site A to use the augmented reality device 10 of site A to The module (such as an application) 11 uses the video call function to perform a video connection or a video collaboration connection with the remote video call module (such as an application) 31 of the electronic device 30 supporting terminal C.

In other words, the augmented reality module (such as an application) 11 of the augmented reality device 10 of the site A can utilize the video call function to communicate with the remote video call of the electronic device 30 of the supporting terminal C. The phone module (such as an application) 31 is used for video connection or video collaboration connection. That is, a call function button can be designed on the augmented reality scene 201 of the augmented reality platform 20 to provide a video call function to the augmented reality module (such as application) of the augmented reality device 10 by using the call function button. Program)11. In addition, the details of the call function button can be designed on the augmented reality scene 201 of the augmented reality platform 20 , and the details of the call function button can include the remote video call module (such as an application) 31 of the electronic device 30 An identification code (ID) or address (such as an IP address) is used to create an augmented reality by the remote video call server 40 through the identification code (ID) or address of the remote video call module (such as an application) 31 A video connection or video collaboration connection between the augmented reality module (such as an application) 11 of the environment device 10 and the remote video call module (such as an application) 31 of the electronic device 30.

Step S4: Video connection or video collaboration between the augmented reality module (such as an application) 11 of the augmented reality device 10 and the remote video call module (such as an application) 31 of the electronic device 30 During the duration of the line, the remote video call module (such as an application) 31 of the electronic device 30 obtains or receives real-time information including the device 50 from the augmented reality module (such as an application) 11 of the augmented reality device 10 The live image screen B of the image is provided through the database of the augmented reality platform 20 and the second operation interface 25 associated with the device 50 is provided, and then the remote video call module (such as an application) 31 is associated with the device 50 The associated second operating interface 25 is overlaid on the obtained or received live image screen B including the real-time image of the device 50 .

In other words, the remote video call server 40 creates both the augmented reality module (such as an application) 11 of the augmented reality device 10 and the remote video call module (such as an application) 31 of the electronic device 30 After the video connection or video collaboration connection, the augmented reality module (such as an application) 11 of the augmented reality device 10 and the remote video call module (such as an application) of the electronic device 30 can also be further connected. 31 Establish a point-to-point video channel between the two. Remote video call of electronic device 30 The module (such as an application) 31 can continuously perform video streaming with the augmented reality module (such as an application) 11 of the augmented reality device 10 through a point-to-point video channel established by the remote video call server 40 , to ensure low latency of video streaming between the augmented reality module (such as an application) 11 and the remote video call module (such as an application) 31, and then the remote video call module of the electronic device 30 (eg, application program) 31 continuously obtains or receives the live image frame B including the real-time image of the device 50 from the augmented reality module (eg, application program) 11 of the augmented reality device 10 .

The remote video call server 40 can be between the augmented reality module (such as an application) 11 of the augmented reality device 10 and the remote video call module (such as an application) 31 of the electronic device 30 In addition to establishing a point-to-point video channel, the augmented reality module (such as an application) 11 of the augmented reality device 10 and the remote video call module (such as an application) 31 of the electronic device 30 can also be established. Establish a data channel to transmit information or data, so that the augmented reality module (such as an application) 11 of the augmented reality device 10 can use the identified identification code (ID) of the device 50 and other information through a remote video call The data channel established by the server 40 is sent to the remote video call module (such as an application) 31 of the electronic device 30 .

A second operating interface 25 and at least one (eg, plural) second control items 26 associated with the device 50 can be added or designed on the augmented reality scene 201 of the augmented reality platform 20 to support terminal C. The remote video call module (such as an application) 31 of the electronic device 30 uses the second operation interface 25 and the second control item 26 newly added or designed on the augmented reality scene 201 of the augmented reality platform 20 . In addition, the remote video call module (such as an application) 31 of the electronic device 30 can set the second operating interface 25 associated with the device 50 based on information such as the identification code (ID) of the device 50 obtained from the data channel. Overlaid on the obtained or received live image screen B including the real-time image of the device 50 .

Step S5: Establish a video connection or video collaboration connection between the augmented reality module (such as an application) 11 of the augmented reality device 10 and the remote video call module (such as an application) 31 of the electronic device 30 During the existence of the line, the support terminal C uses the electronic device 30 to remotely control the device 50 at the onsite terminal A through the second operating interface 25 provided by the augmented reality platform 20, and then the support terminal C uses the electronic device 30 The screen provided by the remote video call module (such as an application) 31 is used to view the remote control results of the device 50 at the site A.

In other words, the augmented reality platform 20 can be used to add or design at least one (such as plural) remote control commands (Remote Command) 27 on the augmented reality scene 201, and then use the augmented reality platform 20 to The logic design software tool 22 connects the remote control command 27 to the second operating interface 25 used by the electronic device 30 of the support terminal C. When the support terminal C uses the second operation interface 25 through the electronic device 30, the second operation interface 25 used by the electronic device 30 of the support terminal C sends the remote control command 27 to the amplification of the field terminal A through the data channel. The reality device 10 then uses the first control item 24 used by the augmented reality device 10 of the field end A to connect the corresponding remote control command 27 in series by the logic design software tool 22 of the augmented reality platform 20. Equipment operating instructions.

When the augmented reality device 10 of the field terminal A receives the remote control command 27 from the data channel, the field terminal A triggers the equipment operation command corresponding to the remote control command 27 through the augmented reality device 10 . Then, the second operating interface 25 used by the supporting end C through the electronic device 30 uses the equipment operation command to remotely control the equipment 50 at the on-site end A, so that the supporting end C can use the remote video call mode of the electronic device 30 The screen provided by the group (eg application) 31 views the remote control results of the device 50 at the site A.

Figure 3 shows the remote device control system 1 for augmented reality collaboration according to the present invention. A schematic diagram of an embodiment of the augmented reality platform 20 in the method and the method. Figures 4A and 4B are the remote device control system 1 for augmented reality collaboration according to the present invention and the related device opening building blocks 28 and 28 in the method. A schematic diagram of an embodiment of the button 231 of the first operation interface 23 connected in series. Figure 5 shows the augmented reality device 10 and the remote device control system 1 and its method for augmented reality collaboration according to the present invention. A schematic diagram of an embodiment of the interactive relationship between the augmented reality platform 20 and the device 50. Figure 6 shows the augmented reality device 10 in the remote device control system 1 and its method for augmented reality collaboration according to the present invention. , a schematic diagram of an embodiment of the interactive relationship between the electronic device 30 and the device 50. Figures 7A and 7B illustrate the remote control instructions 27 in the remote device control system 1 and its method for augmented reality collaboration according to the present invention. A schematic diagram of an embodiment of the remote control button 251 and the device opening building block 28 (device opening command) connected in series with the second operating interface 25. Figure 8 is a remote device control system 1 for augmented reality collaboration according to the present invention. A schematic diagram of an embodiment of the interactive relationship between the augmented reality device 10, the augmented reality platform 20, the electronic device 30 and the equipment 50 in the method.

For example, the following embodiments shown in Figures 3 to 8 are used as demonstrations to further illustrate the contents of the remote device control system 1 and its method for augmented reality collaboration according to the present invention, and refer to Figure 1 Refer to Figure 2 for explanation.

For example, the power switch of the device 50 can be turned on or off using Bluetooth, and the augmented reality platform 20 can provide the device 50 with the ability to remotely control the power switch of the device 50 . At the same time, the augmented reality platform 20 is used to add or design the first operation interface 23 and the second operation interface 25 on the augmented reality scene 201, and the logic design software tool 22 of the augmented reality platform 20 (such as Building block logic design tool) to connect the control functions or control behaviors of the device 50 in series.

Step S1: Acquire or receive the live image frame B including the real-time image of the device 50 from the augmented reality module (such as an application) 11 of the augmented reality device 10 of the live end A. that is, The maintenance personnel of the on-site terminal A can capture or receive on-site image frames including real-time images of the equipment 50 through the augmented reality module (such as an application) 11 of the augmented reality device 10 (such as a mobile phone or AR glasses). B, and then the on-site terminal A watches the on-site image B including the real-time image of the device 50 through the augmented reality device 10 .

Step S2: The device 50 is identified by the augmented reality module (such as an application) 11 of the augmented reality device 10 to provide the first operating interface 23 associated with the device 50, and then the first operating interface 23 is set Superimposed on the live image screen B including the real-time image of the device 50, the live end A controls the device 50 through the first operating interface 23 or its button 231 (such as a BLE button). For example, step S2 may include the following procedures P21 to P24.

In program P21, as shown in FIG. 3 , the scene editing software tool 21 of the augmented reality platform 20 is used to add or design the first augmented reality device 10 used by the field end A on the augmented reality scene 201 . An operation interface 23, and the first operation interface 23 may have at least one (such as plural) buttons 231 that are Bluetooth low energy (BLE) buttons, etc., but it is not limited to this.

In program P22, as shown in FIG. 4A , the augmented reality module (such as an application) 11 of the augmented reality device 10 can obtain the remote control operation command 51 of the device 50 (such as a Bluetooth connection power switch operation command). Specifications and parameters, in order to edit the remote control operation command 51 of the device 50 into a device open (Device Open) block 28 in the augmented reality platform 20 based on the specifications and parameters, and then provide the device open block 28 to the augmented reality platform. Use of 20 logic design software tools 22 (such as building block logic design tools).

In program P23, as shown in Figure 4B, the logic design software tool 22 of the augmented reality platform 20 connects the button 231 (such as the BLE button) of the first operation interface 23 to the device to open the building block 28, and then the logic design software The tool 22 connects the buttons of the first operation interface 23 231 and the device opening building block 28 are provided to the augmented reality device 10 of the field end A for use as the control device 50 .

In program P24, as shown in Figure 5, when the maintenance personnel at the site A wants to turn on the power of the device 50, the augmented reality module (such as an application) of the augmented reality device 10 at the site A 11 Identify the type, name or identity of the device 50 so that the augmented reality module (such as an application) 11 can obtain the data from the augmented reality platform 20 based on the identified type, name or identity of the device 50 The first operation interface 23 associated with or having a button 231 (such as a BLE button) used by the augmented reality device 10 of the field terminal A is dynamically obtained from the library, and then the augmented reality module (such as an application) 11 The first operation interface 23 with the button 231 (such as the BLE button) used by the augmented reality device 10 of the field end A is overlaid on the field image screen B including the real-time image of the device 50, so as to have the augmented reality The field end A of the device 10 controls the device 50 through the first operation interface 23 or its button 231 (such as a BLE button).

Step S3: Provide a video call function in the augmented reality module (such as an application) 11 of the augmented reality device 10 at the site A, so that the augmented reality module (such as an application) of the augmented reality device 10 The application) 11 uses the video call function to connect with the remote video call module (such as the application) 31 of the electronic device 30.

That is, when the maintenance personnel of the on-site terminal A are not sure whether the power of the device 50 should be turned on, the on-site terminal A can seek assistance from the remote experts who support the terminal C. For example, the onsite terminal A can input the IP (Internet Protocol) address of the electronic device 30 of the supporting terminal C through the augmented reality device 10 to call the electronic device 30 of the supporting terminal C, and the remote video call server 40 can Use Web Real-Time Communication (WebRTC) technology to establish a remote video call between the augmented reality module (such as an application) 11 of the augmented reality device 10 and the electronic device 30 Module (such as application) 31 video connection or video collaboration connection.

Step S4: The remote video call module (such as an application) 31 of the electronic device 30 receives the live image B including the real-time image of the device 50 to provide the second operation interface 25 associated with the device 50, and then the second operation interface 25 is provided. The two operation interfaces 25 are overlaid on the obtained or received live image screen B including the real-time image of the device 50 . For example, step S4 may include the following procedures P41 to P43.

In the process P41, the video connection or video communication between the augmented reality module (such as an application) 11 of the augmented reality device 10 and the remote video call module (such as an application) 31 of the electronic device 30 During the duration of the collaborative connection, the remote video call module (such as an application) 31 of the electronic device 30 obtains or receives the device 50 from the augmented reality module (such as an application) 11 of the augmented reality device 10 The live image frame B of the real-time image, and then the augmented reality module (such as an application) 11 of the augmented reality device 10 transmits the identification code (ID) of the device 50 to the remote end of the electronic device 30 of the supporting terminal C Video call module (such as application)31.

In the process P42, the scene editing software tool 21 of the augmented reality platform 20 is used to add or design the second operation interface 25 used by the electronic device 30 of the support terminal C on the augmented reality scene 201, and the second operation The interface 25 may have at least one (such as a plurality of) remote control buttons 251, etc., but is not limited thereto.

In program P43, as shown in FIG. 6 , a second operation interface 25 with a remote control button 251 used by the electronic device 30 supporting terminal C is provided or generated on the electronic device 30 to make a remote video call by the electronic device 30 The module (such as an application) 31 overlays the second operation interface 25 with the remote control button 251 on the obtained or received live image screen B including the real-time image of the device 50 .

Step S5: The supporting terminal C uses the electronic device 30 to control the device 50 at the on-site terminal A through the second operating interface 25 provided by the augmented reality platform 20, and then the supporting terminal C uses the remote video of the electronic device 30 The screen provided by the call module (such as an application) 31 views the remote control results of the device 50 at the site A. For example, step S5 may include the following procedures P51 to P53.

In program P51, as shown in FIG. 7A , a remote control command (Remote Command) 27 is added or designed on the augmented reality scene 201 of the augmented reality platform 20 to use the logic of the augmented reality platform 20 The design software tool 22 connects the remote control command 27 to the remote control button 251 of the second operating interface 25 .

In program P52, as shown in FIG. 7B , the logic design software tool 22 of the augmented reality platform 20 connects the remote control command 27 received by the augmented reality device 10 of the field end A to the device opening building block 28 in series. device start command.

In process P53, as shown in FIG. 8 , both the augmented reality module (such as an application) 11 of the augmented reality device 10 and the remote video call module (such as an application) 31 of the electronic device 30 During the duration of the video connection or video collaboration connection, the expert on the supporting end C wants to demonstrate how to turn on the power of the device 50 . Therefore, the expert on the supporting end C can click or press the remote control of the second operating interface 25 through the electronic device 30 Button 251.

When the expert on the support terminal C clicks or presses the remote control button 251 of the second operation interface 25 through the electronic device 30, the augmented reality platform 20 transmits the remote control button 251 of the second operation interface 25 through the network N. The serially connected remote control command 27 is sent to the augmented reality device 10 of the site A. The first control item 24 used by the augmented reality device 10 of site A will monitor (listen) the remote control command 27 sent by the augmented reality platform 20 to control the first control item 24. 24. When receiving the monitoring remote control command 27, the device opening command of the device opening block 28 is triggered, and then the device opening command of the device opening building block 28 is connected through the connection between the augmented reality device 10 of the field end A and the device 50. (such as Bluetooth connection), using a wireless communication protocol such as the Bluetooth Low Energy (BLE) protocol to turn on the power of the device 50 .

In addition, the present invention also provides a computer-readable medium for a remote device control method for augmented reality collaboration, which is applied to a computing device or computer with a processor and/or memory, and the computer-readable medium Instructions are stored and can be executed by a computing device or computer through a processor and/or memory to execute the above content when the computer-readable medium is executed. For example, the processor can be a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc., and the memory can be a random access memory (RAM), a memory card, a hard drive (such as a cloud/network hard drive). discs), databases, etc., but are not limited to this.

In summary, the remote device control system, method and computer-readable medium for augmented reality collaboration according to the present invention have at least the following features, advantages or technical effects.

1. The present invention enables experts at the support end to remotely control the equipment at the user's site at the on-site end, and also facilitates the support end to assist the on-site end in solving equipment control problems.

2. This invention can allow experts at the support end to directly control equipment at the user site at the on-site end, thereby effectively reducing the communication time between the support end and the on-site end, and also providing services or assistance to the on-site end more quickly and efficiently. .

3. The present invention allows the support end expert to directly control the equipment at the user's site through the remote video call module of the electronic device, so as to facilitate the support end to quickly provide or pass on the control method of the equipment (such as control knowledge or experience) ) to the on-site end.

4. The present invention can use image recognition technology to identify the types of different devices. Name or identity, etc., which is conducive to dynamically providing associated first or second operating interfaces based on different devices.

5. The present invention can use image recognition technology to dynamically provide the first or second operating interface corresponding to the device, and there is no need to provide different applications or control programs for different devices.

6. For equipment that cannot be controlled remotely, the present invention enables video connections or video collaboration connections between the on-site augmented reality device and the support-side electronic device to facilitate video connections or video collaboration. It provides safe remote control of the device while connected, and can also control the device more effectively or easily.

The above embodiments are only illustrative of the principles, characteristics and effects of the present invention, and are not intended to limit the scope of the present invention. Anyone skilled in the art can make the above-mentioned modifications without violating the spirit and scope of the present invention. Modify and change the implementation form. Any equivalent changes and modifications made using the contents disclosed in the present invention shall still be covered by the patent application. Therefore, the protection scope of the present invention should be as listed in the patent application scope.

1: Remote device control system

10:Augmented reality device

11:Augmented reality module

20:Augmented reality platform

21: Scene editing software tools

22: Logic design software tools

23:First operating interface

24:First control item

25: Second operating interface

26: Second control item

27:Remote control command

28:Equipment opening building block

30: Electronic devices

31:Remote video call module

40:Remote video call server

50:Equipment

51:Remote control operation instructions

A:Onsite side

B: Live video screen

C:Support end

N:Network

Claims (17)

A remote device control system for augmented reality collaboration, including: The on-site augmented reality device has an augmented reality module to capture or receive on-site image frames including real-time images of the device, and then the augmented reality module of the on-site augmented reality device Overlay the first operating interface associated with the device on the captured or received live image screen including the real-time image of the device; The electronic device of the supporting terminal has a remote video call module for video connection or video collaboration connection with the augmented reality module of the augmented reality device of the field terminal, and the electronic device of the supporting terminal During the video connection or video collaboration connection, the remote video call module obtains or receives the live image including the real-time image of the device from the augmented reality module of the on-site augmented reality device, Then, the remote video call module of the electronic device of the supporting end overlays the second operating interface associated with the device on the obtained or received live image screen including the real-time image of the device; and The augmented reality platform provides the first operating interface and the second operating interface associated with the device, and the support terminal uses the electronic device to use the second operating interface provided by the augmented reality platform Remote control of the equipment at the site allows the support terminal to view the remote control results of the equipment at the site through the screen provided by the remote video call module of the electronic device. The remote equipment control system as described in claim 1, wherein the augmented reality module of the on-site augmented reality device further uses image recognition technology to capture or receive real-time images including the equipment. The type, name or identity of the device is identified in the live image, and then the augmented reality module obtains information related to the device from the augmented reality platform based on the identified type, name or identity of the device. Connect to the first operating interface. The remote device control system as described in claim 1, wherein the first operating interface and at least one first control associated with the device are added or designed on the augmented reality scene of the augmented reality platform. Item, the second operating interface and at least one second control item, for the augmented reality module of the augmented reality device to use the newly added or designed augmented reality scene of the augmented reality platform The first operating interface and the first control item are used by the remote video call module of the electronic device to use the second operating interface and the second control item newly added or designed by the augmented reality platform. The remote device control system as described in claim 1, wherein the augmented reality platform has a scene editing software tool, so that the scene editing software tool creates new information on the augmented reality scene of the augmented reality platform. The first operation interface with at least one button and the second operation interface with at least one remote control button may be designed in addition. The remote device control system as described in claim 1, wherein the augmented reality platform has a logical design software tool, and at least one remote device is added or designed on the augmented reality scene of the augmented reality platform. terminal control command, so that the logic design software tool connects the remote control command to the second operating interface, so that when the supporting terminal uses the second operating interface through the electronic device, the second operating interface sends the Remote control commands to the augmented reality device at the site. The remote device control system as described in request item 1 further includes a remote video call server, wherein a call function button is designed on the augmented reality scene of the augmented reality platform to utilize the call function button to provide The video call function is provided to the augmented reality module of the augmented reality device, and wherein the call function button includes the identification code or address of the remote video call module of the electronic device to make the remote video call The server establishes a video connection or video collaboration between the augmented reality module of the augmented reality device and the remote video call module of the electronic device through the identification code or address of the remote video call module. Wired. The remote device control system as described in claim 1 further includes a remote video call server, which is connected to both the augmented reality module of the augmented reality device and the remote video call module of the electronic device A point-to-point video channel is established between the point-to-point video channel established by the remote video call module of the electronic device through the remote video call server and the augmented reality module of the augmented reality device. The video streaming is continued, and then the remote video call module of the electronic device continues to obtain or receive the live image including the real-time image of the device from the augmented reality module of the augmented reality device. The remote device control system as described in claim 1 further includes a remote video call server, which is connected to both the augmented reality module of the augmented reality device and the remote video call module of the electronic device A data channel is established between the two devices so that the augmented reality module of the augmented reality device transmits the identified identification code of the device to the electronic device through the data channel established by the remote video call server. The remote video call module of the electronic device then overlays the second operating interface associated with the device on the obtained or received information including the device according to the identification code of the device. Real-time video on the live video screen. A remote device control method for augmented reality collaboration, including: The on-site image frame including the real-time image of the device is captured or received by the augmented reality module of the on-site augmented reality device, so that the augmented reality module of the on-site augmented reality device communicates with The first operating interface associated with the device is overlaid on the captured or received live image screen including the real-time image of the device; A video connection or video collaboration connection is made between the remote video call module of the supporting terminal's electronic device and the augmented reality module of the on-site augmented reality device, so that the remote terminal of the supporting terminal's electronic device During the video connection or video collaboration connection, the client video call module obtains or receives the live image including the real-time image of the device from the augmented reality module of the field-side augmented reality device, and then uses The remote video call module of the supporting electronic device will be associated with the device. The second operating interface is overlaid on the obtained or received live image screen including the real-time image of the device; and The supporting end uses the electronic device to remotely control the device at the on-site end through the second operating interface provided by the augmented reality platform, so as to provide the supporting end with remote video calls through the electronic device. The screen provided by the module allows you to view the remote control results of the device at the site. The remote device control method as described in claim 9 further includes using image recognition technology by the augmented reality module of the on-site augmented reality device to capture or receive real-time images including the device. The type, name or identity of the device is identified in the live image, and then the augmented reality module obtains information related to the device from the augmented reality platform based on the identified type, name or identity of the device. Connect to the first operating interface. The remote device control method described in claim 9 further includes adding or designing the first operating interface and at least one first control associated with the device on the augmented reality scene of the augmented reality platform. Item, the second operating interface and at least one second control item, for the augmented reality module of the augmented reality device to use the newly added or designed augmented reality scene of the augmented reality platform The first operating interface and the first control item are used by the remote video call module of the electronic device to use the second operating interface and the second control item newly added or designed by the augmented reality platform. The remote device control method as described in claim 9 further includes using the scene editing software tool of the augmented reality platform to add or design a button with at least one button on the augmented reality scene of the augmented reality platform. The first operation interface and the second operation interface having at least one remote control button. The remote device control method described in claim 9 further includes adding or designing at least one remote control command on the augmented reality scene of the augmented reality platform to control the logic of the augmented reality platform. The design software tool connects the remote control command to the second operating interface so that When the support terminal uses the second operating interface through the electronic device, the remote control command is sent from the second operating interface to the augmented reality device of the field terminal. The remote device control method described in claim 9 further includes designing a call function button on the augmented reality scene of the augmented reality platform, so as to use the call function button to provide a video call function to the augmented reality The augmented reality module of the device, wherein the call function button includes the identification code or address of the remote video call module of the electronic device, so that the remote video call server can use the remote video call module The identification code or address establishes a video connection or video collaboration connection between the augmented reality module of the augmented reality device and the remote video call module of the electronic device. The remote device control method as described in claim 9 further includes using a remote video call server to control both the augmented reality module of the augmented reality device and the remote video call module of the electronic device. Establish a point-to-point video channel between the point-to-point video channel established by the remote video call module of the electronic device through the remote video call server and the augmented reality module of the augmented reality device. Perform video streaming, and then the remote video call module of the electronic device continuously obtains or receives the live image including the real-time image of the device from the augmented reality module of the augmented reality device. The remote device control method as described in claim 9 further includes using a remote video call server to control both the augmented reality module of the augmented reality device and the remote video call module of the electronic device. A data channel is established between the devices so that the augmented reality module of the augmented reality device transmits the identified identification code of the device to the electronic device through the data channel established by the remote video call server. The remote video call module then overlays the second operating interface associated with the device on the obtained or received real-time information including the device based on the identification code of the device. The live image of the image is on the screen. A computer-readable medium, used in a computing device or a computer, stores instructions to execute the remote device control method for augmented reality collaboration as described in any one of claims 9 to 16.

Images