WO2023075644A1

WO2023075644A1 - "industrial mr assistant" system

Info

Publication number: WO2023075644A1
Application number: PCT/RU2022/050334
Authority: WO
Inventors: Станислав Александрович ВОРОНИН
Original assignee: Станислав Александрович ВОРОНИН
Priority date: 2021-10-26
Filing date: 2022-10-20
Publication date: 2023-05-04

Abstract

The invention relates to systems for enabling interaction between a remote expert and an on-site specialist. A system for providing support from a remote expert to a user comprises an automated remote expert workstation and a portable user device. Each module comprises a microcomputer, to which modules of a portable user device are connected; a network communication module; a display; a camera module; a controllable camera stabilizer module; an adaptive lighting module; a controllable laser pointer module designed to allow a remote expert to pinpoint an object with a laser beam; a sound reproduction module; and a human-machine interface module. The invention makes it possible to provide greater information support to on-site specialists when performing start-up operations and to increase the scope of communication between remote experts and on-site specialists.

Description

INDUSTRIAL MR ASSISTANT SYSTEM

FIELD OF TECHNOLOGY

The present technical solution relates to the field of interaction between a remote expert and a field specialist using human-machine interfaces, in particular, user wearable devices, as well as methods for connecting a remote expert to a user wearable device.

BACKGROUND OF THE INVENTION

Known from the prior art is the source of information RU 2 745 205 C1, published on March 22, 2021, which discloses a user wearable device capable of communicating with industrial automation systems. The user wearable device includes three modules. The first module contains electrical and electromechanical components and is equipped with rails for installing the second module. The second one is movable along the guides of the first module to change the focal length of the lens block and is a movable optical block consisting of a lens block and an intermediate reflector. The third module is adapted to the position of the human eye and is a movable holographic reflector. All modules in the aggregate are a housing made with the possibility of mounting on the head element. On the body there is an LED backlight, a camera module capable of photo/video recording of objects around the user for holding a teleconference with a remote specialist, registering user actions, scanning at least one QR code containing information for setting up a user wearable device and obtaining information from industrial automation systems. The invention provides an increase in the information awareness of personnel by displaying information on a movable holographic reflector, using warning light and/or sound signals, as well as integrating into data collection and analysis systems.

The source of information is also known from the prior art, US 20180324229 A1, published on 11/08/2018. revealing a system and a method for providing expert assistance from a remote expert to a user working with an augmented reality device. The essence of the solution lies in the fact that the server receives a request for remote assistance from the first user device controlled by the first user located in the first place, and establishes a network connection between the first user device and the second user device controlled by the second user located in the second place in response to request remote assistance. Visual information captured by the camera of the first user device, provided to the second user device controlled by the second user. The assistive content created by the second user using the second user device is provided to the first user device to present the assistive content to the first user.

The disadvantages of the sources of information known from the prior art are the lack of a camera module stabilizer for obtaining a stabilized video stream without distortion by a remote specialist, as well as the absence of laser pointers, through which a remote expert can point to an object of interest, for example, a breakdown. Another disadvantage is the location of the modules in one housing, which increases the size and weight of the structure, and it is also impossible to replace the modules without turning off the device. The disadvantages also include the inability to connect one expert to multiple user wearable devices and connect multiple remote experts to multiple user devices.

The proposed technical solution differs from those known from the prior art in that the user wearable device is a modular design with the ability to place modules on the user's body and/or elements of clothing and/or personal protective equipment for the user, the presence in the user wearable device of a controllable camera stabilizer module and a module controlled laser designators. Also, the difference of the proposed solution is the ability to connect one expert to several user wearable devices and connect several remote experts to several user devices.

SUMMARY OF THE INVENTION

The technical problem to be solved by the claimed technical solution is insufficiently effective communication between a remote expert and a field specialist in the field specialist support, commissioning and equipment maintenance. The independent claims provide a solution to the above problem, namely a modular user wearable device, as well as methods for connecting a remote expert to a modular user wearable device. Additional embodiments of the present invention are presented in dependent claims.

The technical result achieved by solving the above technical problem is to increase the information awareness of the field specialist in the implementation of commissioning and maintenance of equipment, by providing a stable and secure connection with a remote specialist who, if necessary, points the module of controlled target designators to the object of interest and thereby improves the quality of communication between the remote expert and the field specialist, and also eliminates errors caused by misunderstandings. Additionally, the technical result consists in the implementation of the appointment. Additionally, the technical result is to reduce the time for troubleshooting equipment in the enterprise and thereby not stopping the process of the enterprise.

The claimed result is achieved by means of a user support system by a remote expert, including an automated workstation (AWS) of a remote expert and a user wearable device, the user wearable device being a modular design, with the possibility of placing the modules on the user's body and/or clothing items and/or personal protective equipment. protecting the user, each module is made in a shock-resistant housing and includes: a microcomputer to which modules of the user wearable device are connected, configured to determine the connected modules of the user wearable device; a network interaction module configured to provide stable and wireless communication for connecting the remote expert workstation to the user wearable device and connecting the user wearable device to external network resources; a display configured to display information; a camera module configured to capture a user's wearable device's work view area; a controlled stabilizer module of the camera module, configured to stabilize the shooting frame, as well as change the position of the camera module by a remote expert; an adaptive lighting module configured to illuminate the user's wearable device user's work view area and/or to provide the required exposure of the camera module; a controlled laser designator module, configured to highlight an object with a laser beam by a remote expert, and the object is in the working area of the user's view of the user's wearable device; an audio playback module, configured to reproduce the speech of the remote expert through networking tools, as well as audio signals about ongoing events of the interface of the user wearable device; an audio recording module, configured to record the user's speech and sounds around the user's wearable device, and transmit, via networking means, to the remote expert's workstation; a human-machine interface module configured to interact with a user and a user wearable device.

In a particular implementation of the proposed solution, the module housings are protected by the standard from IP65 to IP68.

In another particular embodiment of the proposed solution, the user wearable device further comprises a portable stand-alone battery module configured to provide electrical power to the user wearable device.

In another particular embodiment of the proposed solution, the user wearable device additionally contains a network interface group module, which is a hub of free network interface connectors.

In another particular embodiment of the proposed solution, the camera module is a high resolution monocular camera or a high resolution stereo camera.

In another particular embodiment of the proposed solution, the controlled stabilizer module of the camera module is an electromechanical three-axis holder-stay of the camera module.

In another particular embodiment of the proposed solution, the controllable stabilizer module of the camera module is a digital stabilizer of the shooting frame.

In another particular embodiment of the proposed solution, the module of controlled laser designators is an optical laser targeting complex.

In another particular embodiment of the proposed solution, the optical laser target indication complex is a controlled device for correcting the direction of the laser beam, a laser light source and a laser rangefinder.

In another particular embodiment of the proposed solution, the optical laser target indication complex is a laser light source, an optical attachment for changing the shape of the laser beam and a laser rangefinder.

In another particular embodiment of the proposed solution, the adaptive lighting module is an LED light source and/or at least one IR light source and/or an IR light source integrated into the camera module;

In another particular embodiment of the proposed solution, the sound reproduction module is made in the form of a mono-acoustic system or a stereo-acoustic system.

In another private embodiment of the proposed solution, the sound reproduction module is made in the form of external speakers located on the body of the user's wearable device. In another private embodiment of the proposed solution, the sound reproduction module is made in the form of plug-in headphones.

In another particular embodiment of the proposed solution, the sound recording module can be made in the form of an electromagnetic microphone or a condenser microphone, or a laryngphone, or a piezomicrophone.

In another particular embodiment of the proposed solution, the human-machine interface module is a module with a set of buttons and/or joysticks and/or touch panels and/or gesture detectors and light indicators of device operation.

The claimed technical result is also achieved due to the operation of the method for connecting a remote expert to a user's wearable device for data exchange, comprising the steps of: a. at the automated workstation (AWP) of the remote expert, parameters are entered for connecting to the user's wearable device; b. make a connection to the user wearable device, and make a direct connection to the user wearable device, if the workstation of the remote expert and the user wearable device are in the local segment of the network space or connect via the Internet using VPN technology to enable the remote expert to be in the same network segment of the workstation of the remote expert and custom wearable device; c. upon successful connection to the user wearable device, the workstation of the remote expert performs authentication and verification of the access level for data exchange with the user device; d. after authentication and verification, the remote expert and the user of the user's wearable device work together, and the remote expert has the ability to control the user's wearable device: receive a video stream from the user's wearable device, receive audio information from the user's wearable device, display information on the display of the user's wearable device, correct the position of the module camera of the user wearable device, control the module of controlled laser designators to designate an object of interest to the user of the user wearable device, send audio information to the user wearable device.

The claimed technical result is also achieved due to the operation of the method for connecting at least two remote experts to a user wearable device for data exchange, comprising the steps of: a. on at least two automated workstations (AWP) of remote experts, enter parameters for connecting to a user wearable device; b. make a direct connection to the user wearable device if at least two workstations of remote experts and the user wearable device are located in the local segment of the network space or connect via the Internet using VPN technology to enable at least two remote expert workstations to be located in one network segment experts and custom wearable device; c. upon successful connection to the user wearable device, at least two workstations of remote experts, performs authentication and verification of the access level for data exchange; d. after authentication and verification, at least two remote experts and the user wearable device work together, and at the same time at least two remote experts exchange data with the user wearable device, and only one remote expert can control the user wearable device at a time, and data exchange with by the user wearable device includes: receiving a video stream from the user wearable device, receiving audio information from the user wearable device, displaying information on the display of the user wearable device, sending audio information to the user wearable device, and controlling the user wearable device includes: adjusting the position of the camera module of the user wearable device, control a module of controlled laser pointers for designating an object of interest to the user of the user's wearable device.

The claimed technical result is also achieved due to the operation of the method for connecting a remote expert to at least two user wearable devices, comprising the steps of: a. at the workstation (AWS) of the remote expert, parameters are entered for connection to the intermediate computing device, the intermediate computing device providing data exchange between the remote expert's workstation and at least two user wearable devices; b. connect the workstation of the remote expert to the intermediate computing device, and make a direct connection to the intermediate computing device, if the workstation of the remote expert and the intermediate computing device are located in the local segment of the network space or connect via the Internet using VPN technology to enable being in the same network segment of the AWS network of a remote expert and an intermediate computing device; c. after connecting the remote expert workstation to the intermediate computing device, the remote expert workstation is authenticated to exchange data with at least two user wearable devices through the intermediate computing device; d. connecting at least two user wearable devices by entering parameters for connecting to an intermediate computing device; e. after connecting at least two user wearable devices to the intermediate computing device, at least two user wearable devices are authenticated to exchange data with the remote expert's workstation through the intermediate computing device; f. after authentication, the remote expert and at least two user wearable devices work together, and at least two user wearable devices are controlled from the workstation of the remote expert: they receive a video stream from at least two user wearable devices, receive audio information from at least two user wearable devices wearable devices, display information on the display of at least two user wearable devices, correct the position of the camera module of at least two user wearable devices, control the module of controlled laser designators to designate an object of interest to the user of the user wearable device, send audio information to at least two user wearable devices devices.

The claimed technical result is also achieved due to the operation of the method for connecting at least two remote experts to at least two user wearable devices in a group, comprising the steps of: a. on at least two automated workstations (AWS) of remote experts, enter parameters for connecting to an intermediate computing device, and the intermediate computing device provides data exchange between at least two workstations of remote experts and at least two user wearable devices; b. connect at least two workstations of remote experts to the intermediate computing device, and make a direct connection to the intermediate computing device, if at least two workstations of remote experts and the intermediate computing device are located in the local segment of the network space or connect via the Internet using the technology VPN for realizing the possibility of being in one network segment of the network at least two workstations of remote experts and an intermediate computing device; c. after connecting at least two remote expert workstations to the intermediate computing device, at least two remote expert workstations are authenticated to exchange data with at least two user wearable devices through the intermediate computing device; d. connecting at least two user wearable devices by entering parameters for connecting to an intermediate computing device; e. after connecting at least two user wearable devices to the intermediate computing device, at least two user wearable devices are authenticated to exchange data with at least two workstations of remote experts through the intermediate computing device; f. after authentication, at least two remote experts and at least two user wearable devices work together in a group, moreover, at least two remote experts simultaneously exchange data with at least two user wearable devices, and control at least two user wearable devices Only one remote expert can use devices at a time, and data exchange with at least two user wearable devices includes: receiving a video stream from at least two user wearable devices, receiving audio information from at least two user wearable devices, displaying information on the display of at least two user wearable devices, sending audio information to at least two user wearable devices, and controlling at least two user wearable devices includes: correcting the position of the camera module of at least two user wearable devices, controlling the controllable laser pointer module to designate an object of interest at least at least two users of the user's wearable device.

DESCRIPTION OF THE DRAWINGS

The implementation of the invention will be described hereinafter in accordance with the accompanying drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

Fig. 1 illustrates the general layout of a user's wearable device.

Fig. 2 illustrates a flowchart of a user wearable device algorithm.

Fig. 3 illustrates the scheme for connecting a remote expert's workstation to a user's wearable device. Fig. 4 illustrates a diagram of connecting at least two workstations of remote experts to a user wearable device.

Fig. 5 illustrates the scheme for connecting a remote expert workstation to at least two user insect devices.

Fig. 6 illustrates a diagram of connecting at least two remote expert workstations to at least two user wearable devices in a group.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the implementation of the invention, numerous implementation details are provided to provide a clear understanding of the present invention. However, one skilled in the art will appreciate how the present invention can be used, both with and without these implementation details. In other cases, well-known methods, procedures and components have not been described in detail so as not to obscure the features of the present invention.

Furthermore, it will be clear from the foregoing that the invention is not limited to the present implementation. Numerous possible modifications, changes, variations and substitutions that retain the spirit and form of the present invention will be apparent to those skilled in the subject area.

The proposed solution is a comprehensive system for remote expert support of field specialists through the use of mixed reality technology (Industrial AR Assistant). or means of personal protection of the user through the use of fasteners, clips, clothespins for fixing the modules of the user wearable device.

The design of the proposed user wearable device is modular and can be supplemented or reduced at any time at the request of the user, for example, when in an area with confidential data, camera or sound recording modules can be removed, which in turn will not affect the operation of the device and the remaining modules, while the functions associated with the remote modules will not be available (the microcomputer will turn them off, as it will not find a connection with these modules), the user can enter questions for consultation through the human-machine interface module, and the answers of the remote expert will be played back through the audio playback module .

To ensure the protection of the hardware, all modules are made in shockproof cases, including standards from IP65 to IP68. This system reduces the cost of commissioning, maintenance and support for field specialists, including reducing troubleshooting time at the enterprise.

Figure 1 illustrates the general scheme of the user's wearable device, indicating the relationship of each module.

The microcomputer contains such components as: one or more processors, at least one memory, data storage facility, input/output interfaces, I/O facility, network communication facilities.

The device processor performs the basic computing operations necessary for the operation of the device or the functionality of one or more of its components. The processor executes the necessary machine-readable instructions contained in the main memory.

Memory, as a rule, is made in the form of RAM and contains the necessary software logic that provides the required functionality.

The data storage facility can be implemented in the form of HDD, SSD disks, raid array, network storage, flash memory, optical storage media (CD, DVD, MD, Blue-Ray disks), etc. The data storage means allows long-term storage of various types of information, for example, the above-mentioned files with user data sets, a database containing records of time intervals measured for each user, user identifiers, etc.

Interfaces are standard means for connecting and working with the server part, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS/2, Lightning, FireWire, etc.

The choice of interfaces depends on the specific implementation of the device (300), which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, and the like.

As means of I/O data in any embodiment of the system that implements the described method, a keyboard can be used. The keyboard hardware can be any known: it can be either a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server, or other computer device. In this case, the connection can be either wired, in which the keyboard connection cable is connected to the PS / 2 or USB port located on the system unit of the desktop computer, or wireless, in which the keyboard exchanges data via a wireless communication channel, for example, a radio channel, with base station, which, in turn, is directly connected to the system unit, for example, to one of the USB ports. In addition to the keyboard, data I/O can also use: joystick, display (touch screen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

Network interaction means are selected from a device that provides network data reception and transmission, for example, an Ethernet card, WLAN / Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. With the help of tools, the organization of data exchange over a wired or wireless data transmission channel, for example, WAN, PAN, LAN (LAN), Intranet, Internet, WLAN, WMAN or GSM, is provided.

Modules of a user wearable device are connected to the microcomputer via network bus interfaces of the analog-discrete input-output groups, and locking connectors are also provided to ensure the reliability of connecting the modules. In the microcomputer, the processor executes instructions stored in the storage means for remote assistance by a remote expert. The microcomputer is configured to determine the connection of modules of the user wearable device, by means of instructions, namely, when the module is connected or disconnected, the control program will automatically unlock or block the functions. The following modules are connected to the microcomputer.

1. The network interaction module is a network interface (adapter) for connecting a modular user wearable device to network segments of the Internet and / or Intranet using network data transfer protocols, it is possible to work with the following communication standards: GSM, 2G, 3G, 4G LTE , pLTE, WLAN, as well as through this module, connect the remote expert's workstation to the user's wearable device. This module is configured to provide stable and wireless communication for connecting a remote expert to a user wearable device and connecting a user wearable device to external network resources. That is, a field specialist using a user's wearable device can connect from it to external network resources (for example, the Internet).

2. Wearable display capable of displaying information. The wearable display is a combined optical-hardware complex equipped with a high-brightness, high-resolution color display module and an optical image conversion system. For example, it can be made in the form of a projector-emitter (display module), a lens system and a holographic reflector (it is translucent), or a display and a lens system (it is not translucent). Through the use of an optical conversion system through lens systems, the wearable display can be used in close proximity from the user's eye, which in turn allows the user to view without difficulty any information projected on the wearable display. The wearable display can be mounted on a mount with the ability to adjust the position along three axes (longitudinal movement, vertical and horizontal positions).

3. The camera module can be made as a high-resolution monocular camera (camera with one lens and a matrix) or a stereoscopic camera. The camera module captures the working area of the user's view of the user's wearable device and / or captures a panoramic image for video streaming to a remote expert, and can also scan a QR code, bar code, etc. to obtain equipment parameters and settings that will be displayed on the wearable display. It also provides the ability to read a QR code to connect to a user wearable device to an intermediate computing device.

4. The module of the controlled stabilizer of the camera module is an automatic electromechanical 2x or 3x-axis stabilizer of the camera "auto-horizon" with the possibility of constant correction of the zero position of the center of the image of the camera's video stream (horizon). For example, but not limited to, it can be made in the form of an electromechanical 3x-axis holder-stay of the camera, equipped with brushless electric motors and its own controller for continuous correction of the camera’s position “in the horizon” based on the own position of the controller of the controlled camera stabilizer module, as well as an interface for controlling by setting the zero position of the “horizon” or in the form of a software (digital) video stream stabilizer, where due to the increased number of pixels of the camera matrix during rattling and a sharp shift in the camera position, the camera’s own microprocessor activates additional pixels in antiphase to the movement.

The controllable stabilizer module of the camera module performs stabilization of the shooting frame so that the remote expert receives a clear video stream, even when the user of the user's wearable device moves his head. Also, the remote expert has the ability to change the position of the camera module through the camera module's controlled stabilizer module, that is, the remote expert has the ability to change the direction of the camera module's view through the camera module's controlled stabilizer module, this change in position is carried out by executing instructions on the microcomputer and transmitting a signal to the controller of the controlled camera module. camera module stabilizer. When the head of the user of the modular custom wearable device rotates by a certain angle, the controller of the camera module's gimbal control module uses the user's head position data (camera module's own controllable gimbal module position) and calculates the compensation rotation angle of the controllable gimbal module of the camera module, namely the compensation angle is aligned in antiphase with the user's head deflection angle, which in turn allows the camera's controlled stabilizer module to be located on the point that the remote expert pointed.

5. The adaptive lighting module is a device for illuminating the working area of the user's view of the user's wearable device and can be made in the form of LED lighting sources and/or infrared lighting sources, and infrared lighting sources can be integrated into the camera module. This module illuminates the working area of the user's view of the user's wearable device and / or provides the necessary exposure of the camera module, when the adaptive device module of the work area is illuminated, the camera module will automatically adjust the desired exposure.

6. The module of controlled laser designators is an optical laser complex for indicating a target (an object of interest of a remote expert). For example, but not limited to, the optical laser target designator complex can be made from a controlled optical-mechanical device for correcting the direction of the laser beam (XY galvanometric scanner) and a laser light source, additionally equipped with a laser rangefinder to create a correction factor, or it can be made from a laser light source and optical nozzle for changing the shape of the laser beam. This module is mechanically connected to the camera stabilization module for the convenience of remote control of the laser designator.

This module is configured to select an object with a laser beam located in the user's work area of view of the user's wearable device, by a remote expert to attract the attention of the user of the wearable device, which in turn is an element of mixed reality and allows the user of the wearable device to focus their attention on the object without wasting time interest of a remote expert. The selection of an object by a laser beam (laser illumination) can be performed in the form of a laser point indicating an object of interest or a selection by a group of points along the perimeter of an object of interest.

7. The sound reproduction module is a sound reproduction system and can be made in the form of a mono-acoustic system (one speaker) or a stereo-acoustic system (two speakers). The sound reproduction module can be made in the form of headphones (stereo or mono) and/or in the form of active external speakers (stereo or mono). This module reproduces the speech of a remote expert (for example, but not limited to receiving advice and commands from a remote expert), through the network interaction module, as well as playing audio signals about ongoing events of the user wearable device interface and receiving notification signals from external devices (if connected wearable device to external devices). 8. The sound recording module can be made in the form of an electromagnetic microphone, a condenser microphone, a laryngphone, a piezo microphone. The sound recording module records the user's speech and sounds around the user's wearable device, and transmits them, through the network interaction module on the workstation, to a remote expert.

9. The human-machine interface module is a module with a set of buttons and / or joysticks and / or touch panels and / or gesture detectors and light indicators of the device operation. This module performs user interaction with the visual interface of the user wearable device, namely, enter data and settings for the control program of the wearable device, as well as control system functions, for example, turn the device on or off.

10. Portable battery module, to provide electrical power to the user's wearable device.

Additionally, a user wearable device may contain a network interface group module, which is a hub of free network interface connectors for connecting additional devices via USB, RS232, RJ45, LPT, COM, HDMI, PS/2, Lightning, FireWire, etc. protocols.

To ensure the operation of the user wearable device modules, the microcomputer memory stores control instructions executable by the microcomputer processor and includes the following operation steps with reference to FIG. 3.

The microcomputer can run any operating system known in the art.

The control instructions include the main algorithm, which is a basic loop that calls the necessary subroutines. The main algorithm includes the following subroutines: a subroutine for interacting with a graphical user interface (GUI), a subroutine for interacting with modules of a user wearable device, a subroutine for working with network functions, a subroutine for interacting with physical signals, auxiliary subroutines, and the main algorithm is also connected with the service functions of a microcomputer .

The subroutine for interaction with the graphical interface (GUI) allows you to implement scenarios of interaction between the user of the user's wearable device and the program's graphical interface (for example, navigating through the graphical interface and/or mini-chat with a remote expert, and others).

The subroutine for interaction with the modules of the user wearable device allows you to structure and process the necessary variables or tags, parameters, settings for further transfer to the modules of the modular user wearable device. The subroutine for working with network functions allows you to initiate data transfer to the modules of the user wearable device and the network exchange module (network interaction module), as well as to get the control program access to the microcomputer utility functions (for example, activating network adapters, changing network adapter parameters and other functions) .

The subroutine for interaction with physical signals initiates work with service functions for accessing the hardware resources of the microcomputer (for example, input and output contact groups).

Auxiliary routines allow you to control the "integrity" parameters of a wearable device (for example, when you connect or disconnect a module of a user wearable device, the control program automatically unlocks or locks functions), and auxiliary routines are also needed to run custom (additional) scenarios for the user wearable device (for example , beeping when a remote expert is connected, or displaying information about the status of a wearable user device).

Service functions are standard functions for providing access of control instructions to the resources of the microcomputer and security functions (for example, commands for accessing network adapters, contact group of input/output signals, data buses, and other functions).

Next will be given examples of the placement of modules of the user's wearable device. One skilled in the art will appreciate that this arrangement is an example and the modules may be arranged differently.

On the helmet, which is a means of personal protection, the following modules are placed: a microcomputer placed on a special mount on the side of the helmet; a display placed on a swivel mount next to the microcomputer; a camera module, a camera module controlled stabilizer module, a controlled laser pointer module, are in conjunction with each other and are located on the central part of the front side of the helmet; the adaptive lighting module is located on the side parts of the helmet (left and right), the sound reproduction module is made in the form of wireless or wired headphones and is located in the user's auricles, the sound recording module is made in the form of a lavalier microphone and is attached to a clip on the user's clothing; the portable battery module is connected to the microcomputer by wire and is located in the pocket of the user's clothing or on the waist belt of the user's clothing; the human-machine interface module is located on the side of the helmet, next to the microcomputer.

In another option for placing the modules of a user wearable device, instead of a helmet, an elastic head strap is used, on which is placed: a microcomputer placed on a special mount on the side of the elastic head belt, display placed on a swivel mount, next to the microcomputer; the camera module, the controlled stabilizer module of the camera module, the controlled laser pointer module, are in conjunction with each other and are located on the central part of the front side of the elastic head strap; the adaptive lighting module is located on the side parts of the elastic head strap (left and right), the sound reproduction module is made in the form of wired or wireless headphones and is located in the user's ears, the sound recording module is made in the form of a lavalier microphone and is clipped onto the user's clothing; the portable battery module is connected to the microcomputer by wire and is located in the pocket of the user's clothing or on the waist belt of the user's clothing; the human-machine interface module is located on the microcomputer.

The remote expert connects to the user's wearable device through the workstation (AWP) of the remote expert. When connecting the workstation of a remote expert and a modular user wearable device must be in the same network segment of the network.

The remote expert workstation is a computing device or a mobile computing device with pre-installed software for connecting to a remote node of a modular user wearable device (for example, a web browser or a remote desktop client, and so on).

The remote expert's computer workstation contains such components as: one or more processors, at least one memory, data storage facility, input/output interfaces, I/O facility, network interaction facilities.

Interfaces are standard means for connecting and working with the server part, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS/2, Lightning, FireWire, etc. The choice of interfaces depends on the specific version of the device, which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, etc.

As means of I/O data in any embodiment of the system that implements the described method, a keyboard can be used. The keyboard hardware can be any known: it can be either a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server, or other computer device. In this case, the connection can be either wired, in which the keyboard connection cable is connected to the PS / 2 or USB port located on the system unit of the desktop computer, or wireless, in which the keyboard exchanges data via a wireless communication channel, for example, a radio channel, with base station, which, in turn, is directly connected to the system unit, for example, to one of the USB ports. In addition to the keyboard, the following I/O devices can also be used: joystick, display (touchscreen), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

The proposed solution also provides for a multi-user mode of operation, where several users of user wearable devices and / or several workstations of remote experts take part in the connection at once within one session of the proposed solution. To implement this mode, an intermediate computing device (PC Host) is used.

The work session is understood as the period during which the remote expert provides support to the user of the user's wearable device.

The intermediate computing device can be represented as a server computing device, a tablet computing device, an industrial computing device. The intermediate computing device acts as a platform for launching a data exchange application with two-way authentication (authentication from the remote expert's workstation and authentication from users of modular user wearable devices).

For example, when connecting several modular user wearable devices to an intermediate computing device and one workstation of a remote expert, all data, content, information exchange is carried out through software data exchange of an intermediate computing device. Data can be video/audio, mini-chat, settings for camera stabilizer, laser designators and other peripheral modules of user wearable devices.

The intermediate computing device resides in the same network space as the user's wearable devices and/or allows direct communication with the user's wearable devices.

The proposed solution has several options for connecting a remote expert workstation to a user wearable device:

1. Operation mode, where one workstation of a remote expert is connected to one user device.

2. Multi-user mode of operation, where at least two workstations of remote experts are connected to one user device and provides the simultaneous presence of experts from different areas within one consultation.

3. Multi-user mode of operation, where one workstation of a remote expert is connected to at least two user wearable devices and provides simultaneous consultation of several users of user wearable devices solving one task.

4. Combined mode of operation, where at least two workstations of a remote expert are connected to at least two modular user wearable devices and work is carried out in a group (Combines multi-user mode 2 and multi-user mode 3).

Next, we will discuss each mode of operation with reference to FIGS. 3-6.

In FIG. Figure 3 illustrates the connection diagram of one workstation of a remote expert to one modular user wearable device.

At the workstation of the remote expert, the request for remote connection to the user's wearable device is initialized via the web interface, where the remote expert enters the necessary parameters for connecting the user's wearable device.

To connect to the user wearable device, the remote expert, via the remote expert workstation, enters parameters for connection, for example, the network address of the modular user wearable device and the port.

The entered data passes through the firewall to provide protection against unauthorized access.

A workstation of a remote expert is connected to a modular user wearable device. When working in the local segment of the network space, a direct connection is made to a modular user wearable device. When working on the Internet, they connect using VPN technology to be able to be in one network segment of a workstation of a remote expert and a modular user wearable device. For example, there may be a corporate network segment.

Once connected, the data passes through the firewall of the user's wearable device.

Upon successful connection of the remote expert workstation to the user wearable device, the access level of the remote expert workstation is authenticated and verified to exchange data by login and password, token or unique serial number of the user wearable device.

After authentication and verification, the remote expert and the user of the user's wearable device work together. As part of the joint work, a remote expert, through the remote expert's workstation, has the ability to control a modular user wearable device, namely: receive a video stream from a user wearable device, receive audio information from a user wearable device, display information on the display of a user wearable device, correct the position of the camera module of the user wearable device, control the module of controlled laser designators to designate an object of interest to the user of the user's wearable device, send audio information to the user's wearable device.

Figure 4 illustrates a diagram of a multi-user connection of at least two workstations of remote experts to one user wearable device.

On one workstation of a remote expert, a request for a remote connection to a modular user wearable device is initialized. This workstation of the remote expert creates a connection session. After the session is created, sequentially, other workstations of a remote expert can be connected, the scheme for connecting the workstations of remote experts is the same and is described below.

To connect to the user wearable device, at least two remote experts enter connection parameters via at least two remote expert workstations, for example, the network address of the modular user wearable device and the port.

At least two workstations of remote experts are connected to the user's wearable device. When working in the local segment of the network space, a direct connection is made to the user's wearable device. When working on the Internet, a connection is made using VPN technology to enable being in one network segment at least two workstations of remote experts and a user wearable device. For example, there may be a corporate network segment.

Upon successful connection of at least two workstations of remote experts to the user wearable device, authentication and verification of the access level of at least two workstations of remote experts is carried out to exchange data by login and password, token or unique serial number of the user wearable device.

After authentication and verification, at least two remote experts and a user of the user's wearable device work together. Simultaneously, at least two workstations of remote experts exchange data (receiving audio / video from a modular user wearable device, transferring content to the display of a modular user wearable device, transmitting audio to a user wearable device) with the user wearable device, and control the user wearable device (control controllable camera stabilizer module for correcting the position of the camera module, controllable laser designator module) at a time can only be one workstation of a remote expert.

If necessary, control of the user's wearable device may transfer the remote expert's workstation 1 to the remote expert's workstation N and vice versa.

Figure 5 illustrates a diagram of a multi-user connection of a remote expert workstation to at least two user wearable devices.

At the workstation of a remote expert, a request for a remote connection to an intermediate computing device is initialized.

To connect to the intermediate computing device of a remote expert by means of the workstation of the remote expert, parameters for connection are entered, for example, the network address of the intermediate computing device and the port.

The remote expert workstation is connected to an intermediate computing device. When working in the local segment of the network space, a direct connection is made to an intermediate computing device. When working on the Internet, a connection is made using VPN technology to make it possible for a remote expert and an intermediate computing device to be in the same network segment of the workstation. For example, there may be a corporate network segment.

Once connected, the data passes through the firewall of the intermediate computing device. After successful connection of the remote expert's workstation to the intermediate computing device, the remote expert's workstation performs authentication in the data exchange software of the intermediate computing device.

After successful authentication of the remote expert workstation, at least two modular user wearable devices are expected to connect.

Users of the modular user wearable device, through the HMI module using the virtual keyboard and joystick or buttons of the HMI module (the user moves the joystick or presses the buttons, and enters on the virtual keyboard), enters parameters to connect to the intermediate computing device, (for example, the network address of the intermediate computing device and the port. Also, the parameters of the intermediate computing device can be entered by reading the QR code containing the necessary data for connection and/or authentication by the camera module).

Since the intermediate computing device and at least two modular user wearable devices are in the same network segment (eg corporate network segment), a direct connection (address input) is performed.

Once connected, the data passes through the firewall of the intermediate computing device.

Upon successful connection of the at least two modular user wearable devices to the intermediate computing device, the at least two user wearable devices authenticate to the communication software of the intermediate computing device. Authentication can be carried out using the same means that were used to enter the parameters of the intermediate computing device. Authentication can be carried out by login and password, token or unique serial number of the user wearable device.

When all authentication steps are completed, the remote expert workstation and at least two user wearable devices work together, and at least two user wearable devices are controlled from the remote expert workstation: they receive a video stream from at least two user wearable devices, receive audio information from at least two user wearable devices, display information on the display of at least two user wearable devices, correct the position of the camera module of at least two user wearable devices, control the module of controlled laser designators to designate an object of interest to the user of the user wearable device, send audio information to the at least two user wearable devices. In this multi-user mode of operation, a single remote expert can maintain and control at least two modular user wearable devices by communicating through an intermediate computing device.

Figure 6 illustrates a diagram of a combined connection of at least two workstations of remote experts to at least two user wearable devices in a group mode.

On one workstation of a remote expert, a request for a remote connection to an intermediate computing device is initialized. This workstation of the remote expert creates a connection session. After the session is created, sequentially, other workstations of remote experts can be connected, the scheme for connecting the workstations of remote experts is the same and is described below.

To connect to the intermediate computing device, at least two remote experts enter connection parameters via at least two remote expert workstations, for example, the network address of the intermediate computing device and the port.

At least two workstations of remote experts are connected to an intermediate computing device. When working in the local segment of the network space, a direct connection is made to an intermediate computing device. When working on the Internet, a connection is made using VPN technology to enable at least two workstations of remote experts and an intermediate computing device to be located in one network segment. For example, there may be a corporate network segment.

After successfully connecting at least two remote expert workstations to the intermediate computing device, at least two remote expert workstations perform authentication in the communication software of the intermediate computing device. Authentication can be carried out by login and password, token.

After successful authentication of at least two remote expert workstations, at least two user wearable devices are expected to connect.

Users of the modular user wearable device, through the HMI module using the virtual keyboard and joystick or buttons of the HMI module (the user moves the joystick or presses the buttons, and enters on the virtual keyboard), enters parameters to connect to the intermediate computing device, (for example, the network address of the intermediate computing device and the port. Also, entering parameters intermediate computing device can be carried out by reading a QR code containing the necessary data for connection and / or authentication by the camera module).

Since the intermediate computing device and at least two user wearable devices are on the same network segment (eg, a corporate network segment), a direct connection is made.

Upon successful connection of the at least two user wearable devices to the intermediate computing device, the at least two user wearable devices authenticate to the communication software of the intermediate computing device. Authentication can be carried out using the same means that were used to enter the parameters of the intermediate computing device. Authentication can be carried out by login and password, token or unique serial number of the user wearable device.

When all authentication steps have been completed, at least two remote expert workstations and at least two user wearable devices work together as a group. At least two remote experts communicate with at least two user wearable devices at a time, and only one remote expert can control at least two user wearable devices at a time. Communication with at least two user wearable devices includes: receiving a video stream from at least two user wearable devices, receiving audio information from at least two user wearable devices, displaying information on the display of at least two user wearable devices, sending audio information to at least two user wearable devices, and control of at least two user wearable devices includes: correcting the position of the camera module of at least two user wearable devices, controlling the module of controlled laser designators to designate an object of interest for at least two users of the user wearable device.

Claims

Formula

1. A system for supporting a user by a remote expert, including an automated workstation (AWS) of a remote expert and a user wearable device, wherein the user wearable device is a modular design, with the ability to place the modules on the user's body and/or elements of clothing and/or personal protective equipment of the user each module is made in a shock-resistant housing and includes: a microcomputer to which modules of the user wearable device are connected, configured to determine the connected modules of the user wearable device; a network interaction module configured to provide stable and wireless communication for connecting the remote expert workstation to the user wearable device and connecting the user wearable device to external network resources; a display configured to display information; a camera module configured to capture a user's wearable device's work view area; a controlled stabilizer module of the camera module, configured to stabilize the shooting frame, as well as change the position of the camera module by a remote expert; an adaptive lighting module configured to illuminate the user's wearable device user's work view area and/or to provide the required exposure of the camera module; a controlled laser designator module, configured to highlight an object with a laser beam by a remote expert, and the object is in the working area of the user's view of the user's wearable device; an audio playback module, configured to reproduce the speech of the remote expert through networking tools, as well as audio signals about ongoing events of the interface of the user wearable device; an audio recording module, configured to record the user's speech and sounds around the user's wearable device, and transmit, via networking means, to the remote expert's workstation; a human-machine interface module configured to interact with a user and a user wearable device.

2. User wearable device according to claim 1, characterized in that the module housings have protection standards from IP65 to IP68.

24

3. The user wearable device according to claim 1, characterized in that it further comprises a portable stand-alone battery module configured to provide electrical power to the user wearable device.

4. The user wearable device according to claim 1, characterized in that it additionally contains a network interface group module, which is a hub of free network interface connectors.

5. The user wearable device according to claim 1, characterized in that the camera module is a high-resolution monocular camera or a high-resolution stereo camera.

6. User wearable device according to claim 1, characterized in that the controllable stabilizer module of the camera module is an electromechanical three-axis holder-stay of the camera module.

7. The user wearable device according to claim 1, characterized in that the controllable stabilizer module of the camera module is a digital stabilizer of the shooting frame.

8. User wearable device according to claim 1, characterized in that the module of controlled laser designators is an optical laser targeting complex.

9. User wearable device according to claim 8, characterized in that the optical laser target indication complex is a controlled device for correcting the direction of the laser beam, a laser light source and a laser range finder.

10. User wearable device according to claim 8, characterized in that the optical laser target indication complex is a laser light source, an optical attachment for changing the shape of the laser beam and a laser range finder.

11. The user wearable device according to claim 1, characterized in that the adaptive lighting module is an LED light source and/or at least one IR light source and/or an IR light source integrated into the camera module.

12. User wearable device according to claim 1, characterized in that the sound reproduction module is made in the form of a mono acoustic system or a stereo acoustic system.

13. The user wearable device according to claim 12, characterized in that the sound reproduction module is made in the form of external speakers located on the body of the user wearable device.

14. User wearable device according to claim 12, characterized in that the sound reproduction module is made in the form of plug-in headphones.

15. User wearable device according to claim 1, characterized in that the sound recording module can be made in the form of an electromagnetic microphone or a condenser microphone, or a laryngphone, or a piezo microphone.

16. User wearable device according to claim 1, characterized in that the human-machine interface module is a module with a set of buttons and/or joysticks and/or touch pads and/or gesture detectors and light indicators of device operation.

17. A method for connecting a remote expert to a user wearable device for data exchange, comprising the steps of: a. at the automated workstation (AWP) of the remote expert, parameters are entered for connecting to the user's wearable device; b. make a connection to the user wearable device, and make a direct connection to the user wearable device, if the workstation of the remote expert and the user wearable device are in the local segment of the network space or connect via the Internet using VPN technology to enable the remote expert to be in the same network segment of the workstation of the remote expert and custom wearable device; c. upon successful connection to the user wearable device, the workstation of the remote expert performs authentication and verification of the access level for data exchange with the user device; d. after authentication and verification, the remote expert and the user of the user's wearable device work together, and the remote expert has the ability to control the user's wearable device: receive a video stream from the user's wearable device, receive audio information from the user's wearable device, display information on the display of the user's wearable device, correct the position of the module camera of the user wearable device, control the module of controlled laser designators to designate an object of interest to the user of the user wearable device, send audio information to the user wearable device.

18. A method for connecting at least two remote experts to a user wearable device for data exchange, comprising the steps of: a. on at least two automated workstations (AWP) of remote experts, enter parameters for connecting to a user wearable device; b. make a direct connection to the user wearable device if at least two workstations of remote experts and the user wearable device are located in the local segment of the network space or connect via the Internet using VPN technology to enable at least two remote expert workstations to be located in one network segment experts and custom wearable device; c. upon successful connection to the user wearable device, at least two workstations of remote experts, performs authentication and verification of the access level for data exchange; d. after authentication and verification, at least two remote experts and the user wearable device work together, and at the same time at least two remote experts exchange data with the user wearable device, and only one remote expert can control the user wearable device at a time, and data exchange with by the user wearable device includes: receiving a video stream from the user wearable device, receiving audio information from the user wearable device, displaying information on the display of the user wearable device, sending audio information to the user wearable device, and controlling the user wearable device includes: adjusting the position of the camera module of the user wearable device, control a module of controlled laser pointers for designating an object of interest to the user of the user's wearable device.

19. A method for connecting a remote expert to at least two user wearable devices, comprising the steps of: a. at the workstation (AWS) of the remote expert, parameters are entered for connection to the intermediate computing device, the intermediate computing device providing data exchange between the remote expert's workstation and at least two user wearable devices; b. connect the workstation of the remote expert to the intermediate computing device, and make a direct connection to the intermediate computing device, if the workstation of the remote expert and the intermediate computing device are located in the local segment of the network space or connect via the Internet using

27 VPN technologies for realizing the possibility of being in the same network segment of the AWS network of a remote expert and an intermediate computing device; c. after connecting the remote expert workstation to the intermediate computing device, the remote expert workstation is authenticated to exchange data with at least two user wearable devices through the intermediate computing device; d. connecting at least two user wearable devices by entering parameters for connecting to an intermediate computing device; e. after connecting at least two user wearable devices to the intermediate computing device, at least two user wearable devices are authenticated to exchange data with the remote expert's workstation through the intermediate computing device; f. after authentication, the remote expert and at least two user wearable devices work together, and at least two user wearable devices are controlled from the workstation of the remote expert: they receive a video stream from at least two user wearable devices, receive audio information from at least two user wearable devices wearable devices, display information on the display of at least two user wearable devices, correct the position of the camera module of at least two user wearable devices, control the module of controlled laser designators to designate an object of interest to the user of the user wearable device, send audio information to at least two user wearable devices devices.

20. A method for connecting at least two remote experts to at least two user wearable devices in a group, comprising the steps of: a. on at least two automated workstations (AWS) of remote experts, enter parameters for connecting to an intermediate computing device, and the intermediate computing device provides data exchange between at least two workstations of remote experts and at least two user wearable devices; b. connect at least two workstations of remote experts to the intermediate computing device, and make a direct connection to the intermediate computing device, if at least two workstations of remote experts and the intermediate computing device are located in the local segment of the network space or connect through the network

28 the Internet using VPN technology to implement the possibility of being in one network segment of the network at least two workstations of remote experts and an intermediate computing device; c. after connecting at least two remote expert workstations to the intermediate computing device, at least two remote expert workstations are authenticated to exchange data with at least two user wearable devices through the intermediate computing device; d. connecting at least two user wearable devices by entering parameters for connecting to an intermediate computing device; e. after connecting at least two user wearable devices to the intermediate computing device, at least two user wearable devices are authenticated to exchange data with at least two workstations of remote experts through the intermediate computing device; f. after authentication, at least two remote experts and at least two user wearable devices work together in a group, moreover, at least two remote experts simultaneously exchange data with at least two user wearable devices, and control at least two user wearable devices Only one remote expert can use devices at a time, and data exchange with at least two user wearable devices includes: receiving a video stream from at least two user wearable devices, receiving audio information from at least two user wearable devices, displaying information on the display of at least two user wearable devices, sending audio information to at least two user wearable devices, and controlling at least two user wearable devices includes: correcting the position of the camera module of at least two user wearable devices, controlling the controllable laser pointer module to designate an object of interest at least at least two users of the user's wearable device.