KR102331172B1 - System for remote service and supporting of maintenance-repairment based in cloud employing smart glass - Google Patents

Abstract

The embodiment relates to a remote service and maintenance support solution using smart glasses.
Specifically, the solution exchanges field work-related data from the remote control center and the cloud virtual server that provides virtual web service by the cloud through smart glasses to high-speed web RTC by the API of the corresponding browser and mobile application.
In this case, the API includes video, audio, and information exchange protocols including codecs, network information exchange, and WebRTC peer connection.
Therefore, it is characterized in that the remote support function of the high-speed web RTC communication method is performed when remote service and maintenance are remotely supported through smart glasses.
Therefore, through this, when a worker wearing smart glasses works at a site far from the remote control center such as a management center or development center, maintenance engineers, etc. can stay in the remote control center without having to go on a business trip directly. and support

Description

Cloud-based remote service and supporting of maintenance-repairment based in cloud employing smart glass}

The content disclosed in this specification relates to a solution that supports maintenance by providing field work-related data through smart glasses in conjunction with a remote control center when a worker wearing smart glasses works in a field such as a logistics warehouse will be.

Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application, and inclusion in this section is not an admission that it is prior art.

Recently, as the center of the smart IT industry moves to wearable devices, it is a situation that is changing the market from smart watches to smart glasses as auxiliary devices worn by workers when working in industrial fields. In addition, since the growth of the content market is expected to accelerate rather than the supply of hardware in the future, the need for content development using smart glasses is increasing along with the spread of smart glasses.

Among these smart glasses, in particular, the hands-free type smart glasses removes many restrictions in doing various tasks because it is easy to transmit and work with data such as work instructions while using both hands freely. This can change the working method and increase the productivity by improving the machine operation time, and it can improve the accuracy of the work because the work is performed while checking in real time.

On the other hand, in order to support work, abnormalities, and maintenance at worksites that are dispersed at home and abroad or in remote locations, it is necessary to visit the site through business trips to solve the problem. Currently, there is a solution that supports photos or voices using a smartphone, but in the actual field, you can receive prompt and accurate support only if you receive support while doing work or maintenance. This can improve the availability of high-level personnel of a company or organization and reduce travel expenses.

In addition, recently, manufacturing-related companies are required to reduce travel costs, save work time, respond quickly, and improve work accuracy and work efficiency for on-site emergency situations or emergency support, failures, or training for new and unskilled workers.

Therefore, when a person visits the site for maintenance or the like or wants to meet these needs, it can be solved by developing a system according to the above-mentioned hands-free smart glasses.

When examining the related prior art as the background for this background, the following patent documents, which are worthy of reference as related prior art, have been found by examining the prior art from the patent document, and the number of related cases has been investigated in general.

(Patent Document 1) KR1020190016015 A

(Patent Document 2) KR1020190016016 A

For reference, the technology of these patent documents is to the extent of recognizing the QR code attached to the product through the smart glasses worn by the worker in the warehouse of the logistics center and transmitting it to the warehouse management system of the logistics center to provide warehouse management and return management. .

Disclosed content is to provide a cloud-based remote service and maintenance support solution using smart glasses that enables workers wearing smart glasses to exchange data in real time through smart glasses when working in the field.

In addition, we intend to provide a cloud-based remote service and maintenance support solution using smart glasses that allows maintenance engineers to stay in the remote control center and provide support without having to go to the site directly.

A cloud-based remote service and maintenance support solution using smart glasses according to an embodiment,

When performing field work through smart glasses, field work related data is provided from at least one remote control center and a cloud virtual server that provides virtual web service by the cloud for each smart glass for each of a number of different field work sites.

In addition, at this time, the smart glasses and the remote control center are communicated from the browser of the common web RTC signaling server according to the cloud virtual server and the API of the mobile application program to achieve high-speed web RTC.

In this case, the API includes video, audio, and information exchange protocols including codecs, network information exchange, and WebRTC peer connection.

Therefore, it is characterized in that the remote support function of the high-speed web RTC communication method is performed when remote service and maintenance are remotely supported through smart glasses.

According to embodiments, when a worker wearing smart glasses works in the field, field work related data is exchanged in real time through the smart glasses.

Specifically, when a worker wearing smart glasses works in a remote site such as a management center or development center, the above solution provides a remote support function of high-speed web RTC communication method to provide on-site situation and immediate feedback do.

And, when a worker wearing smart glasses works at a site far away from a remote control center such as a management center, a maintenance engineer from the solution provides support while staying in the remote control center without having to go on a business trip directly. do

In addition, it brings rapid response, work accuracy and work efficiency improvement from the solution in the situation where manufacturing-related companies recently require on-site emergency situations, emergency support, failure, or training of new and unskilled workers.

1 is a schematic diagram showing the overall system to which a remote service and maintenance support system using smart glasses according to an embodiment is applied;
2 is a view showing smart glasses applied to a solution according to an embodiment;
3 is a view showing the configuration of smart glasses according to an embodiment applied to the smart glasses of FIG.
4 is a diagram for explaining a platform structure applied to a solution according to an embodiment;
5 is a diagram showing a high-speed WebRTC structure applied to a solution according to an embodiment;
6 is a flowchart sequentially illustrating a procedure of a high-speed WebRTC according to an embodiment applied to the high-speed WebRTC structure of FIG. 5;
7 is a flowchart sequentially illustrating the operation of a remote service and maintenance support solution using smart glasses according to an embodiment;

1 is a schematic diagram showing the overall system to which a cloud-based remote service and maintenance support system (hereinafter referred to as a 'solution') using smart glasses according to an embodiment is applied.

As shown in FIG. 1 , the system provides a plurality of smart glasses 100 that inform on-site information by wearing for each worker when working in the field, and provides feedback from a remote location to each smart glass 100 when notified. and a control center 200 .

At this time, data related to field work from the cloud virtual server that provides virtual web service by the remote control center and the cloud through smart glasses is converted to high-speed web RTC by the browser of the common web RTC signaling server and the API of the mobile application program. Includes exchangeable solutions.

The smart glasses 100 are to be worn by each worker when a worker is working in a field workshop such as a logistics warehouse, so that an expert in a remote control center can see the situation of the field through video and audio, etc. to provide feedback. to be provided And, at this time, these smart glasses 100 are used by different positions of the smart glasses from a number of different mutually spaced field workshops (#1, ... , #5, etc.) for each workplace.

The remote control center 200 provides remote support by providing feedback from a maintenance engineer or a support engineer to the smart glasses 100 when each field situation is notified by the smart glasses 100 .

The solution is based on the assumption that when field work related data according to the field work of a worker wearing smart glasses is input, the data is provided through the hardware processor through communication between the smart glasses and the remote control center to remotely support maintenance, etc. In this state, the solution remotely supports the provision of data from at least one remote control center through the smart glasses and a cloud virtual server that provides virtual web service by the cloud according to a common platform when performing field work of smart glasses. . In addition, at this time, the solution makes the communication from the common WebRTC signaling server according to the cloud virtual server to a high-speed WebRTC by the API of the corresponding browser and mobile application program. In other words, it implements high-speed web RTC without CPU load in browsers and mobile applications through simple APIs without the help of web browser plug-ins. So, the solution as a whole performs the remote support function of the high-speed web RTC communication method.

Therefore, the system of FIG. 1 implements a comprehensive interactive joint platform from the solution when a worker wearing smart glasses works in a field remote from a management center or development center, etc. to provide. For reference, the interactive joint platform according to this embodiment is in the form of a platform suitable for field work through smart glasses.

In addition, when a worker wearing smart glasses is working in a remote location such as a management center, a remote expert can accurately view the situation in the management center without having a maintenance engineer directly go to the site. support

In addition, the solution supports work through voice/video bidirectional transmission and augmented reality text, graphic markers, document sharing, etc., since remote experts can accurately see the situation of the field wearing smart glasses.

For example, such support includes rapid response and prevention, immediate response to failure, provision of on-site information sharing, etc., engineering information sharing, facility and production operation request, maintenance support request, IT infrastructure support request, emergency support request, etc.

FIG. 2 is a view showing smart glasses applied to a solution according to an embodiment applied to the system of FIG. 1 .

As shown in FIG. 2 , the smart glasses according to an embodiment allow a worker to work while looking at a monitor attached to the smart glasses while working in a field such as a logistics warehouse, a manufacturing plant, etc., and the user's convenience and usability provides

Specifically, the smart glasses of one embodiment, when the worker wearing the smart glasses is working, the worker sees the field information in the workplace where the worker is working, according to the movement of the worker, a real-time image of a remote control center such as a support center in a remote location provided to And, at this time, by providing voice data related to the state of the device being worked on from the voice of the worker, the job-related information is notified to the remote control center in more detail.

And, at this time, these smart glasses receive feedback from a remote maintenance center or support center far from the workplace from the aforementioned monitor in real time according to augmented reality, and provide remote guide instructions to the workers in the workplace.

FIG. 3 is a diagram showing the configuration of smart glasses according to an embodiment applied to the smart glasses of FIG. 2 .

As shown in FIG. 3 , the configuration of the smart glasses is a wireless communication module that transmits the situation of the field from wireless Wi-Fi to the remote control center or receives feedback from the remote control center when communicating with a remote control center at a remote location. (301).

Additionally, the configuration of the smart glasses of an embodiment is installed on the smart glasses when a worker works in the field, and a camera 302 for providing the on-site situation as an image to the remote control center in real time and a microphone for providing voice ( 303).

In addition, the configuration of such smart glasses is provided with an LED display 304 for guiding the workers of the workplace because the feedback is provided and displayed from a remote control center far away from the workplace when the on-site situation is provided.

Additionally, in the above configuration, when field information such as field conditions according to field work is input, various commands of the user are input, including commands for transmitting field information to the remote control center according to the user's key operation. Includes a receiving touchpad.

When communicating with a remote control center at a remote location, the wireless communication module 301 transmits the situation of the field from, for example, 2.4/5 GHz dual-band wireless Wi-Fi to the remote control center or immediate feedback from the remote control center. is to receive

The camera 302 is installed on smart glasses in the form of a 16MP camera when a worker works in the field, so that the field situation is captured in real time and provided as an image to the remote control center.

The microphone 303 receives work-related device state information and the like when the worker works in the field, converts it into a voice signal, and provides the worker's voice to the remote control center.

When the LED display 304 provides the on-site situation in this way, the feedback is provided from the remote control center far from the workplace and displayed, so that the guide instructions are delivered to the workers of the workplace.

In this case, the LED display 304 transmits the augmented reality instructions to the smart glasses through a web browser, but it is different from the user communicating using a virtual interface object (physical spatial information). Specifically, the LED display interfaces the augmented reality through the wearable smart glass itself, not the virtual, when the augmented reality instructions are delivered in this way.

In more detail, when the feedback is provided by the high-speed web RTC, the LED display 304 interfaces with the augmented reality for delivering feedback from the device itself of the smart glasses to expand the remote support function of the high-speed web RTC communication method. do.

And, when the augmented reality is interfaced in this way, the LED display 304 strengthens the augmented reality function because it tracks and shows the marking at a designated location while actually moving the marking from object shape recognition and feature-based position recognition.

In addition, to explain more, the current augmented reality delivery function is to deliver the graphic drawing by the absolute coordinates to the smart glasses when the graphic marker is performed.

Accordingly, in order to strengthen the function of augmented reality, augmented reality is implemented so that the mark is tracked and shown at the designated position while actually moving by recognizing the characteristics and location of the target object.

4 is a diagram for explaining a common platform structure applied to a solution according to an embodiment.

As shown in FIG. 4 , the platform shares field work related data from the cloud virtual server that provides virtual web service as a cloud-based virtual web service as a whole client with smart glasses using the remote control center as a web agent when remote support is provided.

The cloud virtual server forms a common platform such as a remote support function by performing a virtual web service based on the cloud with one or more floating smart glasses and all clients using one or more remote control centers as a web agent when remote support is provided. Incidentally, this joint platform is a joint platform with a function to maximize the user's convenience and usability from the above-mentioned remote support when a worker wearing smart glasses actually works in the field.

Additionally, the remote control center is, for example, an engineering support center, a field engineer shift manager, a facility operator maintenance engineer, a support engineer of a facility supplier, ERP/CRM/FMS/MES, industrial IoT, and the like.

5 and 6 are diagrams for explaining a high-speed WebRTC applied to a solution according to an embodiment.

Specifically, FIG. 5 is a diagram showing a high-speed WebRTC structure according to an embodiment, and FIG. 6 is a flowchart of the high-speed WebRTC structure of FIG. 5 .

First, as shown in FIG. 5, the WebRTC structure enables communication between the web browser and the remote control center from the common WebRTC signaling server according to the cloud virtual server when remote support is provided by the corresponding browser and mobile application APIs. This is done with WebRTC.

For example, the API of the browser and mobile application program includes an information exchange protocol according to database registration information, network information exchange, and operations related to WebRTC peer connection in an order according to preset real-time communication.

Therefore, real-time communication of video and audio is implemented without CPU load in browsers and mobile applications through a simple API without the help of a web browser plug-in.

Next, as shown in FIG. 6 , the procedure of such a high-speed web RTC structure is performed as follows.

First, when working in the field, from the server join related API of the browser and mobile application as described above, the smart glasses, client B, and the web agent of the remote control center of client A connect to the server and connect with each other. .

Next, when the connection is established in this way, the web agent of the remote control center requests information exchange protocols such as video, voice, codec, etc. from the smart glasses from the information exchange protocol related API of the browser and mobile application.

Then, when such a request is made, the smart glasses respond to the web agent to exchange information such as video, audio, and codec.

Then, when the information is exchanged in this way, the web agent requests the smart glasses to exchange network information from the network information exchange related API of the browser and mobile application.

Then, when such a request is made, the smartglass responds to the web agent so that network information is exchanged.

So, when network information is exchanged in this way, a WebRTC peer connection is initiated between the smartglasses and the web agent from the browser and the mobile application WebRTC peer connection related API to perform WebRTC.

Therefore, it uses the WebRTC communication method, which is fast, real-time, high-quality communication without delay.

Therefore, when a worker wearing smart glasses is working in a field such as a logistics warehouse, a high-speed web RTC is made between the smart glasses and the web agent, so that the operator on the field and the remote control center communicate smoothly.

7 is a flowchart sequentially illustrating operations of a cloud-based remote service and maintenance support solution using smart glasses according to an embodiment.

As shown in FIG. 7 , the solution is based on the assumption that when field work related data of smart glasses is input, the hardware processor provides the data through communication between the smart glasses and the remote control center to provide remote support for maintenance, etc. .

In this state, the solution provides the data from a cloud virtual server that provides virtual web service by at least one remote control center and the cloud for each smart glass for each of a number of different field workshops at this time (S701).

And, in this case, the cloud virtual server is made by setting a web agent from at least one remote control center and making it a joint platform as a smart glass and all multi-clients.

Incidentally, the joint platform is a cloud-based remote support solution, and it is a joint platform technology that enables bilateral or third-party participation and functions to maximize user convenience and usability.

Next, in addition to this cloud-based remote support, the solution performs the communication from the common WebRTC signaling server according to the cloud virtual server in high-speed WebRTC by the API of the corresponding browser and mobile application (S702) .

In this case, the API includes video, audio, and information exchange protocols including codecs, network information exchange, and WebRTC peer connection.

Therefore, real-time communication of video and audio is implemented without CPU load in browsers and mobile applications through a simple API without the help of a web browser plug-in, and WebRTC communication method, which is fast real-time high-quality communication without delay, is used.

Accordingly, when remote service and maintenance through smart glasses are remotely supported, according to an embodiment, the remote support function of the high-speed web RTC communication method is performed.

Therefore, through this, one embodiment provides the remote support function from these solutions when a worker wearing smart glasses works in a field far away from a remote control center such as a management center or a development center to provide the field situation and immediate feedback provides

Further, in one embodiment, when a worker wearing smart glasses works at a site far from the management center or development center, the maintenance engineer stays in the management center, etc. without having to go on a business trip directly to the site. support

As described above, one embodiment provides field work related data from at least one remote control center and a cloud virtual server that provides web service by the cloud for each smart glass for each of a number of different field work sites when performing field work of smart glasses. .

In addition, when data is provided by the cloud virtual server, the smart glasses and the remote control center are communicated from the browser of the common web RTC signaling server according to the cloud virtual server and the API of the mobile application to achieve high-speed Web RTC.

In this case, the API includes video, audio, and information exchange protocols including codecs, network information exchange, and WebRTC peer connection.

Therefore, when remote service and maintenance through smart glasses are supported remotely, it functions as a remote support function of the high-speed web RTC communication method.

Therefore, through this, one embodiment provides the field situation and immediate feedback by providing the remote support function from such a solution when a worker wearing smart glasses works in a field remote from the management center or development center.

Further, in one embodiment, when a worker wearing smart glasses works at a site far from the management center or development center, the maintenance engineer stays in the management center, etc. without having to go on a business trip directly to the site. support

100: smart glasses 200: remote control center

Claims (3)

Cloud-based remote service and maintenance support system using smart glasses that provides remote support by mutually providing the data from the hardware processor through communication between the smart glasses and the remote control center when the data related to the field work of smart glasses is input In
When the data is provided to the hardware processor, the data is provided from a cloud virtual server that provides virtual web service by at least one remote control center and the cloud for each smart glass for each of a number of different field worksites,
When data is provided by the cloud virtual server, the communication from the common WebRTC signaling server according to the cloud virtual server is performed in high-speed WebRTC by the API of the corresponding browser and mobile application,
The API functions as a remote support function for high-speed WebRTC communication including information exchange protocols including video, audio, and codecs, network information exchange, and WebRTC peer connection,
When feedback is provided by the high-speed web RTC, the remote support function of the high-speed web RTC communication method is expanded by interfacing the augmented reality for delivering feedback from the device itself of the smart glasses,
When the augmented reality is interfaced, a marking for tracking the characteristics and location of the target is generated from object shape recognition and feature-based location recognition for the target, and the target is moved while the marking of the target is moved according to the actual movement of the smart glass. The augmented reality function is strengthened by tracking and showing it in a designated location,
The remote support
First remote including on-off control of video/audio of smart glasses, storage of recorded files in a pre-registered terminal or information processing device, flash on-off control for brightness adjustment of lighting, pause, and live return functions support department
A second remote support that participates between one or more remote control centers and smart glasses,
When the remote control center and smart glasses participate according to the second remote support, in addition to the third party of the related technician's terminal or information processing device, the smart glasses comprising a third remote support A cloud-based remote service and maintenance support system utilizing


delete delete

Images